add worker lambda template that's deployed by AWS CDK #395

damienung1 · 2021-04-01T13:06:27Z

Why?
Reduce the setup costs for AWS CDK deployed workers.

changeset-bot · 2021-04-01T13:06:29Z

🦋 Changeset detected

Latest commit: c93dca8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
skuba	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

72636c

Thanks for the contribution! This looks like a generally useful addition 🙇

There's a bit of detail to work through as the template is fairly different from the others; my read is that this is unnecessary divergence but I may be mistaken. I'd recommend looking through the other templates and skuba initing the greeter and lambda-sqs-worker templates to get a feel for how those are wired up. If you'd like to propose changes to the status quo, I'd like to land those across all templates rather than just this one.

template/lambda-sqs-worker-cdk/.me

template/lambda-sqs-worker-cdk/builder.dockerfile

template/lambda-sqs-worker-cdk/shared/context-types.ts

72636c · 2021-04-01T13:32:42Z

template/lambda-sqs-worker-cdk/package.json

+    "@types/jest": "^24.0.22",
+    "@types/node": "10.17.5",
+    "aws-cdk": "^1.18.0",
+    "esbuild": "^0.11.2",


We can't accept this template while it rolls its own build system. This is at odds with skuba's goals and any built-in template should stick to its commands (i.e. skuba build). I also don't understand why we're using a bundler for backend code.

We use it to simplify transpile + packaging depedencies. Not so much for any code bundle optimisation that we can get - though that's a nice side effect. We could do without for sure.

I noticed that there's some babel experimentation going on in build cli. Does that mean we can expect to be able to swap out the compile/transpile tool underneath skuba build in the future or is it going to be just tsc indefinitely?

Also curious, would the use of this construct, which does automatic transpilation and bundling with esbuild under the hood be deviating from the vision? It seems it only does one extra step over Serverless framework's packaging (i.e. bundling).

Just curious as the bottom line is I'm happy to swap the bundler out for vanilla transpile and package.

Does that mean we can expect to be able to swap out the compile/transpile tool underneath skuba build in the future or is it going to be just tsc indefinitely?

I'm envisioning that we leave the door open to defaulting to a different tool if it better meets our needs. Babel gives us more features like macros and custom import paths at compile time, but it isn't the default yet as we haven't really dug into its drawbacks (worse REPL, incomplete TypeScript syntax support, another moving part which will trail TypeScript to some degree).

I may have misunderstood the issue here. In other templates we simply compile to an output directory (i.e. lib) then include that and node_modules in the final image/zip. Are we using a bundler here to work around aws/aws-cdk#984?

In practice my team tends to have a few types of workers (that implement functional and "non-functional" requirements) within the same domain, and the worker types of that domain live in the same repository. So we have multiple points of entry to package for.

Using a bundler isn't a workaround, as we probably aren't blocked by the issue without one (TBD - I haven't explored this fully). However, until AWS CDK supplies a packaging feature, using a bundler eliminates the need to roll our own packaging script.

Could esbuild be added as a non-default build tool instead? I'm aware it's stretching (maybe even breaking) the current definition by virtue of it being a bundler.

This template doesn't demonstrate multiple workers / monorepoing though, and it's not clear to me how esbuild enables that use case. Lambda function entry points are configured separate from their bundles and there are plenty of SEEK repos that implement the configuration you describe with the sensible defaults of tsc and Serverless.

I don't think it makes sense for skuba build to proxy esbuild unless it enables a broadly useful scenario that can't be accomplished with tsc or babel today.

I've removed esbuild and replaced it with basic packaging.

template/lambda-sqs-worker-cdk/package.json

template/base/tsconfig.json

template/lambda-sqs-worker-cdk/src/app.ts

72636c · 2021-04-01T23:04:27Z

README.md

@@ -169,6 +169,14 @@ This initialises a new directory and Git repository.
  a message queue is employed between the source topic and the Lambda function,
  and unprocessed events are sent to a dead-letter queue for manual triage.

+- `lambda-sqs-worker-cdk`
+
+  An asynchronous [worker] built on [AWS Lambda] and deployed with [AWS CDK].


This doesn't seem in line with the Technology Strategy: https://tech-strategy.ssod.skinfra.xyz/docs/v1/technology.html#lambda

I don't mind merging this, but can we kick off internal discussions over the merit of the CDK as a Lambda deployment tool?

Possibly a misunderstanding on my part:
https://tech-strategy.ssod.skinfra.xyz/docs/v1/components.html#infrastructure

I read it to include Lambda workers. We're already using CDK to deploy workers in practice (among other things).

I think @72636c is referring to:

Lambda functions SHOULD be deployed using Serverless

This also repeated under the Worker section:

SHOULD deploy using Serverless

And the Serverless section:

https://serverless.com/

Serverless is our current default choice for packaging and deploying AWS Lambda functions.

Tech Strategy allowing CDK for infrastructure wouldn’t override the above guidance.

If your team has decided to deviate from SEEK’s Tech Strategy you could BYO GitHub template instead of including it in Skuba’s default templates.

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

To avoid naming collisions with built-in JavaScript and Typescript constructs

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

Was set by `cdk init` and overlooked

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

template/lambda-sqs-worker-cdk/.nvmrc

Sorry, couldn't suggest this change in the GitHub UI.

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

72636c

Apologies for the delay here, it's looking good. I just have a few outstanding questions.

README.md

template/lambda-sqs-worker-cdk/Dockerfile

template/lambda-sqs-worker-cdk/src/app.ts

template/lambda-sqs-worker-cdk/infra/appStack.ts

72636c · 2021-04-07T23:36:58Z

template/lambda-sqs-worker-cdk/infra/index.ts

@@ -0,0 +1,15 @@
+#!/usr/bin/env node


Does the shebang do anything given this is TypeScript source code and there's a script to run it through ts-node?

It's redundant, so I've removed it.

72636c · 2021-04-07T23:43:04Z

template/lambda-sqs-worker-cdk/package.json

+    "format": "skuba format",
+    "lint": "skuba lint",
+    "test": "skuba test",
+    "package": "yarn build && yarn --prod --modules-folder ./lib/node_modules",


Is this a local operation or does Yarn have to fetch dependencies over the network? Ideally we'd try to avoid additional installation at this point so our docker-ecr-cache actually pays off and to limit exposure of the npm token.

It's used to build the docker image, and also doubles as a local operation.

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

72636c

Legend! Thanks for your perseverance here 🙇

damienung1 requested a review from a team as a code owner April 1, 2021 13:06

damienung1 force-pushed the cdk-lambda-sqs-worker-template branch 2 times, most recently from dcb1be1 to 3e99bab Compare April 1, 2021 13:11

damienung1 closed this Apr 1, 2021

damienung1 reopened this Apr 1, 2021

damienung1 added 2 commits April 2, 2021 07:50

add worker lambda template that's deployed by AWS CDK

d1c21c7

update base template test snapshot

1b3150d

damienung1 force-pushed the cdk-lambda-sqs-worker-template branch from a94244f to 1b3150d Compare April 1, 2021 20:55

72636c reviewed Apr 1, 2021

View reviewed changes

damienung1 and others added 17 commits April 2, 2021 17:23

Update template/lambda-sqs-worker-cdk/.me

c37e1e1

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

relax build container version to major version only

f473201

follow style recommendation for runtype imports

afaeb62

To avoid naming collisions with built-in JavaScript and Typescript constructs

remove unused package

1a7f7e1

remove maxWorkers flag from test step

8b1ac25

add @seek/logger, log hello world only

d62dd06

remove CDK generated object that's not actually being used

acb8a44

Update template/lambda-sqs-worker-cdk/.buildkite/pipeline.yml

9f34eeb

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

remove @types/jest as it's bundled in skuba

d8d3616

remove code portion that's supposed to be populated by skuba init

d7286fe

update @types/node

56dd771

Was set by `cdk init` and overlooked

keep to default Dockerfile naming

acac1f4

Update README.md

0b3ba19

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

Update template/lambda-sqs-worker-cdk/infra/appStack.test.ts

0a8a707

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

remove unnecessary shell out to bash

f0daac2

move CDK gitignores to base

a98046a

update snapshot test

bf61a52

72636c mentioned this pull request Apr 3, 2021

Switch to Node.js --enable-source-maps #399

Closed

72636c reviewed Apr 3, 2021

View reviewed changes

template/lambda-sqs-worker-cdk/.nvmrc Outdated Show resolved Hide resolved

Tweak base .gitignore

c788c18

Sorry, couldn't suggest this change in the GitHub UI.

damienung1 and others added 4 commits April 3, 2021 20:49

Update template/lambda-sqs-worker-cdk/.nvmrc

5658f51

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

remove bash and git install step from the build docker image

44bd211

upstream resolveJsonModule

211232e

remove esbuild

5bcd4ff

72636c reviewed Apr 7, 2021

View reviewed changes

damienung1 and others added 6 commits April 8, 2021 11:30

Update README.md

5b8bcf5

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

Update template/lambda-sqs-worker-cdk/src/app.ts

5021fc8

Co-authored-by: Ryan Ling <ryan@outlook.com.au>

prune configuration that needs to be managed in CDK context

238e850

fix ECR caching and packaging

b15ccc2

remove redundant shebang

bd4ae58

Merge branch 'master' into cdk-lambda-sqs-worker-template

ef8f9d4

72636c approved these changes Apr 12, 2021

View reviewed changes

72636c added 3 commits April 12, 2021 11:55

Create empty-shirts-film.md

357ac30

Add to CI

4619139

Add to init prompt

c93dca8

72636c merged commit 13b3bce into master Apr 12, 2021

72636c deleted the cdk-lambda-sqs-worker-template branch April 12, 2021 03:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add worker lambda template that's deployed by AWS CDK #395

add worker lambda template that's deployed by AWS CDK #395

damienung1 commented Apr 1, 2021

changeset-bot bot commented Apr 1, 2021 •

edited

72636c left a comment

72636c Apr 1, 2021

damienung1 Apr 2, 2021

damienung1 Apr 2, 2021

damienung1 Apr 2, 2021 •

edited

72636c Apr 3, 2021

72636c Apr 3, 2021

damienung1 Apr 3, 2021

72636c Apr 4, 2021

damienung1 Apr 5, 2021

72636c Apr 1, 2021

damienung1 Apr 2, 2021

etaoins Apr 2, 2021 •

edited

72636c left a comment

72636c Apr 7, 2021

damienung1 Apr 8, 2021

72636c Apr 7, 2021

damienung1 Apr 8, 2021

72636c left a comment

add worker lambda template that's deployed by AWS CDK #395

add worker lambda template that's deployed by AWS CDK #395

Conversation

damienung1 commented Apr 1, 2021

changeset-bot bot commented Apr 1, 2021 • edited

🦋 Changeset detected

72636c left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damienung1 Apr 2, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etaoins Apr 2, 2021 • edited

Choose a reason for hiding this comment

72636c left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

72636c left a comment

Choose a reason for hiding this comment

changeset-bot bot commented Apr 1, 2021 •

edited

damienung1 Apr 2, 2021 •

edited

etaoins Apr 2, 2021 •

edited