-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(lambda): higher level construct for Node.js #5532
Conversation
Introduce bundlers for S3 assets and implement Parcel for JavaScript/TypeScript entry files. Add a `fromParcel` static method to `@aws-cdk/aws-lambda.Code` to create Parcel bundled Lambda functions. Bundlers are defined in `@aws-cdk/cx-api` and extend the `Bundler` abstract class. They implement a `bundle` method that takes a `FileAssetMetadataEntry` and returns a transformed `FileAssetMetadataEntry`. The zip packaging is then applied on the transformed asset. Since assets are currently "transported" to `aws-cdk` using metadata, concrete TypeScript class objects cannot be "transported". To overcome this limitation the class name and constructor options of the bundler are used as metadata. A new instance of the bundler is then "re-created" in `aws-cdk` where the real bundling occurs. This has the disadvantage that bundlers can only be implemented in `@aws-cdk/cx-api` (user provided bundlers will not work).
This is far from perfect but is here to start the discussion about asset bundling... something that will greatly boost CDK adoption for JS/TS developers working with Lambda... especially those coming from the serverless framework and looking for a "it just works" solution |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
This is great. Definitely an area we've been exploring. |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
This is awesome 🔥. One thing we talked about briefly is which bundler to use. I don't have much experience with parcel, but it does have quite a lot of dependencies compared to something like rollup. @jogold do you believe the benefits of the "zero" config api warrant that cost? I think it could depending on just how "magic" we could make the experience for end users. For example, it looks like when using typescript, parcel just knows how to build/bundle the source and will use whatever .tsconfig file it finds? If a user wanted to use different configs, or specify one for only their lambda code (or other assets) how does that work? Does babel work similarly, just finding the relevant .babelrc etc... Is that something we need to consider exposing in the These may not necessarily need to be answered immediately just thoughts. May be something we could use feedback to help decide. |
Yes it does and is indeed linked to the fact that it offers a "zero" config api. I personally think this is great.
Parcel performs no type checking, it uses
Parcel uses a default Babel transform which can be overridden in a It's important to decide what kind of user experience the CDK wants to offer here, I see three options:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is definitely a desired feature, but I think we need to come up with a way to do this that will not require us to hard-code every type of bundler into the framework and CLI. So people can then vend libraries that integrate with various bundlers and implement any type of logic they want.
I am also leaning more and more towards just implementing these things at the app-level and not split the responsibilities between the app and the CLI. Originally we had some reasons related to security to split those up, but those concerns are mitigated by the fact that our CI/CD story isolates the deployment action (which is where you need administrative privileges) from the rest of the process.
The only remaining "problem" is how to execute async code during synthesis, but maybe we can simply just defer to shell commands and use execSync
. It's a JS technicality, not a philosophical issue.
@jogold @MrArnoldPalmer how about scheduling some time that we are all on-line and we can brainstorm a little around this topic? Probably worth writing up a little RFC to explore some options.
Will have time next week Monday or Wednesday in the afternoon |
@MrArnoldPalmer @eladb a "sync" version at the app level master...jogold:parcel-sync to be discussed |
About the "limitations" mentioned yesterday: parcel-bundler/parcel#2493 |
I really like this idea. Does the Bundler interface allow implementing bundlers that build the ZIP themselves and return this to CDK? I am thinking of Claudia as a bundler. |
@hoegertn It has been decided for this PR to go for a construct that offers a better user experience for Node.js Lambda functions: simplified props, automatic bundling, etc... The "bundler interface" remains extending the abstract |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Is adding support to create a Lambda Layer from node modules in scope for this PR? I often need to include node modules that can't be bundled through Parcel, Rollup, or Webpack so that I can use native features or external libraries inside the Lambda function. This may seem like a silly anecdote, but the question of how I can include external node modules in Lambda was one of the reasons I shied away from node/TypeScript in Lambda for quite some time. If it helps, here's a snippet of code that I use in my projects: https://gist.github.com/misterjoshua/51b1db755cbff547c0f0d70b5f22ab15 Using the above, I create lambda layers from my node_modules like this: const nodeModulesLayer = new lambda.LayerVersion(this, "TestLayer", {
code: NodeModulesCode.fromNodeModules("./"),
}); |
@misterjoshua that's a good question. Can you give an example of some dependencies that you use often that are not friendly to bundling? @jogold does parcel allow for "excluding" certain dependencies from the output? I can't find it in the documentation off hand. I'm wondering how common this strategy is and how often its needed. |
|
||
This library provides constructs for Node.js Lambda functions. | ||
|
||
### Node.js Function |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For better discoverability, let's put this information in the @aws-cdk/aws-lambda README under "Language-specific APIs" or something like that and here just redirect to the @aws-cdk/aws-lambda module.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The whole README?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe just a reference to this module from the main module?
By default, the construct will look up for a `index.ts` or `index.js` that exports a `handler` | ||
function in the folder named as the construct's id: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was actually thinking (albeit don't know for sure that this is the right approach) that the default behavior should be to use the name of the defining file with some suffix. You will need to roll back the stack trace to find the name of the defining file name but that should be possible.
.
├── boom.ts # this is where NodejsFunction is defined
├── boom.lambda.ts . # this is where `handler` is exported
Since we now have tree shaking and all that nice stuff, I'd argue that creating a special directory for each lambda function is a big hassle. Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In your example, lambda
is the construct's id of the NodejsFunction
in boom.ts
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, the name of the lambda handler file is based on the name of the declaring file. Requires a bit of stack trace magic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And if there are multiple NodejsFunction
declared in boom.ts
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well.... didn’t think that far out :-)
I guess using the construct id as the suffix might be a good solution.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Used the construct id, see latest changes
`buildDir` and `cacheDir`. | ||
|
||
Parcel transpiles your code (every internal module) with `@babel/preset-env` and uses the | ||
`engines.node` defined in `package.json` as target (defaults to Node.js 8). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The default node version should be read from process.versions.node
instead of package.json
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I can tell Parcel to do that via CLI
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not possible to specify the babel target via CLI. This just means that the code gets transpiled "too much" if nothing is specified in engines.node
.
An alternative would be to save the package.json
, update it with a engines.node
key and then revert it...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be nice to tell parcel to compile to the node target specified in the function props runtime
argument. Doesn't seem thats easily achievable though. This feels like the ideal over process.versions.node
to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Found a solution for this, see latest commit 2410689.
@eladb @MrArnoldPalmer are you ok with modifying and then restoring a user's package.json
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am ok with it.
Parcel transpiles your code (every internal module) with `@babel/preset-env` and uses the | ||
`engines.node` defined in `package.json` as target (defaults to Node.js 8). | ||
|
||
Configuring Babel with Parcel is possible via a `.babelrc` or a `babel` config in `package.json`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also think it should be possible to configure these through the construct (probably a better practice).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the only way to configure Babel when using Parcel (no CLI option). So we would have to write a .babelrc
at the root of the package before calling parcel
... not really fan of this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we write a babelrc file in the .build directory and then remove it at the end? I do think it makes more sense that all config will be in code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if this config already exists (either in .babelrc
or package.json
)? See also my comment about the babel target.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
without a pre-existing .babelrc, parcel already does some base level transpilation it seems. Does that mean to user only has to define a .babelrc
if they want babel config beyond the default? That's the ideal setup in my mind. Most users that are using babel and know about it are used to .babelrc
and it feels like the idiomatic way to do config to me.
From parcel docs
In addition to any transforms specified in .babelrc, Parcel always uses Babel on all modules to compile modern JavaScript into a form supported by browsers. See the JavaScript/Default Babel Transforms section for more information.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does that mean to user only has to define a .babelrc if they want babel config beyond the default?
Yes
|
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Imo @eladb we have been discussing some limitations of parcel that will prevent certain users from being able to use this construct. Basically, some dependencies are not friendly to bundling, mostly those with native code/functionality, for example sharp. Webpack lets us expose options that will not bundle code from node_modules and then we will copy the dependencies we need to the asset directory instead of attempting to inline them into the single script. @jogold I'm okay with moving forward as is if we can change the bundler under the hood with no breaking api changes in the future. I just want to make sure we can add this feature safely since it does feel pretty important. |
+1 for |
The runtime and runtime family enums currently use |
I don't care really (for a change) |
Let's move forward with this for now. I think the case for native modules is actually more complicated. We will probably need to |
Here is another relevant issue: #1620 (guess who opened it?) |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Alright lets move forward with parcel and see how people end up using it what type of changes they request. Excited to start using this! |
I need to change the integ test. It uses |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Thank you for contributing! Your pull request is now being automatically merged. |
Was this package ever pushed to npmjs.com? |
@stephankaag not yet, it will be released with v1.23.0 |
Great. Can't wait to use it. |
Thank you for this wonderful addition to AWS CDK. I've just started using it in a real-life project and so far it's been looking great. |
…cdk/lambda-layer-awscli (#25012) Bumps [awscli](https://github.com/aws/aws-cli) from 1.27.104 to 1.27.109. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst">awscli's changelog</a>.</em></p> <blockquote> <h1>1.27.109</h1> <ul> <li>api-change:<code>dlm</code>: Updated timestamp format for GetLifecyclePolicy API</li> <li>api-change:<code>docdb</code>: This release adds a new parameter 'DBClusterParameterGroupName' to 'RestoreDBClusterFromSnapshot' API to associate the name of the DB cluster parameter group while performing restore.</li> <li>api-change:<code>fsx</code>: Amazon FSx for Lustre now supports creating data repository associations on Persistent_1 and Scratch_2 file systems.</li> <li>api-change:<code>lambda</code>: This release adds a new Lambda InvokeWithResponseStream API to support streaming Lambda function responses. The release also adds a new InvokeMode parameter to Function Url APIs to control whether the response will be streamed or buffered.</li> <li>api-change:<code>quicksight</code>: This release has two changes: adding the OR condition to tag-based RLS rules in CreateDataSet and UpdateDataSet; adding RefreshSchedule and Incremental RefreshProperties operations for users to programmatically configure SPICE dataset ingestions.</li> <li>api-change:<code>redshift-data</code>: Update documentation of API descriptions as needed in support of temporary credentials with IAM identity.</li> <li>api-change:<code>servicecatalog</code>: Updates description for property</li> </ul> <h1>1.27.108</h1> <ul> <li>api-change:<code>cloudformation</code>: Including UPDATE_COMPLETE as a failed status for DeleteStack waiter.</li> <li>api-change:<code>greengrassv2</code>: Add support for SUCCEEDED value in coreDeviceExecutionStatus field. Documentation updates for Greengrass V2.</li> <li>api-change:<code>proton</code>: This release adds support for the AWS Proton service sync feature. Service sync enables managing an AWS Proton service (creating and updating instances) and all of it's corresponding service instances from a Git repository.</li> <li>api-change:<code>rds</code>: Adds and updates the SDK examples</li> </ul> <h1>1.27.107</h1> <ul> <li>bugfix:eks: Fix eks kubeconfig validations closes <code>[#6564](aws/aws-cli#6564) <https://github.com/aws/aws-cli/issues/6564></code><strong>, fixes <code>[#4843](aws/aws-cli#4843) <https://github.com/aws/aws-cli/issues/4843></code></strong>, fixes <code>[#5532](aws/aws-cli#5532) <https://github.com/aws/aws-cli/issues/5532></code>__</li> <li>api-change:<code>apprunner</code>: App Runner adds support for seven new vCPU and memory configurations.</li> <li>api-change:<code>config</code>: This release adds resourceType enums for types released in March 2023.</li> <li>api-change:<code>ecs</code>: This is a document only updated to add information about Amazon Elastic Inference (EI).</li> <li>api-change:<code>identitystore</code>: Documentation updates for Identity Store CLI command reference.</li> <li>api-change:<code>ivs-realtime</code>: Fix ParticipantToken ExpirationTime format</li> <li>api-change:<code>network-firewall</code>: AWS Network Firewall now supports IPv6-only subnets.</li> <li>api-change:<code>servicecatalog</code>: removed incorrect product type value</li> <li>api-change:<code>vpc-lattice</code>: This release removes the entities in the API doc model package for auth policies.</li> </ul> <h1>1.27.106</h1> <ul> <li>api-change:<code>amplifyuibuilder</code>: Support StorageField and custom displays for data-bound options in form builder. Support non-string operands for predicates in collections. Support choosing client to get token from.</li> <li>api-change:<code>autoscaling</code>: Documentation updates for Amazon EC2 Auto Scaling</li> <li>api-change:<code>dataexchange</code>: This release updates the value of MaxResults.</li> <li>api-change:<code>ec2</code>: C6in, M6in, M6idn, R6in and R6idn bare metal instances are powered by 3rd Generation Intel Xeon Scalable processors and offer up to 200 Gbps of network bandwidth.</li> <li>api-change:<code>elastic-inference</code>: Updated public documentation for the Describe and Tagging APIs.</li> <li>api-change:<code>sagemaker-runtime</code>: Update sagemaker-runtime command to latest version</li> <li>api-change:<code>sagemaker</code>: Amazon SageMaker Asynchronous Inference now allows customer's to receive failure model responses in S3 and receive success/failure model responses in SNS notifications.</li> <li>api-change:<code>wafv2</code>: This release rolls back association config feature for webACLs that protect CloudFront protections.</li> </ul> <h1>1.27.105</h1> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-cli/commit/300c55450414e611a461accc8d15857703c19e7a"><code>300c554</code></a> Merge branch 'release-1.27.109'</li> <li><a href="https://github.com/aws/aws-cli/commit/0f39d4d98e418390fa46f9408a704d8e6ee17642"><code>0f39d4d</code></a> Bumping version to 1.27.109</li> <li><a href="https://github.com/aws/aws-cli/commit/8c3681ff072698869446a23d15ade9b1124868dc"><code>8c3681f</code></a> Merge commit '23360bea246fc1d11ebd8c216fd29e5aa29695df' into stage-release-de...</li> <li><a href="https://github.com/aws/aws-cli/commit/83b647ea31156d5c9cb848259fde52e7631776c4"><code>83b647e</code></a> Update changelog based on model updates</li> <li><a href="https://github.com/aws/aws-cli/commit/746f66ed34139cf3832ff59052a1a4312de44c4f"><code>746f66e</code></a> Merge branch 'release-1.27.108'</li> <li><a href="https://github.com/aws/aws-cli/commit/3d21d242ffc04df36e42234c5a2947844d7d7096"><code>3d21d24</code></a> Merge branch 'release-1.27.108' into develop</li> <li><a href="https://github.com/aws/aws-cli/commit/2a7efe33c452a87877553d3f5fb56880aad19ea2"><code>2a7efe3</code></a> Bumping version to 1.27.108</li> <li><a href="https://github.com/aws/aws-cli/commit/c52959e85c73f4701eef2f253cbf89e3419c8303"><code>c52959e</code></a> Update changelog based on model updates</li> <li><a href="https://github.com/aws/aws-cli/commit/98e06f77e460e2cbf86ec8b5fe1231eefb0eba59"><code>98e06f7</code></a> Merge branch 'release-1.27.107'</li> <li><a href="https://github.com/aws/aws-cli/commit/1f6b68f3f8e99a39ba6b5625699a3978f1c0f48c"><code>1f6b68f</code></a> Merge branch 'release-1.27.107' into develop</li> <li>Additional commits viewable in <a href="https://github.com/aws/aws-cli/compare/1.27.104...1.27.109">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=awscli&package-manager=pip&previous-version=1.27.104&new-version=1.27.109)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
Introduce bundlers for S3 assets and implement Parcel for JavaScript/TypeScriptentry files. Add a
fromParcel
static method to@aws-cdk/aws-lambda.Code
tocreate Parcel bundled Lambda functions.
Bundlers are defined in@aws-cdk/cx-api
and implement theIBundler
interfaceThey implement a
bundle
method that takes aFileAssetMetadataEntry
andreturns a transformed
FileAssetMetadataEntry
. The zip packaging is thenapplied on the transformed asset.
Since assets are currently "transported" toaws-cdk
using metadata, concreteTypeScript class objects cannot be "transported". To overcome this limitation
the class name and constructor options of the bundler are used as metadata. A
new instance of the bundler is then "re-created" in
aws-cdk
where the realbundling occurs. This has the disadvantage that bundlers can only be implemented
in
@aws-cdk/cx-api
(user provided bundlers will not work).Higher level API to work with Lambda functions written in Node.js.
Closes #1620
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license