Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lambda-python): support for providing a custom bundling docker image #18082

Merged
merged 43 commits into from Dec 30, 2021

Conversation

setu4993
Copy link
Contributor

@setu4993 setu4993 commented Dec 18, 2021

This refactors the bundling process to match the NodeJs and Go Lambda functions and allows providing a custom bundling docker image.

Changes:

  • refactor bundling to use cdk.BundlingOptions
  • Use updated Bundling class
  • Update tests to use updated Bundling class

Fixes #10298, #12949, #15391, #16234, #15306

BREAKING CHANGE: assetHashType and assetHash properties moved to new bundling property.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Dec 18, 2021

Copy link
Contributor

@ranjith-jagadeesh ranjith-jagadeesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By mistake I have approved

@setu4993 setu4993 force-pushed the feat/lambda-python/improve-bundling branch from d55c1f6 to 5bcab84 Compare December 25, 2021 09:23
Comment on lines -54 to -56
// Then, write a requirements.txt without the extraneous dependency and synth again
const afterDeps = 'certifi==2020.6.20\nchardet==3.0.4\nidna==2.10\nurllib3==1.26.7\nrequests==2.26.0\nPillow==8.4.0\n';
fs.writeFileSync(requirementsTxtPath, afterDeps);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test kept failing for me, regardless of how I modified the afterDeps value. In the traceback, I could only see one of the commands being run, which makes me think it has something to do with how synth + caching occurs with the bundling process.

In the previous version, dependency installation was occurring during the Docker build process (compared to Docker image run now), so maybe that has something to do with it? Not sure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noticed this in the logs:

### `cdk synthesize`
aws-cdk: Synthesizes the CDK app and produces a cloud assembly to a designated output (defaults to `cdk.out`)
aws-cdk: Typically you don't interact directly with cloud assemblies. They are files that include everything
aws-cdk: needed to deploy your app to a cloud environment. For example, it includes an AWS CloudFormation
aws-cdk: template for each stack in your app, and a copy of any file assets or Docker images that you reference
aws-cdk: in your app.

which to me reads like: app.synth() in the above referenced stack wouldn't actually create the asset; that would be during cdk deploy.

@setu4993
Copy link
Contributor Author

@corymhall : Given all tests, including integration tests are succeeding, I feel good about this PR. I'm happy to start documenting this into the readme and rebase to master if this feels like a good direction.

@corymhall
Copy link
Contributor

@corymhall : Given all tests, including integration tests are succeeding, I feel good about this PR. I'm happy to start documenting this into the readme and rebase to master if this feels like a good direction.

Sounds good.

@setu4993 setu4993 force-pushed the feat/lambda-python/improve-bundling branch from 5bcab84 to 0a43b76 Compare December 28, 2021 01:58
@setu4993 setu4993 changed the title feat(lambda-python): Improve bundling to align with other Lambda functions, support custom bundling Docker image feat(lambda-python): Refactor bundling to align with other Lambda functions, support custom bundling Docker image Dec 28, 2021
Copy link
Contributor Author

@setu4993 setu4993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@corymhall : This is complete and ready for review.

Comment on lines 50 to 60
```ts
new lambda.PythonFunction(this, 'MyFunction', {
entry: '/path/to/my/function',
runtime: Runtime.PYTHON_3_6,
layers: [
new lambda.PythonLayerVersion(this, 'MyLayer', {
entry: '/path/to/my/layer', // point this to your library's directory
}),
],
});
```
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this up.

Comment on lines +42 to +45
```ts
new lambda.PythonLayerVersion(this, 'MyLayer', {
entry: '/path/to/my/layer', // point this to your library's directory
})
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this example.

Comment on lines 75 to 80
Packaging is executed using the `Packaging` class, which:

1. Infers the packaging type based on the files present.
2. If it sees a `Pipfile` or a `poetry.lock` file, it exports it to a compatible `requirements.txt` file with credentials (if they're available in the source files or in the bundling container).
3. Installs dependencies using `pip`.
4. Copies the dependencies into an asset that is bundled for the Lambda package.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some details on how bundling occurs.

Comment on lines 114 to 127
## Custom Bundling

Custom bundling can be performed by using custom Docker images for bundling dependencies. If using a custom Docker image for bundling, the dependencies are installed with `pip`, `pipenv` or `poetry` by using the `Packaging` class.

A different bundling Docker image that is in the same directory as the function can be specified as:

```ts
const entry = '/path/to/function';
const image = DockerImage.fromBuild(entry);

new PythonFunction(stack, 'function', {
entry,
bundling: { image },
});
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added example that shows how to use a custom bundling image.

Comment on lines 6 to 8
# Ensure rsync is installed
RUN yum -q list installed rsync &>/dev/null || yum install -y rsync
# Upgrade pip (required by cryptography v3.4 and above, which is a dependency of poetry)
RUN pip install --upgrade pip
RUN pip install pipenv poetry
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need rsync and can just use cp.

},
platform: architecture.dockerPlatform,
});
this.image = image ?? defaultImage;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where a custom Docker image would get pulled in when bundling.

*/
readonly assetHashType?: AssetHashType;
readonly runtime?: Runtime;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this optional.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We decided to make this a required property in #18143

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I missed that, thanks for the tip! Wouldn't it be nicer to default to Python 3.8 instead to better resolve #10248? That has a much longer EOL, has been around for a few years and is well supported.

*/
readonly assetHashType?: AssetHashType;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to BundlingOptions.

*/
readonly assetHash?: string;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to BundlingOptions.

if (props.runtime && props.runtime.family !== lambda.RuntimeFamily.PYTHON) {
throw new Error('Only `PYTHON` runtimes are supported.');
}
const { index = 'index.py', handler = 'handler', runtime = Runtime.PYTHON_3_7 } = props;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defining defaults while unpacking props.

@corymhall corymhall changed the title feat(lambda-python): Refactor bundling to align with other Lambda functions, support custom bundling Docker image feat(lambda-python): support for providing a custom bundling docker image Dec 30, 2021
Copy link
Contributor

@corymhall corymhall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@setu4993 This looks awesome! Thanks for all the work on this, we're closing a lot of issues :)

@mergify
Copy link
Contributor

mergify bot commented Dec 30, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: b94dbac
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit c3c4a97 into aws:master Dec 30, 2021
@mergify
Copy link
Contributor

mergify bot commented Dec 30, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@setu4993
Copy link
Contributor Author

@setu4993 This looks awesome! Thanks for all the work on this, we're closing a lot of issues :)

Thanks @corymhall! Glad we got this in place :).

@chrispykim
Copy link

chrispykim commented Jan 9, 2022

Hi @setu4993, sorry if this isn't an appropriate place to raise this, but I am having issues with my PythonFunction since this change.

my_lambda = lambda_python.PythonFunction(self, "MyLambdaName",
            entry=path.join(this_dir, "lambda_with_dependencies"),
            index="hello_lambda.py",
            runtime=lambda_.Runtime.PYTHON_3_8,
            log_retention=logs.RetentionDays.THREE_MONTHS,
            reserved_concurrent_executions=1,
            timeout=core.Duration.seconds(60),
            environment={
                "SOMEVAR": "some value"
            })

This is my CDK code, which worked perfectly fine with CDK 1.137.0. With 1.138.0 (and 1.138.1), without making any changes to the above CDK code or the directory structure or the lambda implementation, my lambda suddenly stopped working with this error:

{
  "errorMessage": "Unable to import module 'hello_lambda': No module named 'hello_lambda'",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

Perhaps I am not fully understanding the implications of the breaking changes, but my understanding is that my code should still set up the Docker container just as it did previously. Can you point me towards a solution that works with CDK 1.138.0 and above?

@setu4993
Copy link
Contributor Author

@chrispykim : Thanks for flagging! I missed a /* in there, which was also flagged in #18301 and attempted to fix in #18306, but can confirm it wasn't resolved for me in "1.138.1". I have a fix for it open in #18335, which I have tested locally.

mergify bot pushed a commit that referenced this pull request Jan 10, 2022
Asset files are incorrectly being bundled under the `asset-input` directory instead of the root of the bundle.

To also copy over hidden files (#18306 (comment)), I switched from using `-R` to `-a` based on what I found on [SO](https://stackoverflow.com/a/13020113) and the [man page](https://linux.die.net/man/1/cp). (`-a` is equivalent to `-dR`.)

Fixes #18301 and @chrispykim's comment: #18082 (comment).

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
TikiTDO pushed a commit to TikiTDO/aws-cdk that referenced this pull request Feb 21, 2022
…mage (aws#18082)

This refactors the bundling process to match the NodeJs and Go Lambda functions and allows providing a custom bundling docker image.

Changes:
- refactor bundling to use `cdk.BundlingOptions`
- Use updated `Bundling` class
- Update tests to use updated `Bundling` class


Fixes aws#10298, aws#12949, aws#15391, aws#16234, aws#15306

BREAKING CHANGE: `assetHashType` and `assetHash` properties moved to new `bundling` property.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
TikiTDO pushed a commit to TikiTDO/aws-cdk that referenced this pull request Feb 21, 2022
Asset files are incorrectly being bundled under the `asset-input` directory instead of the root of the bundle.

To also copy over hidden files (aws#18306 (comment)), I switched from using `-R` to `-a` based on what I found on [SO](https://stackoverflow.com/a/13020113) and the [man page](https://linux.die.net/man/1/cp). (`-a` is equivalent to `-dR`.)

Fixes aws#18301 and @chrispykim's comment: aws#18082 (comment).

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
mergify bot pushed a commit that referenced this pull request Feb 28, 2022
…19143)

#18082 added the ability to pass bundling parameters to `PythonFunction`'s. Currently when the `image` parameters is passed both the passed image and the default image are built even though the default image is not used

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[aws-lambda-python] Allow the use of CodeArtifact
6 participants