fix(lambda-python): commands run non-sequentially on Graviton when building container image #22398
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
github-actions
bot
added
beginning-contributor
aws-cdk-automation
previously requested changes
Oct 6, 2022
|
Docker image was built using updated Dockerfile. Packages installed within the image |
TheRealAmazonKendra
changed the title
TheRealAmazonKendra
added
pr-linter/exempt-test
TheRealAmazonKendra
approved these changes
Oct 12, 2022
aws-cdk-automation
dismissed
their stale review
✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.
|
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
|
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
4 tasks
mergify bot
pushed a commit
that referenced
this pull request
Oct 19, 2022
…ll (#22512) This is another attempt at fixing #22012. #22398 intended to fix it, but the issue still occurs when building on arm64 machines with both `compatible_runtimes=[lambdas.Runtime.PYTHON_3_9]` and `compatible_architectures=[lambdas.Architecture.ARM_64]` The core problem is that the build leaves root-owned files under `/tmp/pip-cache`, which then cause permission problems when subsequent layers calling `pip install` are unable to create temporary directories. Example error: ``` WARNING: Building wheel for alembic-utils failed: [Errno 13] Permission denied: '/tmp/pip-cache/wheels/5c' ``` Expanding on the idea within #22398, this PR adds one more step to the chain of operations, removing any temporary files from the cache that are no longer needed (because the packages are now installed), and ensuring that subsequent layers are able to create temporary files inside the cache directories. Comparing the contents of the Docker image before and after this change... Before: ``` $ ls -aFl /tmp/*cache /tmp/pip-cache: total 16 drwxrwxrwx 4 root root 4096 Oct 15 00:57 ./ drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../ drwxr-xr-x 18 root root 4096 Oct 15 00:57 http/ drwxr-xr-x 2 root root 4096 Oct 15 00:57 selfcheck/ /tmp/poetry-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 00:57 ./ drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../ ``` After: ``` $ ls -aFl /tmp/*cache /tmp/pip-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 01:00 ./ drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../ /tmp/poetry-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 00:59 ./ drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../ ``` ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
mrgrain
pushed a commit
to mrgrain/aws-cdk
that referenced
this pull request
Oct 24, 2022
…ll (aws#22512) This is another attempt at fixing aws#22012. aws#22398 intended to fix it, but the issue still occurs when building on arm64 machines with both `compatible_runtimes=[lambdas.Runtime.PYTHON_3_9]` and `compatible_architectures=[lambdas.Architecture.ARM_64]` The core problem is that the build leaves root-owned files under `/tmp/pip-cache`, which then cause permission problems when subsequent layers calling `pip install` are unable to create temporary directories. Example error: ``` WARNING: Building wheel for alembic-utils failed: [Errno 13] Permission denied: '/tmp/pip-cache/wheels/5c' ``` Expanding on the idea within aws#22398, this PR adds one more step to the chain of operations, removing any temporary files from the cache that are no longer needed (because the packages are now installed), and ensuring that subsequent layers are able to create temporary files inside the cache directories. Comparing the contents of the Docker image before and after this change... Before: ``` $ ls -aFl /tmp/*cache /tmp/pip-cache: total 16 drwxrwxrwx 4 root root 4096 Oct 15 00:57 ./ drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../ drwxr-xr-x 18 root root 4096 Oct 15 00:57 http/ drwxr-xr-x 2 root root 4096 Oct 15 00:57 selfcheck/ /tmp/poetry-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 00:57 ./ drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../ ``` After: ``` $ ls -aFl /tmp/*cache /tmp/pip-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 01:00 ./ drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../ /tmp/poetry-cache: total 8 drwxrwxrwx 2 root root 4096 Oct 15 00:59 ./ drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../ ``` ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #22012
There were reports that Dockerfile RUN commands are done out of order when building the container image on Graviton. I combined all the separate RUN commands into a single command and using the shell to explicitly enforce the sequence.
All Submissions:
Adding new Unconventional Dependencies:
New Features
yarn integto deploy the infrastructure and generate the snapshot (i.e.yarn integwithout--dry-run)?By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license