feat(core,s3-assets,lambda): custom asset bundling #7898

jogold · 2020-05-10T22:01:38Z

Adds support for asset bundling by running a command inside a Docker container.

The asset path is mounted in the container at /asset-input and is set as the working
directory. The container is responsible for putting content at /asset-output. The content
at /asset-output will be zipped and used as the final asset.

This allows to use Docker for Lambda code bundling.

It will also be possible to refactor aws-lambda-nodejs and create other language
specific modules.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Add `Code.fromDockerImage` and `Code.fromDockerAsset` that offer a generic way of working with Docker for Lambda code. `Code.fromDockerImage`: run a command in an existing Docker image `Code.fromDockerAsset`: build an image and then run a command in it Both `Code` classes take an `assetPath` prop that corresponds to the path of the asset directory that will contain the build output of the Docker container. This path is automatically mounted at `/asset` in the container. Using a combination of image and command, the container is then responsible for putting content at this location. Additional volumes can be mounted if needed. This will allow to refactor `aws-lambda-nodejs` and create other language specific modules.

jogold · 2020-05-10T22:03:51Z

@eladb see integ test

aws-cdk-automation · 2020-05-10T22:11:03Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 3840a4a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-05-16T13:11:05Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: ca62f37
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-05-16T16:01:24Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 116d985
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-05-16T16:06:46Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: d2a66c8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-05-16T19:04:18Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 13057c8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-05-16T19:26:33Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: aad39af
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-05-16T20:24:30Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 2d780e6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

eladb

These APIs implies that the code comes from a docker image ("code from docker image" and "code from docker asset"), which could be a pretty interesting feature but this is not what this is doing.

My mental model for this is basically this is a new option we add to the normal lambda.Code.fromAsset:

code: lambda.Code.fromAsset('/path/to/source', {
  bundle: {
    image: lambda.BundleImage.fromBuild('/directory', { buildOpts: ... }),
    // or
    image: lambda.BundleImage.fromImage('python'),
    command: [ 'foo', 'bar' ],
    volumes: // ...
    environment: // ...
  }
});

Here's how the integ test would look like:

new lambda.Function(this, 'Function', {
  code: lambda.Code.fromAsset(path.join(__dirname, 'python-lambda-handler'), {
    bundle: {
      image: lambda.BundleImage.fromImage('python:3.6'),
      command: [ 'pip', 'install', '-r ./requirements.txt', '-t .' ],
    }
  })
});

packages/@aws-cdk/aws-lambda/README.md

packages/@aws-cdk/aws-lambda/test/integ.docker.ts

packages/@aws-cdk/aws-lambda/lib/code.ts

Pull request has been modified.

jogold · 2020-05-18T10:39:45Z

My mental model for this is basically this is a new option we add to the normal lambda.Code.fromAsset:

@eladb addressed all your feedback, are you sure about the naming here? is bundle the right prop name?

aws-cdk-automation · 2020-05-18T11:02:33Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 3d3e7c2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

packages/@aws-cdk/aws-lambda/lib/code.ts

eladb

I really like where this is going. @nija-at please review as well...

@jogold I think we should think about this as "custom code bundling". The docker stuff is just the mechanism (update PR title & description accordingly).

Also, I am bothered by the fact that the input and output are the same directory. Feels to me that they should be seprated to /src and /dist (or /bundle) or /output or /outdir or something like that. If the bundling logic wants to copy all the sources they can do it but a general rule, I am not sure it makes total sense to bundle "in-place". What do you think?

packages/@aws-cdk/aws-lambda/README.md

packages/@aws-cdk/aws-lambda/lib/docker.ts

eladb · 2020-05-18T11:21:11Z

packages/@aws-cdk/aws-lambda/lib/docker.ts

+   *
+   * @param image the image name
+   */
+  public static fromImage(image: string) {


I am thinking that maybe we should model this like ecs.ContainerImage

fromImage => fromRegistry

fromBuild => fromAsset

And the question will be: do we want to also add fromEcrRepository, but we can add that later.

we should model this like ecs.ContainerImage

done

do we want to also add fromEcrRepository

for another PR I would say

packages/@aws-cdk/aws-lambda/lib/docker.ts

packages/@aws-cdk/aws-lambda/lib/code.ts

packages/@aws-cdk/core/lib/asset-staging.ts

eladb · 2020-06-09T08:26:13Z

packages/@aws-cdk/core/lib/asset-staging.ts

+    const hashType = props.assetHash ? AssetHashType.CUSTOM : props.assetHashType ?? AssetHashType.SOURCE;
+    this.assetHash = this.calculateHash(hashType, props.assetHash);


Technically you are missing a validation here (assetHash is defined but assetHashType is not CUSTOM).

It's on L73-L75 above, will move it closer

I would just move the validation into calculateHash and pass in props.

packages/@aws-cdk/core/lib/asset-staging.ts

eladb · 2020-06-09T08:26:47Z

packages/@aws-cdk/core/lib/asset-staging.ts

+  }
+
+  private calculateHash(hashType: AssetHashType, assetHash?: string): string {
+    if (hashType === AssetHashType.SOURCE) {


use switch?

aws-cdk-automation · 2020-06-09T08:46:19Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 2795e87
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-06-09T09:05:09Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 3b77e7f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-06-09T09:15:12Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 1ff601d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-06-09T09:24:20Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 17df113
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-06-09T10:03:47Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 96df801
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-06-09T10:05:50Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 16fe727
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2020-06-09T10:06:11Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2020-06-09T11:26:36Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 8f372ff
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2020-06-09T11:26:55Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

imtiazmangerah · 2020-06-11T03:36:27Z

@jogold Thanks for this - been waiting for it for a while and had some custom tooling to do exactly what this is doing.

Quick note - I have tested it, the output files from the container is owned by root.

bundling_options = core.BundlingOptions(image=aws_lambda.Runtime.PYTHON_3_7.bundling_docker_image,
                                        command=[
                                            'bash', '-c',
                                            'pip install -r requirements.txt -t /asset-output && rsync -r . /asset-output',
                                        ])
self.source_code = aws_lambda.Code.from_asset('src/test', bundling=bundling_options)


handler = aws_lambda.Function(self, f"{self.id}-TestBundling",
                              function_name=f"{self.id}-TestBundling",
                              handler=f'handlers.test',
                              runtime=aws_lambda.Runtime.PYTHON_3_7,
                              code=self.source_code)

Anything I am doing wrong or workaround for this? If not will create an issue for it.

EDIT

I can work around this in linux by passing in -u 1000:1000 in bundling.ts as part of dockerArgs (where 1000 is my uid). Not sure what a cross platform solution for this looks like.

eladb · 2020-06-11T05:07:27Z

I can work around this in linux by passing in -u 1000:1000 in bundling.ts as part of dockerArgs (where 1000 is my uid). Not sure what a cross platform solution for this looks like.

I think we should be able to pass -u $UID:$UID by default. @imtiazmangerah care to raise a issue?

imtiazmangerah · 2020-06-11T05:49:20Z

New issue created - #8489

eikeon · 2020-06-12T22:10:24Z

Works great; now to figure out what to do when cdk is running from within a docker container as part of CI/CD and I hit Error: spawnSync docker ENOENT. Anyone navigate around this already? Trying docker:dind but guess I don't have it all set up yet

eladb · 2020-06-14T05:42:55Z

Works great; now to figure out what to do when cdk is running from within a docker container as part of CI/CD and I hit Error: spawnSync docker ENOENT. Anyone navigate around this already? Trying docker:dind but guess I don't have it all set up yet

You will need your ci/cd environment to support “docker in docker”.

rudaporto · 2020-06-19T08:53:57Z

Works great; now to figure out what to do when cdk is running from within a docker container as part of CI/CD and I hit Error: spawnSync docker ENOENT. Anyone navigate around this already? Trying docker:dind but guess I don't have it all set up yet

You will need your ci/cd environment to support “docker in docker”.

There are some other tricks..

When not using your own Dockerfile to create the bundle it relies in the /asset-input being mounted in the base container.

But docker-in-docker means that the new container will look for the path in the host filesystem.
And it usually means that /asset-input will be empty if the path inside the base container is not the same as the path as in the host.

It's possible to pass a new custom volume in the volumes parameter to mount the code using the correct path if available in the host filesystem but it's something hard to guarantee. It depends of how much control of the environment do you have to make sure that the path where the code is checkout out by the CI is mounted in a bind volume from the HOST to be then also mounted again in the bundling docker container using the same absolute path.

I made it work by:

using a custom Dockerfile (from lambda ci/cd) and added the COPY command to copy the lambda code inside during the build of the bundling image.
and adding the workingDirectory parameter to specify the directory where it will run the command script, this case /var/task

mhart · 2020-07-25T14:02:03Z

Given that we have no way of controlling how /asset-output is zipped – because this is hardcoded here:

aws-cdk/packages/cdk-assets/lib/private/archive.ts

Lines 8 to 16 in dac9bb3

    
           // The below options are needed to support following symlinks when building zip files: 
        
           // - nodir: This will prevent symlinks themselves from being copied into the zip. 
        
           // - follow: This will follow symlinks and copy the files within. 
        
           const globOptions = { 
        
             dot: true, 
        
             nodir: true, 
        
             follow: true, 
        
             cwd: directory, 
        
           };

Is there a way to provide the output as a zipfile instead of a directory? ie, do the zipping in the docker build to /asset-output.zip or similar – and then the CDK can use that.

My use case: I currently can't see any way to get CDK to zip up a directory that contains symlinks correctly (see #9251 ). If I could have full control over the zipping in a docker build, then that would at least be a workaround for this issue (and anyone else who wanted to control the zipping themselves).

Currently the only way seems to be (please correct me) to do the zipping externally to the CDK process, and then specify the zipfile as the asset.

mhart · 2020-07-27T23:09:42Z

For anyone else wanting to do this, I created my own construct (based largely on AssetStaging) to get around this problem – though I still believe it should be supported in some way (ideally by zipping assets and respecting follow options).

Usage:

const layer = new lambda.LayerVersion(this, "GitLayer", {
  code: new BundledZipCode({
    bundling: {
      image: cdk.BundlingDockerImage.fromRegistry("lambci/yumda:2"),
      user: "root",
      command: [
        "bash",
        "-c",
        `yum install -y git &&
        cd /lambda/opt &&
        find . -exec touch -h -t 198001010000 {} + &&
        zip -qXyr /asset-output/bundle.zip .`,
      ],
    },
  }),
});

(the touch and zip options will ensure reproducible zip contents, so new layer versions will only be added if the contents change)

And here's the construct:

class BundledZipCode extends lambda.AssetCode {
  constructor(options) {
    const { bundling, ...assetCodeOptions } = options;
    const stagingTmp = path.join(".", ".cdk.staging");
    if (!fs.existsSync(stagingTmp)) {
      fs.mkdirSync(stagingTmp);
    }
    const bundleDir = path.resolve(fs.mkdtempSync(path.join(stagingTmp, "asset-bundle-")));
    fs.chmodSync(bundleDir, 0o777);
    let user = bundling.user;
    if (!user) {
      const userInfo = os.userInfo();
      user = userInfo.uid !== -1 ? `${userInfo.uid}:${userInfo.gid}` : "1000:1000";
    }
    const volumes = [
      {
        hostPath: bundleDir,
        containerPath: cdk.AssetStaging.BUNDLING_OUTPUT_DIR,
      },
      ...(bundling.volumes || []),
    ];
    bundling.image._run({
      command: bundling.command,
      user,
      volumes,
      environment: bundling.environment,
    });
    super(path.join(bundleDir, "bundle.zip"), assetCodeOptions);
  }
}

SomayaB assigned nija-at May 11, 2020

jogold mentioned this pull request May 12, 2020

feat(lambda-nodejs): external and install modules #7870

Closed

nija-at assigned eladb and unassigned nija-at May 13, 2020

jogold added 2 commits May 16, 2020 14:27

Merge branch 'master' into lambda-docker-code

4afe2f5

environment

ca62f37

jogold added 2 commits May 16, 2020 17:49

fix integ test

116d985

.gitignore in python-lambda-handler

d2a66c8

jogold added 2 commits May 16, 2020 20:52

fix integ test 2

7120eba

start of README

13057c8

calcSourceHash

aad39af

AssetOptions

2d780e6

eladb previously requested changes May 17, 2020

View reviewed changes

PR feedback

3d3e7c2

jogold changed the title ~~feat(lambda): docker code~~ feat(lambda): add Docker support for asset code May 18, 2020

eladb changed the title ~~feat(lambda): add Docker support for asset code~~ feat(lambda): custom code bundling May 18, 2020

jogold commented May 18, 2020

View reviewed changes

packages/@aws-cdk/aws-lambda/lib/code.ts Outdated Show resolved Hide resolved

eladb previously requested changes May 18, 2020

View reviewed changes

eladb reviewed Jun 9, 2020

View reviewed changes

packages/@aws-cdk/core/lib/asset-staging.ts Show resolved Hide resolved

jogold added 2 commits June 9, 2020 10:17

extract to method + if/else

3b77e7f

remove fingerprint() method

1ff601d

eladb reviewed Jun 9, 2020

View reviewed changes

switch

17df113

jogold and others added 2 commits June 9, 2020 11:15

calculateHash(props: AssetStagingProps)

96df801

Merge branch 'master' into lambda-docker-code

16fe727

eladb approved these changes Jun 9, 2020

View reviewed changes

Merge branch 'master' into lambda-docker-code

8f372ff

mergify bot merged commit 888b412 into aws:master Jun 9, 2020

imtiazmangerah mentioned this pull request Jun 11, 2020

Custom asset bundling for lambda using docker results in output code that is owned by root #8489

Closed

jogold deleted the lambda-docker-code branch June 13, 2020 09:49

eladb mentioned this pull request Jul 26, 2020

[assets] symlinks are always followed on publish, FollowMode has no effect #9251

Open

		const hashType = props.assetHash ? AssetHashType.CUSTOM : props.assetHashType ?? AssetHashType.SOURCE;
		this.assetHash = this.calculateHash(hashType, props.assetHash);

feat(core,s3-assets,lambda): custom asset bundling #7898

feat(core,s3-assets,lambda): custom asset bundling #7898

Conversation

jogold commented May 10, 2020 • edited Loading

jogold commented May 10, 2020 • edited Loading

aws-cdk-automation commented May 10, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented May 16, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented May 16, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented May 16, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented May 16, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented May 16, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented May 16, 2020

AWS CodeBuild CI Report

eladb left a comment

Choose a reason for hiding this comment

jogold commented May 18, 2020

aws-cdk-automation commented May 18, 2020

AWS CodeBuild CI Report

eladb left a comment

Choose a reason for hiding this comment

eladb May 18, 2020

Choose a reason for hiding this comment

eladb May 18, 2020

Choose a reason for hiding this comment

jogold May 18, 2020

Choose a reason for hiding this comment

eladb Jun 9, 2020

Choose a reason for hiding this comment

jogold Jun 9, 2020

Choose a reason for hiding this comment

eladb Jun 9, 2020

Choose a reason for hiding this comment

eladb Jun 9, 2020

Choose a reason for hiding this comment

aws-cdk-automation commented Jun 9, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Jun 9, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Jun 9, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Jun 9, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Jun 9, 2020

AWS CodeBuild CI Report

aws-cdk-automation commented Jun 9, 2020

AWS CodeBuild CI Report

mergify bot commented Jun 9, 2020

aws-cdk-automation commented Jun 9, 2020

AWS CodeBuild CI Report

mergify bot commented Jun 9, 2020

imtiazmangerah commented Jun 11, 2020 • edited Loading

eladb commented Jun 11, 2020

imtiazmangerah commented Jun 11, 2020

eikeon commented Jun 12, 2020 • edited Loading

eladb commented Jun 14, 2020

rudaporto commented Jun 19, 2020 • edited Loading

mhart commented Jul 25, 2020

mhart commented Jul 27, 2020 • edited Loading

jogold commented May 10, 2020 •

edited

Loading

jogold commented May 10, 2020 •

edited

Loading

imtiazmangerah commented Jun 11, 2020 •

edited

Loading

eikeon commented Jun 12, 2020 •

edited

Loading

rudaporto commented Jun 19, 2020 •

edited

Loading

mhart commented Jul 27, 2020 •

edited

Loading