When multiple Lambda functions share code, build a single asset shared between them

[tomfaulhaber]

Use case:

In order to build lambdas that use the Python science stack, we need to build a giant (50+MB) zip file with all the dependencies. We then add all of our Python functions to that zip and specify the same zip as the code for each lambda with a different handler.

The problem:

Each of the lambda functions that I create have a unique asset object (even if I create a single, shared lambda.Code object). Each of these objects creates an S3 object. So if I have ten lambda functions, I will upload ten separate asset objects at 50+ MB each.

Potential solutions:

1. Make each lambda.AssetCode object unique by checking to see if asset is already set before creating a new one in bind(). This means that users would need to explicitly create the shared Code object and reuse it in each lambda definition.
2. Make it unique to the path in the lambda.AssetCode by adding a map to lambda.AssetCode and pulling the same asset into multiple AssetCode objects. In this scenario, users could simply reference the same file/directory.
3. Build the solution lower down in the assets.Asset class. In this example, we would cache the path and make sure that we only do the upload once.

I think the last solution is the only real solution, because the other two solutions don't handle the case where you create two lambdas, but only wire the second one into your app which is likely to happen in various scenarios.

[rix0rrr]

We can also employ server-side copies (upload once then copy out)

[tomfaulhaber]

@rix0rrr for my use case that would be bad since it would just make lots of extra copies and run up my S3 bill. Playing with the CDK has already created 27GB of stuff in my CDK assets bucket.

FWIW, I implemented option 1 and it seems to work great in my case, but I think it would fail if I didn't use my first lambda in my app.

[rix0rrr]

I understand. But, the reason we have multiple copies of the same file for multiple assets is that they are in different prefixes.

We have to allow that they can evolve individually (maybe they're the same by accident) and the consuming resources need read permissions on the entire prefix so that jobs started on an old asset don't suddenly lose perms to read their file if the "current" version changes.

Not applicable to Lamda so much but definitely to CodeBuild for example

[rix0rrr]

I see the issue with AssetCode. It's being silly by creating the same Asset object multiple times, and every Asset object triggers an upload. The proper solution would be to make AssetCode a construct, but in order not to break API compatibility I'm going to cache the Asset object like you've been doing locally.

I will trigger a server-side copy if the same Asset data is uploaded multiple times to different Asset objects, but I'm not going to deduplicate inside the S3 bucket (in order to not run afoul of permission issues as mentioned before).

I'm sorry about your S3 bill, but I think the more general solution to this will be some kind of garbage collection facility (as opposed to frugally uploading, which helps in the short term but not in the long run).

[AaronHarris]

What's the workaround here? Just pass in the same lambda.AssetCode() object ref to different lambdas?