Layer does not appear to function in Lambda nodejs10.x

[ryanblock]

Git layer fails in the new Lambda nodejs10.x container released this week.

Repro steps:

• I used arn:aws:lambda:us-west-1:553035198032:layer:git:5 in us-west-1
• Lambda configured with nodejs10.x
• Ran the following child_process.exec git clone command (from within /tmp): git clone https://$USERNAME:$TOKEN@github.com/$OWNER/$REPO.git
• Received the following error: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory

This isn't particular surprising considering the messaging AWS has been doing this week about the underlying library changes in Lambda nodejs10.x.

Reference:

• https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/
• https://aws.amazon.com/about-aws/whats-new/2019/05/aws_lambda_adds_support_for_node_js_v10/
• https://aws.amazon.com/amazon-linux-2/

Their announcement email:

Hello,

We are updating the AWS Lambda and AWS Lambda@Edge execution environment to include recent versions of Amazon Linux and software packages.

A majority of functions will seamlessly benefit from the enhancements in this update without requiring you to take any action. However, in rare cases package updates may introduce compatibility issues. Functions that contain libraries or application code compiled against very specific underlying OS packages, specifically those for openssl, glibc, or other system libraries, may potentially be impacted.

Starting May 14, 2019, you can test your functions with the new execution environment. From May 21, 2019, all new functions or updates of existing functions will use the new execution environment. Your existing functions will automatically migrate to using the new execution environment on June 11, 2019.

See the following blog post to learn more about the update including the timeline, testing guidelines, and how you can prepare[1].

[1] https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/

Sincerely,
Amazon Web Services

Edit: per tweeters, ratcheting down urgency since AWS Linux 2 isn't being forced into other Lambda container images yet; re-scoped issue more specifically to nodejs10.x

[mhart]

Yeah, that email you've pasted and the first link there in your issue are about the changes to the existing (non-nodejs10.x) runtimes. This layer should continue to work in existing runtimes – I've tested it lightly – feel free to open a separate issue if you see any incompatibilities there.

In terms of working in the nodejs10.x and any newer runtimes, that's a TODO – I'll update here when I get around to building and bundling all the extra pieces needed.

[ryanblock]

Yeah, I think before our conversation on Twitter I was confused by their language about the new AMI vs Linux 2 end (which I took to be one and the same).

I’m guessing you may have played with the new AMI, if so any differences / breaking changes? Would you like me to open a second issue for tracking that?

[mhart]

Yeah, all docker-lambda images have been updated with the new AMI, and this layer worked with it last I tried. Probably worth giving the ssh integration a decent workout, but git itself over https seemed to be fine.

Haven't ported any production CI loads over yet though, that'll probably be the test.

Removed binaries and libraries are here: https://gist.github.com/mhart/94f18d9327f5a101e8c16c47b7283dcc

But there were also some upgrades which I haven't documented anywhere:

• libssl going from 1.0.1k to 1.0.2k
• libbfd-2.23.52.0.1-55.65 -> libbfd-2.25.1-31.base.66
• libcurl.so.4.4.0 -> libcurl.so.4.5.0
• libelf-0.163 -> libelf-0.168
• libopcodes-2.23.52.0.1-55.65 -> libopcodes-2.25.1-31.base.66
• libstdc++.so.6.0.19 -> libstdc++.so.6.0.24
• python3.4 -> python3.6

Again, I don't think any of these should affect this layer – the few libs it does link to (libcurl, libstdc++) should still be compatible.

[ryanblock]

Thanks for all this information and research @mhart, so great. 🙌🏻

Feels on a single git package that bundles all its own dependencies, and is compatible with both container envs? Based on some tests I ran fully bundling some other Linux utilities a while back, I'm guessing that would be a pretty massive filesize...

[mhart]

The Amazon Linux 2 version will definitely be larger – not really a fan of foisting that on all the other runtime users. Besides, it'll be really fiddly trying to make it backwards compatible with, say, the libc version on Amazon Linux 1, while also including libraries that don't exist on Amazon Linux 2 but that do exist on Amazon Linux 1.

[mhart]

I'm guessing you guys use this in begin to do your CI builds?

[ryanblock]

The Amazon Linux 2 version will definitely be larger – not really a fan of foisting that on all the other runtime users

Yeah, that's legit. Presumably future runtimes in Lambda will be on AWS Linux 2, but for now that'd equate to a hard fork for a single runtime.

We use it in Begin for CI, yes, albeit indirectly. We use lambda-git (basically the package version of your layer, and bundles your git archive) because layer support in @architect is still nascent (and, to a lesser extent, vendor specific).

[ryanblock]

Any thoughts here? (Anything we can do to help?)

[mhart]

Are you wanting to move Begin to the nodejs10.x runtime?

[ryanblock]

@mhart sorry, missed this message. Yes, def, we have some functions that are stranded on Node 8 without a Node 10 image-compatible version of this layer.

Also, with last week's node8.10 EOL announced, seems like a good time.

If from your perspective you no longer want to maintain this layer, happy to help (or fork). Mostly just looking to resolve. Thanks!

[mhart]

Fixed thanks to the launch of https://github.com/lambci/yumda

Created a new layer named git-lambda2 for nodejs10.x

ARN: arn:aws:lambda:<region>:553035198032:layer:git-lambda2:1

[ryanblock]

Yooo, yumda is great. Have been thinking about if/how one could automate package → layer creation at greater scale, is that part of where you want to take it?

[mhart]

Yeah have thought about that a lot.

I think at the moment layers aren't all that great to be built/published at scale – ie it wouldn't be trivial to have a layer for each yum package.

I think we'll need to wait and see where AWS take SAR and see if they start to build in better support for layers.

In the meantime, yumda will at least make it easier to create layers (and ppl can publish them if they want)