WIP: Node 20 lambda support [testing wanted]

[Sparticuz]

closes #185
closes #196

maybe #201?

also closes dependabot pr's:
closes #204
closes #203
closes #199
closes #198
closes #197

needed to add

• libplc4.so
• libplds4.so
• libnspr4.so

during this PR, AL2 also updated libs in a breaking way, so this will also include updated AL2 libs

[Sparticuz]

It looks like in order to support node 20, along with 16 and 18, the node 20 dependencies will be need be bundled separately.

[Sparticuz]

The reason I haven't merged this yet is because I haven't had time to independently test this outside of the automated testing.

[benasher44]

If you publish a pre-release to npm, I'd be happy to give it a spin! I'm not sure I can verify everything you're looking for, but I can verify that we can still use it to render web pages enough to generate PDFs from them :)

[Sparticuz]

@benasher44 thank you, i will try to do that today.

[ibqn]

looking forward for that release, much appreciated 👍

[Sparticuz]

I published 119.0.1-next.0 https://www.npmjs.com/package/@sparticuz/chromium/v/119.0.1-next.0 for testing

[Sparticuz]

I've also tagged it, so you can do @Sparticuz/chromium@next

[benasher44]

It's not working for us. Here are some logs (have dumpio: true):

[1130/195637.816999:FATAL:nss_util.cc(126)] nss_error=-8023, os_error=0
[1130/195637.816994:ERROR:nss_util.cc(124)] Error initializing NSS without a persistent database: NSS error code: -8023
[1130/195637.816914:ERROR:nss_util.cc(239)] Error initializing NSS with a persistent database (sql:/tmp/.pki/nssdb): NSS error code: -8023
[1130/195637.648402:WARNING:angle_platform_impl.cc(49)] RendererVk.cpp:1962 (initialize): ANGLE VMA version: 2003000

I'm not sure if that's helpful (in the past, hasn't been that helpful). If there are tweaks I could make to get you more logs, let me know!

[ibqn]

great, I can confirm that it is working for us with puppeteer-core v21.5.2 and @sparticuz/chromium v119.0.1-next.0 within chromium-v119.0.1-next.0-layer.zip layer in node v20 lambda function

[Sparticuz]

It's not working for us. Here are some logs (have dumpio: true):

[1130/195637.816999:FATAL:nss_util.cc(126)] nss_error=-8023, os_error=0
[1130/195637.816994:ERROR:nss_util.cc(124)] Error initializing NSS without a persistent database: NSS error code: -8023
[1130/195637.816914:ERROR:nss_util.cc(239)] Error initializing NSS with a persistent database (sql:/tmp/.pki/nssdb): NSS error code: -8023
[1130/195637.648402:WARNING:angle_platform_impl.cc(49)] RendererVk.cpp:1962 (initialize): ANGLE VMA version: 2003000

I'm not sure if that's helpful (in the past, hasn't been that helpful). If there are tweaks I could make to get you more logs, let me know!

Are you using custom args?

[benasher44]

using puppeteer 21.5.1, here is how we're launching

import chromium from "@sparticuz/chromium";
import puppeteer from "puppeteer-core";

await puppeteer.launch({
  args: chromium.args.concat(["--single-process"]),
  defaultViewport: null,
  executablePath: await chromium.executablePath(),
  headless: true,
  dumpio: true,
})

[benasher44]

@Sparticuz after some local testing, I found it was failing to launch due to being unable to load nss:

/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory
Error: Failed to launch the browser process!
/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

After adding dnf install -y nss to my build, it seems to work. This lines up with changes for AL2023 as well, since nss no longer comes pre-installed (did before). I still have to do some testing in the production lambda environment though, so I'll circle back once that's done!

In the meantime, would it make sense to do publish another build with nss built-in?

[benasher44]

@ibqn were you testing https requests by chance? or only http?

[Sparticuz]

libnss3.so and libnssutil3.so are both included in the aws.tar.br package, so they should get extracted when run on Lambda

[benasher44]

I just tried again with AWS_EXECUTION_ENV=1 node <args> when testing locally, which should be enough to get it to use aws.tar.br, right? I got the same ld error. Anything else I should try for a better local test?

[Sparticuz]

No, I think you'll need AWS_EXECUTION_ENV=AWS_Lambda_nodejs_20.x. That should satisfy both of the functions here

[benasher44]

Ahh! Okay it's loading nss now, but I am also now reproducing the nssutil error:

[1204/152611.415970:ERROR:nss_util.cc(239)] Error initializing NSS with a persistent database (sql:/root/.pki/nssdb): NSS error code: -8023
[1204/152611.416130:ERROR:nss_util.cc(124)] Error initializing NSS without a persistent database: NSS error code: -8023
[1204/152611.416137:FATAL:nss_util.cc(126)] nss_error=-8023, os_error=0

Would you mind publishing a build with nss but without nss-util? Seeing as I was able to get it to work without explicitly install nss-util, I wonder if nss comes with "enough" for this to work but nss-util has a mismatched version of something (being vague because I really don't know all that much about nss + nss-util, but have worked with packages enough to smell this problem 👃 ).

[Sparticuz]

Github is filtering brotli files, so before using this, just delete the .gz file extension so the file is aws.tar.br and put it in the bin folder. All I did with this is delete the libnssutil3.so file.
aws.tar.br.gz

[benasher44]

Okay so my hunch was wrong. Removing nss-util results in it failing to load nss because it can't load nss-util. However, if I dnf install nss, I noticed that the sha1sum of the nss-util installed via dnf is different than the one extracted to /tmp/aws/lib. If I copy the dnf installed version over the one extracted to /tmp/aws/lib, it works (also confirmed that it still failed with the same error, after running dnf install nss, which confirmed it was still trying to load the one in /tmp/aws/lib)

[Sparticuz]

OK, that's good info. The files in aws.tar.br are older files from 2022. I will update them and push a new file. Hopefully they will continue to work in AL2.

[benasher44]

I compared the sha1sums of the libs in the curl'd version of the aws.tar.br in my most recent comment and the one that comes with -next.1. These libraries differed:

• libexpat.so.1
• libnss3.so
• libnssutil3.so

In other words, -next.1 is using different versions of these libraries somehow

[Sparticuz]

@benasher44 Ok, so the one you used was actually the one with the updated libs. We will need to verify that these ones still work in al2

[benasher44]

I tested on AL 2 and got this output:

/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libnssutil3.so)

It looks like we'll have to further split the libraries for AL2 and AL2023

[Sparticuz]

I tested on AL 2 and got this output:

/tmp/chromium: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/aws/lib/libnssutil3.so)

It looks like we'll have to further split the libraries for AL2 and AL2023

Hmm, I was afraid that might happen. Been busy with work the past few days, hopefully will build a new package today or tomorrow.

[Sparticuz]

I've got it working for me in testing on my local machine, which I haven't been able to get until now. I've got some cleanup to do, then will push another test today

[Sparticuz]

I just published @Sparticuz/chromium@119.0.1-next.2

I'm fairly confident it's working as expected!

[benasher44]

Confirmed working on AL2023!

[onassar]

Thanks for your work and testing here all.

@benasher44 I saw in your example above you're simply importing puppeteer-core

import puppeteer from "puppeteer-core";

Does this need to be imported as a separate layer? I'm able to add the chromium layer (vv119.0.2) but things are failing out with the following when I check for puppeteer:

Cannot find package 'puppeteer-core' imported from /var/task/index.mjs

[benasher44]

Oh I have a custom build that installs deps from a package.json, which included puppeteer-core

[onassar]

Got it thanks @benasher44. For posterity's sake, I ended up having to build puppeteer-core v21_5_1 and including that as a layer. I'll post the ARN once I'm able to make it public for others.

[AnYoung627]

Got it thanks @benasher44. For posterity's sake, I ended up having to build puppeteer-core v21_5_1 and including that as a layer. I'll post the ARN once I'm able to make it public for others.

@onassar did you ever make that public ARN for a layer?

[onassar]

@AnYoung627 here's what I made available: https://onassar.github.io/blogs/web/2024/01/17/public-aws-lambda-function-layers/

Hopefully you find it useful.