[BUG] No longer working in Lambda #271

fire015 · 2024-04-30T12:41:24Z

package.json:

    "@sparticuz/chromium": "123.0.1",
    "puppeteer-core": "22.6.4"

code:

  chromium.setGraphicsMode = false;

  console.log("## Launching browser");
  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath(),
    headless: chromium.headless,
    ignoreHTTPSErrors: true,
  });

  console.log("## Creating new page");
  const page = await browser.newPage();

  console.log("## Setting user agent");
  await page.setUserAgent("Lambda PDF");

  console.log("## Going to " + url);
  await page.goto(url, {
    waitUntil: ["domcontentloaded", "networkidle0"],
  });

It's freezing after "Creating new page" appears in the console logs.

Lambda 18.x (tried with 20.x too) with 2GB memory
Runtime version ARN: arn:aws:lambda:eu-west-2::runtime:b475b23763329123d9e6f79f51886d0e1054f727f5b90ec945fcb2a3ec09afdd

The text was updated successfully, but these errors were encountered:

fire015 · 2024-04-30T13:13:09Z

Downgrading to:

    "@sparticuz/chromium": "122.0.0",
    "puppeteer-core": "22.5.0"

Seems to have worked. Leaving this open just to verify I didnt make a mistake selecting the wrong package versions?

leepa · 2024-04-30T15:24:27Z

This issue hit us today but we've not updated for a while so it seems to effect old versions too. Is there anyway to make this project dump a stacktrace when the browser crashes?

Sparticuz · 2024-04-30T16:03:18Z

Make sure to leave graphics mode enabled. The option will be fixed in the next version. I've been traveling so I haven't had time to update to chromium 124.

Sparticuz · 2024-04-30T16:04:15Z

This issue hit us today but we've not updated for a while so it seems to effect old versions too. Is there anyway to make this project dump a stacktrace when the browser crashes?

You can add puppeteer.launch({dumpio:true}) in the object.

shubhamvyapar · 2024-04-30T18:57:04Z

We have also started facing same issue. We have not update package version from 3 months. While trying to figure out the issue, we found that it is not working only when html contains https URLs as we had some https URLs in img tag and link tag. With http urls, it is working fine.

We had the following package.json

 "@sparticuz/chromium": "^109.0.5",
 "puppeteer-core": "^19.4.0",

Same thing is working fine with https URLs after updating package version to (as mentioned by @fire015 )

"@sparticuz/chromium": "122.0.0",
"puppeteer-core": "22.5.0"

ryan-concora · 2024-04-30T20:46:01Z

Having the same issue, not working in our production app.

urvik-bhalala · 2024-04-30T21:25:04Z

Downgrading to:
    "@sparticuz/chromium": "122.0.0",
    "puppeteer-core": "22.5.0"
Seems to have worked. Leaving this open just to verify I didnt make a mistake selecting the wrong package versions?

Solved.

In my case, Downgrading to version as you describe and additionally I have to install "web-streams-polyfill" package. And added this line : global.ReadableStream = require('web-streams-polyfill').ReadableStream;

leepa · 2024-04-30T21:40:36Z

We've worked around it by rolling back to the previous Lambda Runtime ARN.

ryan-concora · 2024-04-30T22:19:07Z

We've worked around it by rolling back to the previous Lambda Runtime ARN.

How do you do this with Serverless framework?

iLykaPaul · 2024-04-30T22:19:18Z

We also ran into this today, the version changes suggested above:

"@sparticuz/chromium": "122.0.0",
"puppeteer-core": "22.5.0"

and in our case an upgrade of the Chromium Lambda Layer to latest was the change combination that got us back up and running.

ryan-concora · 2024-04-30T22:20:58Z

Ours doesn't seem to want to pull from s3

"An executablePathorchannelmust be specified forpuppeteer-core" name:"Error" stacktrace:"at assert (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/util/assert.js:18:15) at ChromeLauncher.computeLaunchArguments (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/node/ChromeLauncher.js:102:36) at async ChromeLauncher.launch (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/node/ProductLauncher.js:70:28) at async CoverPageGenerator.generatePDF (/var/task/src/CoverPage.js:60:19) at async _getCoverPage (/var/task/src/zip.js:72:10) at async exports.createSubmittal (/var/task/src/zip.js:33:21) at async Worker.work (/var/task/src/worker.js:25:27) at async post (/var/task/src/handler.js:19:3)" type:"ERROR_TYPE_UNCAUGHT"

xanharris · 2024-05-01T02:02:38Z

We've worked around it by rolling back to the previous Lambda Runtime ARN.

How do you do this with Serverless framework?

@ryan-concora
Specify the run time management at the "provider" or "function" level in your serverless.yml file:

functions:
  hello:
    runtimeManagement:
      mode: manual
      arn: arn:aws:lambda:us-east-1::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9

The ARN version in the example above works for me (The offending ARN is mentioned in OP: #271 (comment))
Reference: https://www.serverless.com/framework/docs-providers-aws-guide-functions

gaurav112 · 2024-05-01T08:33:11Z

Facing the same issue.

Error: spawn Unknown system error -8

"@sparticuz/chromium": "^110.0.0",
"puppeteer": "19.6.0"

const browser = await puppeteer.launch({
    dumpio: true,
    args: chromium.args,
    executablePath: await chromium.executablePath(),
    headless: true,
    ignoreHTTPSErrors: true,
})

Tried upgrading to

Downgrading to:
    "@sparticuz/chromium": "122.0.0",
    "puppeteer-core": "22.5.0"
Seems to have worked. Leaving this open just to verify I didnt make a mistake selecting the wrong package versions?

This still did not work. Any other ideas ??

ryan-concora · 2024-05-01T08:44:32Z

Facing the same issue.

Error: spawn Unknown system error -8
"@sparticuz/chromium": "^110.0.0",
"puppeteer": "19.6.0"
const browser = await puppeteer.launch({
    dumpio: true,
    args: chromium.args,
    executablePath: await chromium.executablePath(),
    headless: true,
    ignoreHTTPSErrors: true,
})
Tried upgrading to
Downgrading to:
    "@sparticuz/chromium": "122.0.0",
    "puppeteer-core": "22.5.0"
Seems to have worked. Leaving this open just to verify I didnt make a mistake selecting the wrong package versions?
This still did not work. Any other ideas ??

Our solution:

Added lambda layer, pulling from s3 doesn't seem to be working
Used this as executable path (examples need updating chromium.executablePath('/opt/nodejs/node_modules/@sparticuz/chromium/bin';
Removed "@sparticuz/chromium-min": "122.0.0"
Added as dev dep "@sparticuz/chromium": "122.0.0"
"puppeteer-core": "22.5.0"
Used v122.0.0 lambda layer

LinusU · 2024-05-01T08:59:45Z

Removing our disabling of the graphics mode solved this for us:

- chromium.setGraphicsMode = false
+ // FIXME: re-enable in Chromium 124
+ // ref: https://github.com/Sparticuz/chromium/issues/271#issuecomment-2085784736
+ // chromium.setGraphicsMode = false

with the following versions:

    "@sparticuz/chromium": "123.0.1",
    "playwright-core": "1.43.1",

edit: actually, not sure about this, we're still seeing some errors

2803media · 2024-05-01T14:08:33Z

We've worked around it by rolling back to the previous Lambda Runtime ARN.

How do you do this with Serverless framework?

change this in the Runtime management configuration:

JakeDluhy · 2024-05-01T15:59:03Z

FWIW I was able to get it working with:

"@sparticuz/chromium": "122.0.0",
"puppeteer-core": "^21.1.1",
"puppeteer-extra": "^3.3.6",
"puppeteer-extra-plugin-stealth": "^2.11.2",

adefemi · 2024-05-01T16:36:33Z

setting this works
chromium.setGraphicsMode = true;

AkbarHabeeb · 2024-05-01T17:35:18Z

@Sparticuz Little curious to understand why the below version of packages are getting into an error when the latest package version was released? What is the breaking change we have?
"@sparticuz/chromium": "^106.0.2" "puppeteer-core": "^18.0.5"
@shubhamvyapar For the suggested downgrade version "@sparticuz/chromium": "122.0.0", "puppeteer-core": "22.5.0", looks like we need Node 18, Can you kindly suggest any compatible version for Node 16 runtime env?

We got into an error saying Navigation failed because browser has disconnected! seems like chrome crashed. Is this the same issue?

jmartinezgdev · 2024-05-01T22:26:35Z

setting this works chromium.setGraphicsMode = true;

This worked also for me
chromium.setGraphicsMode = true;

puppeteer-core 22.6.0
@sparticuz/chromium 123.0.1,

casyalex · 2024-05-02T08:19:04Z

I am curious about the root cause, seems it relate to SSL lib update, but why GraphicsMode can make it solved.

vishwassingh47 · 2024-05-02T08:40:44Z

We got into an error saying Navigation failed because browser has disconnected! seems like chrome crashed. Is this the same issue?

Yes we were getting the same error "Navigation failed because browser has disconnected!", but it was happening in those cases only where were are loading some asset using https, http and setOfflineMode(true) works fine.

gaurav112 · 2024-05-02T08:51:51Z

This works for us.

We migrated to a previous version of Runtime on lambda(Node.Js 18.*)

ARN: arn:aws:lambda:ap-south-1::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9

package.json for compatibility.

"@sparticuz/chromium": "^119.0.0",
"puppeteer": "^21.5.2"

bjoepfeiffer · 2024-05-02T16:15:54Z

I am having a similar issue to some of the more recently posted comments in this thread:

Navigation failed because browser has disconnected...

Others are saying that downgrading versions helped them, but I am starting to wonder if there is something else at play here. This has crept up for many of us on all different versions.

Is there a corrupted browser object being cached between lambda executions and deploys?

Is it the downgraded version or the deploy of the downgraded version that is actually fixing the issue? I am wondering if this issue creeps back up for those who have said it has been fixed with a version downgrade.

We started seeing this with these versions, which were previously working for a year. Then all of a sudden we started getting the error.

"@sparticuz/chromium": "114.0.0",
"puppeteer-core": "20.7.3"

Did AWS change something with lambdas and lambda layers that is biting all of us?

Sparticuz · 2024-05-02T16:19:29Z

Did AWS change something with lambdas and lambda layers that is biting all of us?

Yes, AWS updated some libraries that are causing issues with the libraries included in this package. I'm in contact with the team at AWS and we will be working on a solution.

I would recommend pinning the lambda execution environment while this issue gets sorted.

tomasalvarez13 · 2024-05-02T16:19:44Z

I'm having a similar issue; it suddenly stopped working and started giving us this error code:

The input directory "/opt/nodejs/node_modules/@sparticuz/chromium/bin/" does not exist.
I've tried all the solutions mentioned above, but none of them worked.

tomasalvarez13 · 2024-05-02T16:58:14Z

I rolled back manualy the runtime version and it worked

raducretu · 2024-05-02T17:05:20Z

on one of my projects it works, on a different one, same settings same packages it does not work

tyanko1 · 2024-05-02T17:24:36Z

@tomasalvarez13 which runtime version worked for you, specifically?

raducretu · 2024-05-02T17:26:48Z

@tomasalvarez13 which runtime version worked for you, specifically?

for me the one that worked on one place and didn't work on a different project is:
node 18.x
"puppeteer-core": "^22.6.2",
chromium-v123.0.1-layer.zip

same configuration on one account works on the other one it does not work, it goes timeout on:
const page = await browser.newPage();

tomasalvarez13 · 2024-05-02T17:26:54Z

@tyanko1 one from 27/04/2024

LinusU · 2024-05-03T08:38:05Z

@raducretu you can specify the exact version of the runtime under something called "runtime management configuration". If you look at your logs from when things worked (a few days ago) you should see which arn they are using as it's printed in the INIT_START. Look for your old arn that worked, and then go in and change it to that for your Lambda.

More info here: https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html

fernedingd · 2024-05-03T12:21:26Z

For everyone that is still experiencing issues after pinning the runtime version: check if you're properly closing the browser at the end of your lambda.

Our function was running fine for over a year with puppeteer-core 19.7.4 and chromium 111.0.0. After pinning the runtime version, it was kind of working again, but crashed regularly under load. Turns out we never closed the browser, which hasn't been an issue before. Changing

try {
    const browser = await puppeteer.launch({
        // ...
    });
    const page = await browser.newPage();
    // ... actions
    await page.close();
} catch {
    // ... error handling
}

to

var browser;
try {
    browser = await puppeteer.launch({
        // ...
    });
    const page = await browser.newPage();
    // ... actions
    for (const page of await browser.pages()) {
        await page.close();
    }
} catch {
    // ... error handling
} finally {
    if (browser) {
        await Promise.race([browser.close(), browser.close(), browser.close()]);
    }
}

along with pinning the version fixed the issues for us, it's been running fine for a few hours now.

bjoepfeiffer · 2024-05-03T12:25:11Z

We were running our pdf export lambdas on nodejs18.x. Upgrading to nodejs20.x fixed this issue for now.

LinusU · 2024-05-03T13:03:34Z

@fernedingd what is the reasoning behind using await Promise.race([browser.close(), browser.close(), browser.close()]) instead of just await browser.close()? 🤔

fernedingd · 2024-05-03T13:20:21Z

@LinusU we initially removed the await browser.close() call because it wasn't working reliably and the funktion worked without it. Calling close multiple times was a suggestion from the FAQs. A single call should be enough, but I didn't want to risk having those issues again during peak production workloads over the weekend

RubensMalaguteO7S · 2024-05-05T01:20:41Z

We were running our lambda on nodejs16.x and it couldn't be changed for internal reasons. The solution found was to revert the lambda runtime to a version from before the problem and update chromium and puppeteer-core versions to:

"@sparticuz/chromium": "122.0.0"
"puppeteer-core": "17.1.3"

solancer · 2024-05-07T13:27:38Z

This works for us.

We migrated to a previous version of Runtime on lambda(Node.Js 18.*)
ARN: arn:aws:lambda:ap-south-1::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9
package.json for compatibility.
"@sparticuz/chromium": "^119.0.0",
"puppeteer": "^21.5.2"

This worked, thankyou

hasan-wajahat · 2024-05-08T20:57:22Z

👀

alejandro-gstack · 2024-05-08T21:41:10Z

@raducretu you can specify the exact version of the runtime under something called "runtime management configuration". If you look at your logs from when things worked (a few days ago) you should see which arn they are using as it's printed in the INIT_START. Look for your old arn that worked, and then go in and change it to that for your Lambda.

More info here: https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html

This worked for us! Lambdas are up and running after a couple of days of 🔥... But waiting for a definitive solution.

davidhongpulsifi · 2024-05-09T12:04:36Z

INIT_START Runtime Version: nodejs:18.v24	Runtime Version ARN: arn:aws:lambda:eu-central-1::runtime:c09960ad0af4321e1a7cf013174f7c0d7169bf09af823ca2ad2f93c72ade708a

INIT_START Runtime Version: nodejs:18.v24	Runtime Version ARN: arn:aws:lambda:ap-southeast-1::runtime:c09960ad0af4321e1a7cf013174f7c0d7169bf09af823ca2ad2f93c72ade708a

Working with runtime above by configure lambda runtime manually

vince0946 · 2024-05-10T02:19:17Z

We've worked around it by rolling back to the previous Lambda Runtime ARN.

How do you do this with Serverless framework?

serverless: maybe need version 3.x+ since it is a new feature last year, https://www.serverless.com/framework/docs-providers-aws-guide-functions
aws cli: https://docs.aws.amazon.com/cli/latest/reference/lambda/put-runtime-management-config.html
You should use lambda runtime management config, and find the old lambda arn in previous log which you could success running. "Runtime Version ARN: xxxx"

davidhongpulsifi · 2024-05-10T02:23:07Z

We've worked around it by rolling back to the previous Lambda Runtime ARN.

How do you do this with Serverless framework?

serverless: maybe need version 3.x+ since it is a new feature last year, https://www.serverless.com/framework/docs-providers-aws-guide-functions aws cli: https://docs.aws.amazon.com/cli/latest/reference/lambda/put-runtime-management-config.html You should use lambda runtime management config, and find the old lambda arn in previous log which you could success running. "Runtime Version ARN: xxxx"


import { custom } from './custom';
import { functions } from './functions';
import { plugins } from './plugins';
import { stepFunctions } from './step-functions';

export const main = {
    service: some-report-fn',
    frameworkVersion: '3',
    useDotenv: true,
    configValidationMode: 'error',
    package: {
        individually: true,
    },
    provider: {
        name: 'aws',
        runtime: 'nodejs18.x',
        runtimeManagement: {
            mode: 'manual',
            arn: 'arn:aws:lambda:${opt:region}::runtime:${ssm:/configs/LAMBDA_NODEJS18_RUNTIME_ARN}',
        },

how we do now is store the ARN into SSM then map to runtimemanagement > arn

activeliang · 2024-05-16T16:18:28Z

setting this works chromium.setGraphicsMode = true;

This worked also for me chromium.setGraphicsMode = true;

puppeteer-core 22.6.0 @sparticuz/chromium 123.0.1,

This worked also for me

denlukia · 2024-05-19T00:57:38Z

To everyone who's reading this thread looking for answers for Vercel, what eventually worked for me:

(I'm not sure what here is responsible for a fix)

Set engine in project settings to 20.x
Switched from pnpm to npm
Updated @sparticuz/chromium-min to newest (123.0.1 as of writing)
Updated chromium tar link also to newest (chromium-v123.0.1-pack.tar as of writing)

Just in case: puppeteer-core is "^21.3.8"

krewetka · 2024-06-11T07:27:00Z

Yes, AWS updated some libraries that are causing issues with the libraries included in this package. I'm in contact with the team at AWS and we will be working on a solution.

I would recommend pinning the lambda execution environment while this issue gets sorted.

@Sparticuz any news on this? Pinning to older version for a long run is not perfect solution for security reasons.

fire015 added the bug Something isn't working label Apr 30, 2024

alecmocatta mentioned this issue May 1, 2024

page.goto() throwing error continuously in AWS Lambda with playwright-aws-lambda JupiterOne/playwright-aws-lambda#74

Open

Sparticuz pinned this issue May 2, 2024

activeliang mentioned this issue May 16, 2024

Secure URLs Crashing Chromium in Local SAM Testing #196

Closed

krewetka mentioned this issue May 17, 2024

version 33 stop working on lambda from 2 day shelfio/chrome-aws-lambda-layer#66

Open

brianfeucht mentioned this issue May 23, 2024

Add support for net8 Lambda Run Time litmus/HeadlessChromium.Puppeteer.Lambda.Dotnet#123

Draft

[BUG] No longer working in Lambda #271

[BUG] No longer working in Lambda #271

Comments

fire015 commented Apr 30, 2024 • edited

fire015 commented Apr 30, 2024

leepa commented Apr 30, 2024 • edited

Sparticuz commented Apr 30, 2024

Sparticuz commented Apr 30, 2024 • edited

shubhamvyapar commented Apr 30, 2024 • edited

ryan-concora commented Apr 30, 2024

urvik-bhalala commented Apr 30, 2024

leepa commented Apr 30, 2024

ryan-concora commented Apr 30, 2024

iLykaPaul commented Apr 30, 2024 • edited

ryan-concora commented Apr 30, 2024

xanharris commented May 1, 2024 • edited

gaurav112 commented May 1, 2024

ryan-concora commented May 1, 2024

LinusU commented May 1, 2024 • edited

2803media commented May 1, 2024

JakeDluhy commented May 1, 2024

adefemi commented May 1, 2024

AkbarHabeeb commented May 1, 2024

jmartinezgdev commented May 1, 2024

casyalex commented May 2, 2024

vishwassingh47 commented May 2, 2024

gaurav112 commented May 2, 2024

bjoepfeiffer commented May 2, 2024

Sparticuz commented May 2, 2024

tomasalvarez13 commented May 2, 2024

tomasalvarez13 commented May 2, 2024

raducretu commented May 2, 2024

tyanko1 commented May 2, 2024

raducretu commented May 2, 2024

tomasalvarez13 commented May 2, 2024

LinusU commented May 3, 2024

fernedingd commented May 3, 2024

bjoepfeiffer commented May 3, 2024

LinusU commented May 3, 2024

fernedingd commented May 3, 2024 • edited

RubensMalaguteO7S commented May 5, 2024

solancer commented May 7, 2024

hasan-wajahat commented May 8, 2024

alejandro-gstack commented May 8, 2024

davidhongpulsifi commented May 9, 2024 • edited

vince0946 commented May 10, 2024 • edited

davidhongpulsifi commented May 10, 2024 • edited

activeliang commented May 16, 2024

denlukia commented May 19, 2024

krewetka commented Jun 11, 2024

fire015 commented Apr 30, 2024 •

edited

leepa commented Apr 30, 2024 •

edited

Sparticuz commented Apr 30, 2024 •

edited

shubhamvyapar commented Apr 30, 2024 •

edited

iLykaPaul commented Apr 30, 2024 •

edited

xanharris commented May 1, 2024 •

edited

LinusU commented May 1, 2024 •

edited

fernedingd commented May 3, 2024 •

edited

davidhongpulsifi commented May 9, 2024 •

edited

vince0946 commented May 10, 2024 •

edited

davidhongpulsifi commented May 10, 2024 •

edited