Followed "Run Chrome extensions with Python Selenium on AWS Lambda" and it works Locally but not once Deployed to AWS

[JeffreyKozik]

I followed the "Run Chrome extensions with Python Selenium on AWS Lambda" tutorial which by the way was super helpful, and it worked for me when I use sam local invoke to run it locally. However, once I use 'sam deploy" to deploy it and try testing it on AWS, I get the error:

"errorMessage": "Message: unknown error: Chrome failed to start: exited abnormally\n (unknown error: unable to discover open pages)\n (The process started from chrome location opt/chrome/stable/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n

which I've tried to solve by switching versions of Chrome, Chromium and the driver that I use, but I keep running into the same problem. I was wondering if you've run this on AWS and if it worked for you and if so what I might be doing wrong. Thanks!

[rehanhaider]

Hey Jeff. At the time of writing this article it did work, but it is extremely unstable and I'm not surprised you're facing crashes. The problem is usually either correctly placing chrome/chromedriver with relevant permissions or Lambda's own runtime compilation and configuration which is made worse by unstable Selenium-Python bindings.

Bottom line is, the docker file & script will work well on ECS but will be unstable on Lambda and is not fit for production.

Since then I'm exploring using Puppeteer which has more stable APIs and has a better chance of working on pure Lambda.

[JeffreyKozik]

Okay interesting thanks for the breakdown that makes a little bit more sense about why it doesn't work on AWS Lambda then. I'm curious if any of the other cloud platforms like Google Cloud Run have better support for this or not

[rehanhaider]

My gut feel is, Cloud Run should be able to run it better than Lambda. You might want to re-purpose the article about running selenium in WSL and use it to build a Dockerfile that runs on GCR. Let me know if you succeed or need help, I might be able to eek out another guide from it.

[JeffreyKozik]

I was thinking about doing in on GCR but I decided against it because the only reason I used non-headless chrome was because I was using my chrome extension as part of the selenium automation but I can actually just replicate my chrome extension's functionality using python and selenium anyways so I just followed this tutorial: https://cloudbytes.dev/snippets/run-selenium-in-aws-lambda-for-ui-testing and edited the code a little bit and now it's working. Thanks so much again!

[vinitshetty]

I was thinking about doing in on GCR but I decided against it because the only reason I used non-headless chrome was because I was using my chrome extension as part of the selenium automation but I can actually just replicate my chrome extension's functionality using python and selenium anyways so I just followed this tutorial: https://cloudbytes.dev/snippets/run-selenium-in-aws-lambda-for-ui-testing and edited the code a little bit and now it's working. Thanks so much again!

@JeffreyKozik - "edited the code a little " can you please suggest the changes you did? I am unable to get this working.

[rehanhaider]

@vinitshetty - He didn't get this working, instead, he included the functionality of the chrome extension directly in his Python code.

[marek-polewski]

Hello,

I had a similar issue (but with an empty error message on AWS) and I managed to get it working by dropping python to 3.8 and locking selenium to 3.141.0.

I got this idea by checking what is max version AWS is putting into their canaries: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Library_python_selenium.html

BR
MP

[rehanhaider]

@marek-polewski - Thanks Marek. Let me try and test with your suggestion.

[grantrosse]

For me it seems the issue is the fact that you need this argument to run selenium in Lambda: chrome_options.add_argument("--single-process"). However, including that argument breaks the extension loading process.

[Jonathynlee]

Does anyone have an update on this?

Using selenium with chrome extensions on lambda is perfect for my use-case. I am considering moving to Fargate but its a little less inline with what I am trying to do.

Quick Note,
chrome without extensions is working fine. I can run it headless on lambda with no issues (after much debugging)

Things I have tried:
-Using a pyvirtualdisplay with no visibility
-Running using seleniumbase
-All of the chrome flags one can think of (--no-zygote, --no-sandbox, -single-process,--disable-dev-shm-usage,--remote-debugging-port=9222,--user-data-dir={mkdtemp()},--data-path={mkdtemp()},--disk-cache-dir={mkdtemp()}, --no-xshm, --no-first-run) with many others attempted.
-Increasing vCPU/Memory
-x86_64 vs ARM64

The most frustrating part is that when I run the docker container locally, all is well. The extension works headless and everything completes as needed.

As soon as the driver loads in, I get a message that says

Message: session not created: Chrome failed to start: exited normally.   
(session not created: DevToolsActivePort file doesn't exist)   
(The process started from chrome location /opt/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)