Zombie Process problem. #1825
Comments
|
@bahattincinic thanks for filing this. So if I understand correctly, your workaround with disconnect-and-kill works? This should hint us into what's going on. |
|
Yes. We disconnected browser and killed process. |
|
@bahattincinic @aslushnikov I've briefly touched upon this here; killing the Chromium process aggressively on complete/timeout/errors helped us greatly as well. |
|
I found that doing |
|
@bahattincinic - thanks, I've tried your method of disconnecting + killing the process, and while it does kill the "main" process returned by What's worse, when I run I've tried sending a kill -15, hoping that will allow the main process to clean up its children, but -15 or -9 doesn't make any difference, so I'm still stuck with an ever-growing list of zombies and rising memory... Do you have any advice on how you managed to keep it clean of those as well (if you had a similar experience)? I'm also running on Lambda, same |
|
We are doing following methods to avoid zombie process.
if your project doesn't depend on AWS Lambda, you can use my example project. https://github.com/bahattincinic/puppeteer-docker-example |
|
@bahattincinic - thanks a lot for providing details - waitpid is an interesting approach and I'll definitely try with cleaning /tmp, hopefully that helps! If I don't manage to make it run reliably on Lambda, I'm going to have to try with docker - thanks for linking the example! |
|
@bahattincinic How does stopping the parent process with waitpid help with reaping zombie processes? |
|
Hi, I am also having problems with this, also serverless-chrome with AWS Lambda. In my case, it looks like it does not have anything to do with the browser cleanup process. It looks like it is being caused by something that happens during Puppeteer launch. Running See process 248 which is already defunct at this point. And then after the browser closes: Look at process with pid 248 which now has ppid 1. Is this even a Puppeteer bug? |
|
I have puppeteer running in a Docker container both locally and in a Heroku dyno. 2018-06-16T16:25:16.463094+00:00 app[web.1]: Error: spawn /app/node_modules/puppeteer/.local-chromium/linux-564778/chrome-linux/chrome EAGAIN
2018-06-16T16:25:16.463108+00:00 app[web.1]: at _errnoException (util.js:1022:11)
2018-06-16T16:25:16.463110+00:00 app[web.1]: at Process.ChildProcess._handle.onexit (internal/child_process.js:190:19)
2018-06-16T16:25:16.463111+00:00 app[web.1]: at onErrorNT (internal/child_process.js:372:16)
2018-06-16T16:25:16.463112+00:00 app[web.1]: at _combinedTickCallback (internal/process/next_tick.js:138:11)
2018-06-16T16:25:16.463114+00:00 app[web.1]: at process._tickDomainCallback (internal/process/next_tick.js:218:9)
2018-06-16T16:25:16.495669+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2018-06-16T16:25:16.495980+00:00 app[web.1]: npm ERR! errno 1
2018-06-16T16:25:16.497099+00:00 app[web.1]: npm ERR! irs.gov-form_filling@1.0.0 start: `node app.js`
2018-06-16T16:25:16.497182+00:00 app[web.1]: npm ERR! Exit status 1
2018-06-16T16:25:16.497371+00:00 app[web.1]: npm ERR!
2018-06-16T16:25:16.497499+00:00 app[web.1]: npm ERR! Failed at the irs.gov-form_filling@1.0.0 start script.
2018-06-16T16:25:16.497621+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2018-06-16T16:25:16.502935+00:00 app[web.1]:
2018-06-16T16:25:16.503086+00:00 app[web.1]: npm ERR! A complete log of this run can be found in:
2018-06-16T16:25:16.503166+00:00 app[web.1]: npm ERR! /app/.npm/_logs/2018-06-16T16_25_16_498Z-debug.logHow should I handle that? |
|
@webyneter We're using an alpine-based image. I just installed |
|
@Multiply you mean, you did ENTRYPOINT ["/sbin/tini", "--"]? |
|
@Multiply got that, will try Alpine image puppeteer guide with tini. |
|
@webyneter i previously explained my solution. But you can use my example implementation https://github.com/bahattincinic/puppeteer-docker-example According to puppetter documentation, it suggests yelp/dump-init. Also you can check https://github.com/ebidel/try-puppeteer this project. The repo creator is a core developer of puppetter. |
|
@jdiamond did you try #1825 (comment) it ? |
|
@bahattincinic thanks, I already implemented mine based on your suggestion to employ Alpine and tini. Although I followed the official guide, including the Tips section where it's said to |
|
Found the Increased size of /dev/shm on the Common Runtime official dev article. |
|
You don't have to use alpine to use tini, or dumbinit, it's just what we're using quite successfully so far. |
|
@Multiply I always strive to keep my image size down to minimum, but in my specific case I've just found out I can't use Alpine because "The latest version of puppeteer is not supported by the latest version of chrome alpine supports. Sorry, it's just the way it is right now." whislt we use |
|
I'm having a growing number of sleeping chrome process which is a real blocker for me since Heroku limits the number of processes considerably on lower tier plans. pptruser@21dcf0ed8fc3:/app$ ls -lAh /tmp/
total 36K
drwx------ 3 pptruser pptruser 4.0K Jun 17 09:09 puppeteer_dev_profile-46TBCn
drwx------ 3 pptruser pptruser 4.0K Jun 17 09:10 puppeteer_dev_profile-EW50RG
drwx------ 3 pptruser pptruser 4.0K Jun 17 09:10 puppeteer_dev_profile-KPAMX2
drwx------ 3 pptruser pptruser 4.0K Jun 17 09:10 puppeteer_dev_profile-LsBKzm
drwx------ 3 pptruser pptruser 4.0K Jun 17 09:09 puppeteer_dev_profile-aD98EK
drwx------ 3 pptruser pptruser 4.0K Jun 17 09:10 puppeteer_dev_profile-fqBvuu
drwx------ 3 pptruser pptruser 4.0K Jun 17 09:10 puppeteer_dev_profile-qo6kj3
drwx------ 3 pptruser pptruser 4.0K Jun 17 09:09 puppeteer_dev_profile-uVomwy
drwx------ 2 root root 4.0K Jun 17 08:44 tmp.NvsynR0wqxBesides, I haven't found any samples of |
|
rather than using tini or dumbinit to reap zombies, you can pass the |
|
More information I've discovered: The zombies are created using a "double fork", so they have a parent PID of 1. So waitpid won't work unless your node process has a PID of 1. That's certainly possible in a container environment, though. So @webyneter, you either use an init or waitpid. |
|
Hi guys. I am having Lambda function with puppeteer. It runs perfectly returns what I expect but never exits and I got timedOut. The only thing I do not use on the puppeteer side is
this is the log I am receiving:
|
|
Hi @bryanlarsen, hi @bahattincinic, thanks for your helpful comments. I am pretty new to the Docker&Puppeteer topic. So please, excuse me when I ask (again):
What do you mean @bryanlarsen? Can I just use the solution proposed on puppeteer/troubleshooting? My current image looks similar to: Or how can I (should I) use waitpid? @bahattincinic I can't find waitpid in your example-repo (see comment #1825 (comment)). Am I missing something? And you let
run from your node app after Thanks to both of you for your time and help! |
|
Am try process lookup and kill process. how about this? |
|
@insanehong i just test your code ,but it do not work well. it seem that |
|
@insanehong Can you try the following code? const waitpid2 = require('waitpid2');
const child_process = require('child_process');
const cleanEnv = function() {
console.log('cleanEnv:waitpid');
while (waitpid2.waitpid(-1, 0 | waitpid2.WNOHANG) == -1);
child_process.exec('rm -r /tmp/core.* || true');
}
const screenCapture = function() {
// do something.
}
exports.handler = function(event, context, callback) {
cleanEnv();
screenCapture();
} |
|
I've overcome these issues by adding the flags for chrome headless: I think the child processes are orphaned when the parent is killed and that leads to the zombies. With this, I only get one process and it works pretty well |
Attempts to prevent leaking zombie chromium processes by running chromium with the --single-process and --no-zygote flags. Flag descriptions: https://kapeli.com/cheat_sheets/Chromium_Command_Line_Switches.docset/Contents/Resources/Documents/index GitHub issue where this approach was reported to be successful: puppeteer/puppeteer#1825 Chromium related docs (for background): https://www.chromium.org/developers/design-documents/multi-process-architecture https://chromium.googlesource.com/chromium/src.git/+/master/docs/linux/zygote.md N.B. If the approach is agreed upon and this patch merged, the change will also need to be made separately for production in operations/deployment-charts. Bug: T257679 Change-Id: I8420c4b9e967e098f279fc26e0128e6169a204c6
|
@marius080 Even with your settings zombies processes are left after |
|
what worked fine to me was to destroy the PID and subprocesses associated: |
|
This is is the code i created to fix it, tested it, no more dangling chromes :) just call |
|
If you run it under docker you need to use docker |
|
Er
El lun., 11 de enero de 2021 6:28 p. m., g00dnatur3 <
notifications@github.com> escribió:
… This is is the code i created to fix it, tested it, no more dangling
chromes :)
const browser = await puppeteer.launch({args: ['--no-sandbox']});
const process = browser.process()
const killBrowser = (retries=5) => {
if (process && process.pid && process.kill && !process.killed) {
setTimeout(() => {
console.log(`BROWSER Process Id: ${process.pid}, KILLING IT! retries:`, retries);
if (!process.kill('SIGKILL')) {
retries--
killBrowser(retries)
}
}, 200);
}
}
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1825 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ARXQAM3WCJH4IIPK7JOOJ4LSZOJTXANCNFSM4EME5SAA>
.
|
Same problem on manjaro using headless. I will test your workaround |
|
Hi everyone, just wanted to provide a quick warning about the
I can confirm that the single process model does cause a rendering bug (at least on Chromium |
|
I'm also using puppeteer in docker, and I had also tried the Eventually, I figured out that the |
|
This works for me on Debian 10, just kills the process group based on the browser pid: const browser = await puppeteer.launch({ args: ['--no-sandbox'] })
const page = await browser.newPage()
try {
await page.goto(url, { waitUntil: 'networkidle2', timeout: 10000 })
await page.screenshot({ path })
await page.close()
await browser.close()
} catch (e) {
console.error(e)
} finally {
const pid = -browser.process().pid
try {
process.kill(pid, 'SIGKILL')
} catch (e) {}
} |
|
Here is what happens while the puppeteer is working:
My question is why processes I'm using the Docker image It seems that using the arg |
|
Sorry for the noise, I just wanted to confirm in case it can help somebody: in my case simply adding |
|
Nothing of this helps when the browser crashes, which happens quite often after we upgraded to Node 18 and newer Puppeteer. The child process isn't properly detached from the parent I guess, and thus will stay a zombie until the parent is finished. |
|
I have thoroughly reviewed the documentation and exhausted all available solutions in an attempt to resolve the zombie process issue. Despite my efforts, the problem persisted. I attempted to terminate process IDs, but within the pods, the zombie processes remained resilient. Devoting several consecutive days to diligently updating every package eventually proved successful. The issue was ultimately resolved by making key adjustments: switching the operating system from Node Alpine to Node Slim Linux and transitioning from Chromium to Chrome as the browser. The specific changes implemented to rectify the problem are outlined below. If you are working with Puppeteer and encountering zombie process issues, consider employing the following Docker commands. These commands have proven effective in preventing the creation of zombie processes. FROM node:18-slim RUN apt-get update && apt-get install curl gnupg -y RUN apt-get update && ADD ./puppetron.tar /usr/share/ ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true CMD node main.js; Path of browser change to |
adding "--headless" worked for me, i had a lot of chrome_crashpad that were created per each request even after closing without errors the page and browser. |
and others
Hello,
Recently we talked about this problem in the issues #1823 and #1791.
Environment:
Use Case:
We are using puppeteer on AWS Lambda. We take a screenshot of given HTML template and upload it to S3 and use this image for future requests
It handles over 100 million requests each month. That's why every process should be atomic and immutable. (AWS Lambda has a disk and process limit.)
Example Code:
Problem
When we are using example code, we got disk error from AWS Lambda.
Example
/tmpfolder:When we investigated these files, we understood that it is a
core dump. We removed these files after the process completed.When we monitored process list, we saw
zombie processesZombie chrome processes have been growing increasingly. We can't kill them. AWS Lambda has a maximum process limit. (max 1024 process) That's why we reach the lambda limits.We couldn't use
dump-initon lambda. Because lambda already has aninitsystem.How did we fix it? (very hacky method)
We used
browser.disconnect()instead ofbrowser.close(). We manualy managed chrome processes such askill.Example Code:
Firstly we didn't use this method. We only killed the process after browser disconnect. We got the following error:
I think it looks like a puppeteer process management problem. When we used this method, we didn't receive any puppeteer related errors. How can we fix it?
Thanks.
The text was updated successfully, but these errors were encountered: