Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker crashes when renderer process eats up too much memory #350

Closed
brian-mann opened this issue Dec 16, 2016 · 76 comments · Fixed by #15825
Closed

Docker crashes when renderer process eats up too much memory #350

brian-mann opened this issue Dec 16, 2016 · 76 comments · Fixed by #15825
Labels
browser: chrome external: docker Related to cypress provided docker images

Comments

@brian-mann
Copy link
Member

Related to #348 and #349.

When running headlessly on very long and memory intense applications we are seeing renderer crashes with Docker.

@brian-mann
Copy link
Member Author

brian-mann commented Dec 16, 2016

This is actually not indicative of a memory leak inside of Cypress (nor your application). This has to do with the way that browsers work under the hood.

Luckily, we have a found a simple one line fix for this problem - simply add the flag --ipc=host.

This option is documented here.

The Slack app and Atom has also been documented to crash here, here, and here for the exact same reasons.

If you are using docker run

docker run --ipc=host

If you are using docker compose

version: '2'
services:
  foobarbazservice:
    ipc: host ## this line right here

In the future we are working on a more permanent fix such as described in #349 - either to automatically recover from these crashes, or mostly prevent them up from by nuking the renderer process in between spec files.

@jheijkoop
Copy link

jheijkoop commented Dec 22, 2017

I seem to be getting this problem too, because Chrome is running out of shared memory (/dev/shm). By default docker starts images with a 64M /dev/shm (try running df -h in you instance). To change this you can supply docker with an extra argument: docker run --shm-size 512M my-image. Because we are working with mesos/marathon I had to do this via the "docker": { "parameters": [{"key": "shm-size", "value": "512M"}], ...} (https://mesosphere.github.io/marathon/docs/native-docker.html) in the json configuration when creating the app/instances.

@einomi
Copy link

einomi commented Mar 23, 2018

I had the same issue with Codeship CI. Just needed to change config in my codeship-services.yml file, according to documentation https://documentation.codeship.com/pro/continuous-integration/browser-testing/#chrome-crashing

@jennifer-shehane
Copy link
Member

I wonder if this is related to our other issue with a Codeship CI run failing.#328 Unfortunately, we don't have a pro account, and the codeship-services.yml is only available on pro.

@egucciar
Copy link
Contributor

any advice on how to solve for this in CircleCI? I do not have a services section in my config.yml, only a jobs.

@mitar
Copy link

mitar commented Mar 3, 2019

I think that since this issue has been made there is now a better fix for the problem by asking Chrome not to use /dev/shm. I opened #3633 for more details about this.

@wralitera
Copy link

hello @brian-mann . How do you set up the --ipc=host into the gitlab-ci.yml?

@ccorcos
Copy link

ccorcos commented Mar 27, 2019

Anyone know how to add --ipc=host to CircleCI? It looks like they call docker run somewhere out of our control...

@mitar
Copy link

mitar commented Mar 27, 2019

You cannot. And this is why this workaoround does not really work.

@ccorcos
Copy link

ccorcos commented Mar 27, 2019

@ccorcos
Copy link

ccorcos commented Mar 27, 2019

Actually, a build just finished and it crashed pretty quickly

@ccorcos
Copy link

ccorcos commented Mar 27, 2019

Looks like there's plenty of shared memory.

df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
shm              30G  8.0K   30G   1% /dev/shm

@adamlewkowicz
Copy link

I had the same issue yet I have changed my image from cypress/base to cypress/browsers:node11.13.0-chrome73 and it now works without crashes on gitlab-ci.

image: cypress/browsers:node11.13.0-chrome73

@rwralitera
Copy link

I will test this image: cypress/browsers:node11.13.0-chrome73 right now. Thanks @alk831

@rwralitera
Copy link

rwralitera commented Apr 15, 2019

I tested and it failed:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 2: 0x7fa824226887 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 3: 0x7fa823d95a57 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 4: 0x7fa823d959d5 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 5: v8::internal::Factory::NewStruct(v8::internal::InstanceType) [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 6: 
<--- Last few GCs --->

[425:0x2659cbff8000]  1853805 ms: Mark-sweep 2057.7 (2176.7) -> 2057.7 (2154.2) MB, 2096.2 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2096 ms) last resort 
[425:0x2659cbff8000]  1856036 ms: Mark-sweep 2057.7 (2154.2) -> 2057.7 (2154.2) MB, 2230.4 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x162eb08ad681 <JSObject>
    1: bindContainer(aka bindContainer) [/builds/wynd-qa/BO/node_modules/typescript/lib/typescript.js:~22229] [pc=0x2ef0ae593de3](this=0x2ae892502311 <undefined>,node=0x7235b368b9 <NodeObject map = 0x3516afdfd7d1>,containerFlags=45)
    2: bind(aka bind) [/builds/wynd-qa/BO/node_modules/typescript/lib/typescript.js:~23556] [pc=0x2ef0ae55cdac](this=0x2ae892502311 <undefined>,node=0x7235b368b9 <No...

v8::internal::Factory::NewTuple3(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 7: 0x7fa823b407a1 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 8: 0x7fa823b4347b [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 9: 0x7fa823b431bc [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
10: 0x7fa823b47b5f [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
11: 0x2ef0add043fd

Timed out waiting for the browser to connect. Retrying...

The following error was thrown by a plugin. We've stopped running your tests because a plugin crashed.

@adamlewkowicz
Copy link

@bentrole It crashed only once for me with this image and now it works properly. I would try to split large tests into smaller ones and run without recording until there is no fix for this.

@rwralitera
Copy link

@alk831 I already splited my tests because of that but this bug is there since 2016, If there is no fixes until now, I think that they will not correct that! ;-(

@ccorcos
Copy link

ccorcos commented Apr 15, 2019

@brian-mann can we please reopen this issue? If we cannot scale out our testing library, I don't think we can use this service :/

@jackbravo
Copy link

What about when running on github actions, like https://github.com/cypress-io/github-action ? How do you enable ipc=host?

@mattvb91
Copy link

mattvb91 commented Jan 2, 2021

Started experiencing this again on github actions upgrading from 5.3 to 6.2

@vincentpalita
Copy link

Version 6.2.1 seems to fix that for me.

@mattvb91
Copy link

mattvb91 commented Jan 5, 2021

Unfortunately 6.2.1 still crashing https://github.com/corejam/corejam/runs/1650892998?check_suite_focus=true#step:9:290 staying on 5.3 for now. Maybe someone else can chime in a version number when it started happening for them between 5.3 & 6.2 as I dont have the time to manually go through each version / build the associated docker images

@mattvb91
Copy link

mattvb91 commented Jan 28, 2021

https://github.com/corejam/corejam/runs/1784204472?check_suite_focus=true#step:9:313

Still no luck on 6.3.0

5.3.0 still stable for us have not had any chrome crashes there


Edit crashes on 6.4.0 too: https://github.com/corejam/corejam/runs/1820965122?check_suite_focus=true#step:9:287

@mattvb91
Copy link

mattvb91 commented Feb 12, 2021

Anyone looking to enable IPC=host on github actions it looks like it is possible like this:
https://github.com/corejam/corejam/pull/311/files#diff-014228303dff9a1af15f4bbd18401f906380129b10ae2a2c62f8b8be592ff88eR105

You can see it setting the option here: https://github.com/corejam/corejam/runs/1888917637?check_suite_focus=true#step:2:111
/usr/bin/docker create --name 01f73992fe8148a89514be690697b85d_corejamcypress640_aa44b9 --label 5588e4 --workdir /__w/corejam/corejam --network github_network_3bce8c6715944b47a995b42f42973efa --ipc=host ...

More context here: actions/runner-images#2697 (comment)

Just let it run through 3 times on our setup and its finally running it seems!!

Can anyone else confirm? If its working for others ill put in a MR with the above sample for docs

Edit: Have since put on about 10 runs from merge requests and its stable!

@dobriai
Copy link

dobriai commented Feb 23, 2021

FWIW, in our case adding --shm-size=512m seems to have quieted down the issue. For now. Using cypress@6.4.0.

bep added a commit to bep/netlify-cypress-test that referenced this issue Mar 30, 2021
@TimothyRHuertas
Copy link

If you added --ipc=host and you still get the error, try restarting docker 🤦

@cypress-bot cypress-bot bot added stage: work in progress There is an open PR for this issue [WIP] and removed stage: needs investigating Someone from Cypress needs to look at this labels Apr 6, 2021
@cypress-bot cypress-bot bot added stage: pending release There is a closed PR for this issue and removed stage: work in progress There is an open PR for this issue [WIP] labels Apr 6, 2021
@cypress-bot
Copy link

cypress-bot bot commented Apr 6, 2021

The code for this is done in cypress-io/cypress#15825, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

@flotwig
Copy link
Member

flotwig commented Apr 7, 2021

In 7.0.1, Cypress passes disable-dev-shm-usage to Chrome and Electron on startup, which should remove the need to use --ipc=host or increase the /dev/shm size. Once it's released this should be solved.

@cypress-bot
Copy link

cypress-bot bot commented Apr 7, 2021

Released in 7.0.1.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v7.0.1, please open a new issue.

@cypress-bot cypress-bot bot removed the stage: pending release There is a closed PR for this issue label Apr 7, 2021
@cypress-bot cypress-bot bot locked as resolved and limited conversation to collaborators Apr 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
browser: chrome external: docker Related to cypress provided docker images
Projects
None yet
Development

Successfully merging a pull request may close this issue.