Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker crashes when renderer process eats up too much memory #350

Open
brian-mann opened this issue Dec 16, 2016 · 64 comments
Open

Docker crashes when renderer process eats up too much memory #350

brian-mann opened this issue Dec 16, 2016 · 64 comments

Comments

@brian-mann
Copy link
Member

@brian-mann brian-mann commented Dec 16, 2016

Related to #348 and #349.

When running headlessly on very long and memory intense applications we are seeing renderer crashes with Docker.

@brian-mann
Copy link
Member Author

@brian-mann brian-mann commented Dec 16, 2016

This is actually not indicative of a memory leak inside of Cypress (nor your application). This has to do with the way that browsers work under the hood.

Luckily, we have a found a simple one line fix for this problem - simply add the flag --ipc=host.

This option is documented here.

The Slack app and Atom has also been documented to crash here, here, and here for the exact same reasons.

If you are using docker run

docker run --ipc=host

If you are using docker compose

version: '2'
services:
  foobarbazservice:
    ipc: host ## this line right here

In the future we are working on a more permanent fix such as described in #349 - either to automatically recover from these crashes, or mostly prevent them up from by nuking the renderer process in between spec files.

@jheijkoop
Copy link

@jheijkoop jheijkoop commented Dec 22, 2017

I seem to be getting this problem too, because Chrome is running out of shared memory (/dev/shm). By default docker starts images with a 64M /dev/shm (try running df -h in you instance). To change this you can supply docker with an extra argument: docker run --shm-size 512M my-image. Because we are working with mesos/marathon I had to do this via the "docker": { "parameters": [{"key": "shm-size", "value": "512M"}], ...} (https://mesosphere.github.io/marathon/docs/native-docker.html) in the json configuration when creating the app/instances.

@einomi
Copy link

@einomi einomi commented Mar 23, 2018

I had the same issue with Codeship CI. Just needed to change config in my codeship-services.yml file, according to documentation https://documentation.codeship.com/pro/continuous-integration/browser-testing/#chrome-crashing

@jennifer-shehane
Copy link
Member

@jennifer-shehane jennifer-shehane commented Mar 23, 2018

I wonder if this is related to our other issue with a Codeship CI run failing.#328 Unfortunately, we don't have a pro account, and the codeship-services.yml is only available on pro.

@egucciar
Copy link
Contributor

@egucciar egucciar commented Nov 12, 2018

any advice on how to solve for this in CircleCI? I do not have a services section in my config.yml, only a jobs.

@mitar
Copy link

@mitar mitar commented Mar 3, 2019

I think that since this issue has been made there is now a better fix for the problem by asking Chrome not to use /dev/shm. I opened #3633 for more details about this.

@wralitera
Copy link

@wralitera wralitera commented Mar 20, 2019

hello @brian-mann . How do you set up the --ipc=host into the gitlab-ci.yml?

@ccorcos
Copy link

@ccorcos ccorcos commented Mar 27, 2019

Anyone know how to add --ipc=host to CircleCI? It looks like they call docker run somewhere out of our control...

@mitar
Copy link

@mitar mitar commented Mar 27, 2019

You cannot. And this is why this workaoround does not really work.

@ccorcos
Copy link

@ccorcos ccorcos commented Mar 27, 2019

Actually, a build just finished and it crashed pretty quickly

@ccorcos
Copy link

@ccorcos ccorcos commented Mar 27, 2019

Looks like there's plenty of shared memory.

df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
shm              30G  8.0K   30G   1% /dev/shm
@alk831
Copy link

@alk831 alk831 commented Apr 15, 2019

I had the same issue yet I have changed my image from cypress/base to cypress/browsers:node11.13.0-chrome73 and it now works without crashes on gitlab-ci.

image: cypress/browsers:node11.13.0-chrome73
@rwralitera
Copy link

@rwralitera rwralitera commented Apr 15, 2019

I will test this image: cypress/browsers:node11.13.0-chrome73 right now. Thanks @alk831

@rwralitera
Copy link

@rwralitera rwralitera commented Apr 15, 2019

I tested and it failed:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 2: 0x7fa824226887 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 3: 0x7fa823d95a57 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 4: 0x7fa823d959d5 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 5: v8::internal::Factory::NewStruct(v8::internal::InstanceType) [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 6: 
<--- Last few GCs --->

[425:0x2659cbff8000]  1853805 ms: Mark-sweep 2057.7 (2176.7) -> 2057.7 (2154.2) MB, 2096.2 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2096 ms) last resort 
[425:0x2659cbff8000]  1856036 ms: Mark-sweep 2057.7 (2154.2) -> 2057.7 (2154.2) MB, 2230.4 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x162eb08ad681 <JSObject>
    1: bindContainer(aka bindContainer) [/builds/wynd-qa/BO/node_modules/typescript/lib/typescript.js:~22229] [pc=0x2ef0ae593de3](this=0x2ae892502311 <undefined>,node=0x7235b368b9 <NodeObject map = 0x3516afdfd7d1>,containerFlags=45)
    2: bind(aka bind) [/builds/wynd-qa/BO/node_modules/typescript/lib/typescript.js:~23556] [pc=0x2ef0ae55cdac](this=0x2ae892502311 <undefined>,node=0x7235b368b9 <No...

v8::internal::Factory::NewTuple3(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 7: 0x7fa823b407a1 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 8: 0x7fa823b4347b [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 9: 0x7fa823b431bc [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
10: 0x7fa823b47b5f [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
11: 0x2ef0add043fd

Timed out waiting for the browser to connect. Retrying...

The following error was thrown by a plugin. We've stopped running your tests because a plugin crashed.
@alk831
Copy link

@alk831 alk831 commented Apr 15, 2019

@bentrole It crashed only once for me with this image and now it works properly. I would try to split large tests into smaller ones and run without recording until there is no fix for this.

@rwralitera
Copy link

@rwralitera rwralitera commented Apr 15, 2019

@alk831 I already splited my tests because of that but this bug is there since 2016, If there is no fixes until now, I think that they will not correct that! ;-(

@ccorcos
Copy link

@ccorcos ccorcos commented Apr 15, 2019

@brian-mann can we please reopen this issue? If we cannot scale out our testing library, I don't think we can use this service :/

@lsdkjflasjflkdasj
Copy link

@lsdkjflasjflkdasj lsdkjflasjflkdasj commented Jan 10, 2020

Same thing happening here.

@tommueller
Copy link

@tommueller tommueller commented Jan 13, 2020

We are also constantly running into this issue and by now this is creating a lot of frustration here.
We are running the Cypress tests in a Drone-CI docker setup. All the tests run perfectly fine on my local machine. It's especially confusing too me, as our application is really not that memory hungry (about 80mb max memory usage says Chrome Task Manager) and tests are very short too short (mostly between 5-15 loc).

We already tried these recommendations:

  • increased the size of dev/shm
  • upgraded to latest versions of drone, drone-agent and cypress
  • decreased numTestsKeptInMemory setting
    without any success.

I have not yet tried these options, as they are not possible with our current setup or seem to be too unstable ...

Any updates on this would be a great help for us!

@davidzambrana
Copy link

@davidzambrana davidzambrana commented Jan 14, 2020

We fixed this problem after implementing a few recommendations we saw here:

  • Updated to Cypress 3.8.0
  • Moved to this image cypress/browsers:node10.16.0-chrome77
  • Started using -headless -b chrome in the run command
  • Updated our plugins/index file with
on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--disable-dev-shm-usage')
      return args
    }

    return args
  })

The crashes stopped in all the test collections we have (~15)

@tommueller
Copy link

@tommueller tommueller commented Jan 15, 2020

I can confirm that the changes @davidzambrana proposed also worked for us so far.
We now run into the issue described here #1297 but this is probably more an issue of how we wrote the tests.
I will try to fix the tests and keep an eye on this. Thanks @davidzambrana for now!

@mulholio
Copy link

@mulholio mulholio commented Feb 4, 2020

To anyone wondering what the --disable-dev-shm-usage flag does, this page is a useful explainer https://developers.google.com/web/tools/puppeteer/troubleshooting#tips

@shlima
Copy link

@shlima shlima commented Feb 19, 2020

#350 (comment) can confirm the solution, but I've changed cypress/browsers:node10.16.0-chrome77 image to cypress/included:4.0.2 as it currently includes cypress and chrome.

My specs started to run smoothly on Gitlab CI with cypress in docker-in-docker

@adinizs
Copy link

@adinizs adinizs commented Feb 27, 2020

We resolved this problem using the image cypress/browsers:node10.16.0-chrome on dockerfile and running chrome headless.

@radDunin
Copy link

@radDunin radDunin commented Mar 20, 2020

Hi,
i can also confirmed @davidzambrana solution, what is important is that the image should be this one: cypress/browsers:node10.16.0-chrome77 it works even in headed chrome. For now thats doesn't works for me on chrome 80 and chrome 69

@otalandim
Copy link

@otalandim otalandim commented Apr 18, 2020

I had the same issue yet I have changed my image from cypress/base to cypress/browsers:node11.13.0-chrome73 and it now works without crashes on gitlab-ci.

image: cypress/browsers:node11.13.0-chrome73

For me it doesnot works

@otalandim
Copy link

@otalandim otalandim commented Apr 18, 2020

We fixed this problem after implementing a few recommendations we saw here:

  • Updated to Cypress 3.8.0
  • Moved to this image cypress/browsers:node10.16.0-chrome77
  • Started using -headless -b chrome in the run command
  • Updated our plugins/index file with
on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--disable-dev-shm-usage')
      return args
    }

    return args
  })

The crashes stopped in all the test collections we have (~15)

For me it doesnot works

Another problem happened...can you help me please

  CypressError: cy.visit() failed trying to load:
 We attempted to make an http request to this URL but the request failed without a response.
 We received this error at the network level:
   > Error: ESOCKETTIMEDOUT
 Common situations why this would fail:
   - you don't have internet access
   - you forgot to run / boot your web server
   - your web server isn't accessible
   - you have weird network configuration settings on your computer
 The stack trace for this error is:
 Error: ESOCKETTIMEDOUT
     at ClientRequest.<anonymous> (/root/.cache/Cypress/4.2.0/Cypress/resources/app/packages/server/node_modules/request/request.js:816:19)
     at Object.onceWrapper (events.js:299:28)
     at ClientRequest.emit (events.js:210:5)
     at TLSSocket.emitRequestTimeout (_http_client.js:690:9)
     at Object.onceWrapper (events.js:299:28)
     at TLSSocket.emit (events.js:210:5)
     at TLSSocket.Socket._onTimeout (net.js:468:8)
     at listOnTimeout (internal/timers.js:531:17)
     at processTimers (internal/timers.js:475:7)
 Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite:
@przemuh
Copy link
Contributor

@przemuh przemuh commented May 25, 2020

Switching to Chrome with --disable-dev-shm-usage as @davidzambrana suggested - works for me!

Thanks a lot! 🎉

@otalandim Please remember that since Cypress 4.0.0 there is a change in passing arguments to the browser (https://docs.cypress.io/guides/references/migration-guide.html#Plugin-Event-before-browser-launch)

I use GitlabCI. Here are my changes:

GitlabCI config

-  image: cypress/base:12.16.2
+  image: cypress/browsers:node12.16.2-chrome81-ff75

cypress/plugins/index.js

on("before:browser:launch", (browser = {}, launchOptions) => {
  if (browser.name === "chrome") {
    launchOptions.args.push("--disable-dev-shm-usage");
    return launchOptions;
  }
}

run command:

cypress run --browser chrome --headless
@ttomaszewski
Copy link

@ttomaszewski ttomaszewski commented Jun 17, 2020

Any idea when this will get resolved for Electron?

@digitalkaoz
Copy link

@digitalkaoz digitalkaoz commented Jun 18, 2020

for anyone getting here and want to know how to fix it on gitlab-ci kubernetes executor:

$ kubectl edit cm gitlab-runner-gitlab-runner

config part to add right before # Start the runner in entrypoint
see https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2578#note_186055981

    cat >> /home/gitlab-runner/.gitlab-runner/config.toml << EOF
          [[runners.kubernetes.volumes.empty_dir]]
              name = "shm"
              mount_path = "/dev/shm"
              medium = "Memory"
    EOF
@sabarishnarain
Copy link

@sabarishnarain sabarishnarain commented Jul 31, 2020

This is still a problem in Azure CI with electron (cypress 4.11) when using the base docker images. The workaround is to use chrome by following @przemuh instructions.

@keul
Copy link

@keul keul commented Aug 5, 2020

For posterity: I've this same issue in Gitlab CI.

Unluckily every gitlab-ci.yml workaround above require the change of the base docker image, while I'm not (and I can't) using any of them: I am manually installing cypress on a Centos (is an internal gitlab installation using a private runner server).

In my case this gitlab comment to issue solved the problem.

LeSuisse added a commit to Enalean/tuleap that referenced this issue Aug 25, 2020
…ning Cypress

By default, Docker container starts with a limit of 64MB of shared memory
which can be a bit short when running a browser.

This a known issue when running Electron/Chromium inside a Docker container,
you can read more about it in the Cypress's bug tracker: cypress-io/cypress#350

Change-Id: Ibb87a5532b7616cb6f0801f81d4961aa05d53ba3
@CoryDanielson
Copy link
Contributor

@CoryDanielson CoryDanielson commented Sep 8, 2020

I saw a reduction of 30-50% of memory usage by enabling window.gc() and calling it globally in an afterEach callback.

You can enabled window.gc with browser launch options, or electron command line switches.

// For Chrome, add this flag in plugins
launchOptions.args.push('--js-flags=--expose-gc');
# For Electon, run cypress with this environment variable set
ELECTRON_EXTRA_LAUNCH_ARGS=--js-flags=--expose_gc

plugins/index.js

module.exports = (on, config) => {
	on('before:browser:launch', (browser, launchOptions) => { 
		if (browser.name === 'chrome') {
			launchOptions.args.push('--js-flags=--expose-gc');
		}
		return launchOptions;
	});
};

support/index.js

afterEach(() => {
	cy.window().then(win => {
		if (typeof win.gc === 'function') {
			// calling this more often seems to trigger major GC event more reliably
			win.gc(); win.gc(); win.gc(); win.gc(); win.gc();
		}
	});
});
cypress open peak memory usage - ~50% reduction in peak memory
# --js-flags=--expose-gc is set in plugins/index.js
cypress open

Max memory in MB (recorded by watching Chrome memory tab)

no win.gc win.gc once calling win.gc 5 times
535mb 299 267
585mb 406 369
598mb 372 278
570mb 445 298
--- averages ---
572mb 380 (34% less) 303 (47% less)
cypress run peak memory usage - ~30% reduction in peak memory for chrome headless
# --js-flags=--expose-gc is set in plugins/index.js
cypress run --headless -b chrome

(recorded by watching Activity Monitor while tests were running)

Chrome headless, no win.gc()
Chrome Rendered Memory
Chrome headless calling win.gc() 5 times
Chrome Rendered Memory
890 580 (~30% less)
850 560 (~30% less)
cypress run peak memory usage = ~50% reduction in peak memory for electron
ELECTRON_EXTRA_LAUNCH_ARGS=--js-flags=--expose_gc cypress run

(recorded by watching Activity Monitor while tests were running)

Electron, no win.gc()
Cypress Helper Renderer Memory
Electron, calling win.gc() 5 times
850mb 410 (~50% less)
750mb 450 (~50% less)

Calling window.gc() multiple times was more effective (based on my imperfect tests), I suspect because it possibly caused chrome to trigger a major GC event rather than a minor one.


We switched our CI docker run script to use --ipc=host, --disable-dev-shm-usage, chrome headless, and use window.gc() and it's running really well now with no out-of-memory crashes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.