Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cypress v3.3.0 crashes with ENOSPC in AWS CodeBuild #4283

Closed
luneo7 opened this issue May 22, 2019 · 20 comments
Closed

Cypress v3.3.0 crashes with ENOSPC in AWS CodeBuild #4283

luneo7 opened this issue May 22, 2019 · 20 comments
Assignees

Comments

@luneo7
Copy link

@luneo7 luneo7 commented May 22, 2019

Current behavior:

[AWS CodeBuild Plugin] > cross-env NODE_ENV=test cypress run
[AWS CodeBuild Plugin] 
[AWS CodeBuild Plugin] { Error: watch /codebuild/output/src525851951/src/project/cypress/plugins/index.js ENOSPC
[AWS CodeBuild Plugin]     at _errnoException (util.js:1024:11)
[AWS CodeBuild Plugin]     at FSWatcher.start (fs.js:1386:19)
[AWS CodeBuild Plugin]     at Object.fs.watch (fs.js:1412:11)
[AWS CodeBuild Plugin]     at createFsWatchInstance (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:37:15)
[AWS CodeBuild Plugin]     at setFsWatchListener (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:80:15)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._watchWithNodeFs (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:228:14)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._handleFile (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:255:21)
[AWS CodeBuild Plugin]     at FSWatcher.<anonymous> (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:473:21)
[AWS CodeBuild Plugin]     at /root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/graceful-fs/polyfills.js:285:20
[AWS CodeBuild Plugin]     at FSReqWrap.oncomplete (fs.js:153:5)
[AWS CodeBuild Plugin] 
[AWS CodeBuild Plugin]   code: 'ENOSPC',
[AWS CodeBuild Plugin]   errno: 'ENOSPC',
[AWS CodeBuild Plugin]   syscall: 'watch /codebuild/output/src525851951/src/project/cypress/plugins/index.js',
[AWS CodeBuild Plugin]   filename: '/codebuild/output/src525851951/src/project/cypress/plugins/index.js' }
[AWS CodeBuild Plugin] Error: watch /codebuild/output/src525851951/src/project/cypress/plugins/index.js ENOSPC
[AWS CodeBuild Plugin]     at _errnoException (util.js:1024:11)
[AWS CodeBuild Plugin]     at FSWatcher.start (fs.js:1386:19)
[AWS CodeBuild Plugin]     at Object.fs.watch (fs.js:1412:11)
[AWS CodeBuild Plugin]     at createFsWatchInstance (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:37:15)
[AWS CodeBuild Plugin]     at setFsWatchListener (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:80:15)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._watchWithNodeFs (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:228:14)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._handleFile (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:255:21)
[AWS CodeBuild Plugin]     at FSWatcher.<anonymous> (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:473:21)
[AWS CodeBuild Plugin]     at /root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/graceful-fs/polyfills.js:285:20
[AWS CodeBuild Plugin]     at FSReqWrap.oncomplete (fs.js:153:5)
[AWS CodeBuild Plugin] 
[AWS CodeBuild Plugin] { Error: watch /codebuild/output/src525851951/src/project/env/parse.js ENOSPC
[AWS CodeBuild Plugin]     at _errnoException (util.js:1024:11)
[AWS CodeBuild Plugin]     at FSWatcher.start (fs.js:1386:19)
[AWS CodeBuild Plugin]     at Object.fs.watch (fs.js:1412:11)
[AWS CodeBuild Plugin]     at createFsWatchInstance (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:37:15)
[AWS CodeBuild Plugin]     at setFsWatchListener (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:80:15)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._watchWithNodeFs (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:228:14)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._handleFile (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:255:21)
[AWS CodeBuild Plugin]     at FSWatcher.<anonymous> (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:473:21)
[AWS CodeBuild Plugin]     at /root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/graceful-fs/polyfills.js:285:20
[AWS CodeBuild Plugin]     at FSReqWrap.oncomplete (fs.js:153:5)
[AWS CodeBuild Plugin] 
[AWS CodeBuild Plugin]   code: 'ENOSPC',
[AWS CodeBuild Plugin]   errno: 'ENOSPC',
[AWS CodeBuild Plugin]   syscall: 'watch /codebuild/output/src525851951/src/project/env/parse.js',
[AWS CodeBuild Plugin]   filename: '/codebuild/output/src525851951/src/project/env/parse.js' }
[AWS CodeBuild Plugin] Error: watch /codebuild/output/src525851951/src/project/env/parse.js ENOSPC
[AWS CodeBuild Plugin]     at _errnoException (util.js:1024:11)
[AWS CodeBuild Plugin]     at FSWatcher.start (fs.js:1386:19)
[AWS CodeBuild Plugin]     at Object.fs.watch (fs.js:1412:11)
[AWS CodeBuild Plugin]     at createFsWatchInstance (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:37:15)
[AWS CodeBuild Plugin]     at setFsWatchListener (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:80:15)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._watchWithNodeFs (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:228:14)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._handleFile (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:255:21)
[AWS CodeBuild Plugin]     at FSWatcher.<anonymous> (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:473:21)
[AWS CodeBuild Plugin]     at /root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/graceful-fs/polyfills.js:285:20
[AWS CodeBuild Plugin]     at FSReqWrap.oncomplete (fs.js:153:5)
[AWS CodeBuild Plugin] 

Desired behavior:

Cypress should run normally.

With version 3.2.0 same configuration this does not happens

Steps to reproduce: (app code and test code)

Use AWS Codebuild with NodeJS image, install Cypress 3.3.0 in it and try to run tests, it will fail

Versions

AWS Codebuild, Node 10.15.1, Yarn 1.13.0, Cypress 3.3.0

@ScOut3R
Copy link

@ScOut3R ScOut3R commented May 23, 2019

Same here, thought that disabling watchForFileChanges might help, but unfortunately it didn't.

@jennifer-shehane
Copy link
Member

@jennifer-shehane jennifer-shehane commented May 23, 2019

There was this change made in 3.3.0 related to file watching: #4038

Can you provide any information that is different from a standard kitchen-sink run? What's in your cypress.json? Any unique directory setup? Plugins setup?

@rybon
Copy link

@rybon rybon commented May 23, 2019

Also happens on Jenkins with an Ubuntu build environment. Standard setup, nothing out of the ordinary. Reverting to 3.2.0 fixed it.

@bahmutov
Copy link
Collaborator

@bahmutov bahmutov commented May 23, 2019

Hmm, ENOSPC means there is no disk space, CodeBuild might be limited in space, so the test runner cannot bundle specs?

@rybon
Copy link

@rybon rybon commented May 23, 2019

We checked disk space and inodes, they were fine. I think ENOSPC is a red herring in this case.

@Bkucera
Copy link
Member

@Bkucera Bkucera commented May 23, 2019

From my trials, 3.3.0 is using about 30% more inodes than 3.2.0 (I measured ~680 vs ~530) This could cause some people to get ENOSPC, but I wouldn't think its too big a deal

If you're getting this on your local (linux) machine, you could try increasing your max inodes

@luneo7
Copy link
Author

@luneo7 luneo7 commented May 23, 2019

Cypress.json is

{
  "viewportWidth": 1300,
  "defaultCommandTimeout": 20000,
  "blacklistHosts": [
    "*.google-analytics.com",
    "*.googletagmanager.com",
    "*.hotjar.com"
  ],
  "baseUrl": "http://127.0.0.1:3000",
  "video": false,
  "numTestsKeptInMemory": 0
}

And plugins index.js is:

const webpack = require('@cypress/webpack-preprocessor');
const getEnv = require('../../env/parse');

module.exports = (on, config) => {
  const options = {
    webpackOptions: {
      resolve: {extensions: ['.ts', '.tsx', '.js']},
      module: {rules: [{test: /\.tsx?$/, loader: 'babel-loader'}]},
      watch: false,
    },
     watchOptions: {},
  };

  on('file:preprocessor', webpack(options));

  on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--proxy-bypass-list=<-loopback>');
      return args;
    }
  });

  const {CYPRESS_APP_ENV = 'local', CYPRESS_POST_DEPLOY} = process.env;
  const env = getEnv(CYPRESS_APP_ENV);

  if (CYPRESS_POST_DEPLOY) {
    config.baseUrl = env.PORTAL_URL;
    if (CYPRESS_APP_ENV === 'prod') {
      config.testFiles = '{language,login}/**';
    }
  }

  config.env = env;
  return config;
};

@luneo7
Copy link
Author

@luneo7 luneo7 commented May 23, 2019

This is happening because the max number of files being watched are bigger than fs.inotify.max_user_watches, as we are running this inside a docker container without privileged mode, there is no way to change this.

I discovered something here though... we are using webpack-dev-server with start-server-and-test and webpack-dev-server also watches files by default, so something happened between 3.2.0 and 3.3.0/3.3.1 that the number of files being watched through FS events increased and in conjunction with webpack-dev-server made the number become higher than what the docker container can handle.

So I changed the webpack-dev-server to be run with webpack-dev-server --config webpack.dev.config.js --quiet --inline=false --hot=false --watch-poll 6000000 and Cypress 3.3.0/3.3.1 worked in AWS Codebuild.

It wasn't a matter of enough disk space. I think that using cypress run wouldn't need to be watching files for changes, since we will probably use this feature inside a CI, also as @ScOut3R noted watchForFileChanges didn't help in anything, and I believe that it's because for default it is disabled while using the run command.

@luneo7
Copy link
Author

@luneo7 luneo7 commented May 23, 2019

Yeah... the change #4038 made chokidar use FS events, before it was using polling, so that increased the number of user_watches and made it boom.

It would be nice to have a way to make it use polling by passing a param so we can circumvent this fs.inotify.max_user_watches error in docker containers, or even a way to not need this file watch when running inside a CI, cause the files won't change.

@rybon
Copy link

@rybon rybon commented May 27, 2019

Okay, will this be fixed? Passing a param to disable watching files in CI would be enough I think.

@VinceOPS
Copy link

@VinceOPS VinceOPS commented Jun 3, 2019

I'm glad I found this issue, thanks @luneo7. Having the same problem with Gitlab CI/CD and cypress 3.3.1.

Thus, I'm gonna keep 3.2.0 for a little longer 😄 !

@Bkucera
Copy link
Member

@Bkucera Bkucera commented Jun 4, 2019

I think the best solution here is to not register the file watchers when using cypress run what do you think @jennifer-shehane ?

@flotwig flotwig self-assigned this Jun 11, 2019
@flotwig
Copy link
Member

@flotwig flotwig commented Jun 13, 2019

Does anyone in this thread use any Cypress plugins or have requires in their Cypress plugins file? It may be part of the cause of this issue.


EDIT: I was able to repro the issue in the OP by doing sudo sysctl -w fs.inotify.max_user_watches=0 locally and then trying to run a Cypress project with a pluginsFile. The fix in #4458 makes it work again by not trying to watch the pluginsFile.

I think this should fix it from Cypress's end, all file watching is disabled with this PR.

Users might still run in to this issue if they have other programs like webpack watching files, or if they manually watch files in their plugins file, but there's not much we can do about that.

@cypress-bot
Copy link

@cypress-bot cypress-bot bot commented Jun 14, 2019

The code for this is done in cypress-io/cypress#4458, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

@micksatana
Copy link

@micksatana micksatana commented Jun 14, 2019

FYI, in my case, I found this issue on GitLab CI/CD using cypress/base:8. I fixed by executing sysctl -w fs.inotify.max_user_watches=524288 before running the test. If anyone like to workaround this issue before the fix release, you might wanna try sysctl -w fs.inotify.max_user_watches=524288

@cypress-bot
Copy link

@cypress-bot cypress-bot bot commented Jun 27, 2019

Released in 3.3.2.

@stanvit
Copy link

@stanvit stanvit commented Jul 11, 2019

Fo those who are struggling with this issue on AWS Codebuild, enabling privileged mode on the build allows to modify sysctl parameters, including fs.inotify.max_user_watches:

[Container] 2019/07/11 06:18:34 Running command sysctl fs.inotify.max_user_watches 
fs.inotify.max_user_watches = 8192 
 
[Container] 2019/07/11 06:18:34 Running command sysctl fs.inotify.max_user_watches=524288 
fs.inotify.max_user_watches = 524288 
 
[Container] 2019/07/11 06:18:34 Running command sysctl fs.inotify.max_user_watches 
fs.inotify.max_user_watches = 524288

@AlexMost
Copy link

@AlexMost AlexMost commented Aug 16, 2019

I have met this issue on Ubuntu 18.04. This solved the issue:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

@amitguptagwl
Copy link

@amitguptagwl amitguptagwl commented Sep 9, 2019

I'm facing this issue. I'm using version 3.4.1.

@borbediana
Copy link

@borbediana borbediana commented Nov 22, 2019

I'm also facing the issue on version 3.4.1.
Indeed, the command sysctl -w fs.inotify.max_user_watches=524288 fixes it.
Although, is this the right solution? To change fs.inotify.max_user_watches ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.