Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cypress v3.3.0 crashes with ENOSPC in AWS CodeBuild #4283

Closed
luneo7 opened this issue May 22, 2019 · 17 comments

Comments

10 participants
@luneo7
Copy link

commented May 22, 2019

Current behavior:

[AWS CodeBuild Plugin] > cross-env NODE_ENV=test cypress run
[AWS CodeBuild Plugin] 
[AWS CodeBuild Plugin] { Error: watch /codebuild/output/src525851951/src/project/cypress/plugins/index.js ENOSPC
[AWS CodeBuild Plugin]     at _errnoException (util.js:1024:11)
[AWS CodeBuild Plugin]     at FSWatcher.start (fs.js:1386:19)
[AWS CodeBuild Plugin]     at Object.fs.watch (fs.js:1412:11)
[AWS CodeBuild Plugin]     at createFsWatchInstance (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:37:15)
[AWS CodeBuild Plugin]     at setFsWatchListener (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:80:15)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._watchWithNodeFs (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:228:14)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._handleFile (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:255:21)
[AWS CodeBuild Plugin]     at FSWatcher.<anonymous> (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:473:21)
[AWS CodeBuild Plugin]     at /root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/graceful-fs/polyfills.js:285:20
[AWS CodeBuild Plugin]     at FSReqWrap.oncomplete (fs.js:153:5)
[AWS CodeBuild Plugin] 
[AWS CodeBuild Plugin]   code: 'ENOSPC',
[AWS CodeBuild Plugin]   errno: 'ENOSPC',
[AWS CodeBuild Plugin]   syscall: 'watch /codebuild/output/src525851951/src/project/cypress/plugins/index.js',
[AWS CodeBuild Plugin]   filename: '/codebuild/output/src525851951/src/project/cypress/plugins/index.js' }
[AWS CodeBuild Plugin] Error: watch /codebuild/output/src525851951/src/project/cypress/plugins/index.js ENOSPC
[AWS CodeBuild Plugin]     at _errnoException (util.js:1024:11)
[AWS CodeBuild Plugin]     at FSWatcher.start (fs.js:1386:19)
[AWS CodeBuild Plugin]     at Object.fs.watch (fs.js:1412:11)
[AWS CodeBuild Plugin]     at createFsWatchInstance (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:37:15)
[AWS CodeBuild Plugin]     at setFsWatchListener (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:80:15)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._watchWithNodeFs (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:228:14)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._handleFile (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:255:21)
[AWS CodeBuild Plugin]     at FSWatcher.<anonymous> (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:473:21)
[AWS CodeBuild Plugin]     at /root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/graceful-fs/polyfills.js:285:20
[AWS CodeBuild Plugin]     at FSReqWrap.oncomplete (fs.js:153:5)
[AWS CodeBuild Plugin] 
[AWS CodeBuild Plugin] { Error: watch /codebuild/output/src525851951/src/project/env/parse.js ENOSPC
[AWS CodeBuild Plugin]     at _errnoException (util.js:1024:11)
[AWS CodeBuild Plugin]     at FSWatcher.start (fs.js:1386:19)
[AWS CodeBuild Plugin]     at Object.fs.watch (fs.js:1412:11)
[AWS CodeBuild Plugin]     at createFsWatchInstance (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:37:15)
[AWS CodeBuild Plugin]     at setFsWatchListener (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:80:15)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._watchWithNodeFs (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:228:14)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._handleFile (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:255:21)
[AWS CodeBuild Plugin]     at FSWatcher.<anonymous> (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:473:21)
[AWS CodeBuild Plugin]     at /root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/graceful-fs/polyfills.js:285:20
[AWS CodeBuild Plugin]     at FSReqWrap.oncomplete (fs.js:153:5)
[AWS CodeBuild Plugin] 
[AWS CodeBuild Plugin]   code: 'ENOSPC',
[AWS CodeBuild Plugin]   errno: 'ENOSPC',
[AWS CodeBuild Plugin]   syscall: 'watch /codebuild/output/src525851951/src/project/env/parse.js',
[AWS CodeBuild Plugin]   filename: '/codebuild/output/src525851951/src/project/env/parse.js' }
[AWS CodeBuild Plugin] Error: watch /codebuild/output/src525851951/src/project/env/parse.js ENOSPC
[AWS CodeBuild Plugin]     at _errnoException (util.js:1024:11)
[AWS CodeBuild Plugin]     at FSWatcher.start (fs.js:1386:19)
[AWS CodeBuild Plugin]     at Object.fs.watch (fs.js:1412:11)
[AWS CodeBuild Plugin]     at createFsWatchInstance (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:37:15)
[AWS CodeBuild Plugin]     at setFsWatchListener (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:80:15)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._watchWithNodeFs (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:228:14)
[AWS CodeBuild Plugin]     at FSWatcher.NodeFsHandler._handleFile (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:255:21)
[AWS CodeBuild Plugin]     at FSWatcher.<anonymous> (/root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/chokidar/lib/nodefs-handler.js:473:21)
[AWS CodeBuild Plugin]     at /root/.cache/Cypress/3.3.0/Cypress/resources/app/packages/server/node_modules/graceful-fs/polyfills.js:285:20
[AWS CodeBuild Plugin]     at FSReqWrap.oncomplete (fs.js:153:5)
[AWS CodeBuild Plugin] 

Desired behavior:

Cypress should run normally.

With version 3.2.0 same configuration this does not happens

Steps to reproduce: (app code and test code)

Use AWS Codebuild with NodeJS image, install Cypress 3.3.0 in it and try to run tests, it will fail

Versions

AWS Codebuild, Node 10.15.1, Yarn 1.13.0, Cypress 3.3.0

@ScOut3R

This comment has been minimized.

Copy link

commented May 23, 2019

Same here, thought that disabling watchForFileChanges might help, but unfortunately it didn't.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented May 23, 2019

There was this change made in 3.3.0 related to file watching: #4038

Can you provide any information that is different from a standard kitchen-sink run? What's in your cypress.json? Any unique directory setup? Plugins setup?

@rybon

This comment has been minimized.

Copy link

commented May 23, 2019

Also happens on Jenkins with an Ubuntu build environment. Standard setup, nothing out of the ordinary. Reverting to 3.2.0 fixed it.

@bahmutov

This comment has been minimized.

Copy link
Collaborator

commented May 23, 2019

Hmm, ENOSPC means there is no disk space, CodeBuild might be limited in space, so the test runner cannot bundle specs?

@rybon

This comment has been minimized.

Copy link

commented May 23, 2019

We checked disk space and inodes, they were fine. I think ENOSPC is a red herring in this case.

@Bkucera

This comment has been minimized.

Copy link
Member

commented May 23, 2019

From my trials, 3.3.0 is using about 30% more inodes than 3.2.0 (I measured ~680 vs ~530) This could cause some people to get ENOSPC, but I wouldn't think its too big a deal

If you're getting this on your local (linux) machine, you could try increasing your max inodes

@luneo7

This comment has been minimized.

Copy link
Author

commented May 23, 2019

Cypress.json is

{
  "viewportWidth": 1300,
  "defaultCommandTimeout": 20000,
  "blacklistHosts": [
    "*.google-analytics.com",
    "*.googletagmanager.com",
    "*.hotjar.com"
  ],
  "baseUrl": "http://127.0.0.1:3000",
  "video": false,
  "numTestsKeptInMemory": 0
}

And plugins index.js is:

const webpack = require('@cypress/webpack-preprocessor');
const getEnv = require('../../env/parse');

module.exports = (on, config) => {
  const options = {
    webpackOptions: {
      resolve: {extensions: ['.ts', '.tsx', '.js']},
      module: {rules: [{test: /\.tsx?$/, loader: 'babel-loader'}]},
      watch: false,
    },
     watchOptions: {},
  };

  on('file:preprocessor', webpack(options));

  on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--proxy-bypass-list=<-loopback>');
      return args;
    }
  });

  const {CYPRESS_APP_ENV = 'local', CYPRESS_POST_DEPLOY} = process.env;
  const env = getEnv(CYPRESS_APP_ENV);

  if (CYPRESS_POST_DEPLOY) {
    config.baseUrl = env.PORTAL_URL;
    if (CYPRESS_APP_ENV === 'prod') {
      config.testFiles = '{language,login}/**';
    }
  }

  config.env = env;
  return config;
};
@luneo7

This comment has been minimized.

Copy link
Author

commented May 23, 2019

This is happening because the max number of files being watched are bigger than fs.inotify.max_user_watches, as we are running this inside a docker container without privileged mode, there is no way to change this.

I discovered something here though... we are using webpack-dev-server with start-server-and-test and webpack-dev-server also watches files by default, so something happened between 3.2.0 and 3.3.0/3.3.1 that the number of files being watched through FS events increased and in conjunction with webpack-dev-server made the number become higher than what the docker container can handle.

So I changed the webpack-dev-server to be run with webpack-dev-server --config webpack.dev.config.js --quiet --inline=false --hot=false --watch-poll 6000000 and Cypress 3.3.0/3.3.1 worked in AWS Codebuild.

It wasn't a matter of enough disk space. I think that using cypress run wouldn't need to be watching files for changes, since we will probably use this feature inside a CI, also as @ScOut3R noted watchForFileChanges didn't help in anything, and I believe that it's because for default it is disabled while using the run command.

@luneo7

This comment has been minimized.

Copy link
Author

commented May 23, 2019

Yeah... the change #4038 made chokidar use FS events, before it was using polling, so that increased the number of user_watches and made it boom.

It would be nice to have a way to make it use polling by passing a param so we can circumvent this fs.inotify.max_user_watches error in docker containers, or even a way to not need this file watch when running inside a CI, cause the files won't change.

@rybon

This comment has been minimized.

Copy link

commented May 27, 2019

Okay, will this be fixed? Passing a param to disable watching files in CI would be enough I think.

@VinceOPS

This comment has been minimized.

Copy link

commented Jun 3, 2019

I'm glad I found this issue, thanks @luneo7. Having the same problem with Gitlab CI/CD and cypress 3.3.1.

Thus, I'm gonna keep 3.2.0 for a little longer 😄 !

@Bkucera

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

I think the best solution here is to not register the file watchers when using cypress run what do you think @jennifer-shehane ?

@flotwig flotwig self-assigned this Jun 11, 2019

@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 13, 2019

Does anyone in this thread use any Cypress plugins or have requires in their Cypress plugins file? It may be part of the cause of this issue.


EDIT: I was able to repro the issue in the OP by doing sudo sysctl -w fs.inotify.max_user_watches=0 locally and then trying to run a Cypress project with a pluginsFile. The fix in #4458 makes it work again by not trying to watch the pluginsFile.

I think this should fix it from Cypress's end, all file watching is disabled with this PR.

Users might still run in to this issue if they have other programs like webpack watching files, or if they manually watch files in their plugins file, but there's not much we can do about that.

@cypress-bot

This comment has been minimized.

Copy link

commented Jun 14, 2019

The code for this is done in cypress-io/cypress#4458, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

@micksatana

This comment has been minimized.

Copy link

commented Jun 14, 2019

FYI, in my case, I found this issue on GitLab CI/CD using cypress/base:8. I fixed by executing sysctl -w fs.inotify.max_user_watches=524288 before running the test. If anyone like to workaround this issue before the fix release, you might wanna try sysctl -w fs.inotify.max_user_watches=524288

@cypress-bot

This comment has been minimized.

Copy link

commented Jun 27, 2019

Released in 3.3.2.

@stanvit

This comment has been minimized.

Copy link

commented Jul 11, 2019

Fo those who are struggling with this issue on AWS Codebuild, enabling privileged mode on the build allows to modify sysctl parameters, including fs.inotify.max_user_watches:

[Container] 2019/07/11 06:18:34 Running command sysctl fs.inotify.max_user_watches 
fs.inotify.max_user_watches = 8192 
 
[Container] 2019/07/11 06:18:34 Running command sysctl fs.inotify.max_user_watches=524288 
fs.inotify.max_user_watches = 524288 
 
[Container] 2019/07/11 06:18:34 Running command sysctl fs.inotify.max_user_watches 
fs.inotify.max_user_watches = 524288
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.