Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incredibly slow (near unusable) on Docker container #1395

Closed
mbrio opened this issue Aug 9, 2016 · 26 comments

Comments

@mbrio
Copy link
Contributor

@mbrio mbrio commented Aug 9, 2016

I've begun using Docker containers for many of my projects and have noticed how incredibly slow Jest runs in a Docker container when executing on a shared folder on OS X. Within Docker you can mount a local path and map it to a path within the container (e.g. ~/Source/projecta:/usr/src/app). When using this configuration my tests which consists of a single file with a single unit test takes over 30 seconds; but when run locally outside of Docker it takes 1 second. I'm using the command jest specs/unit; which I got from another Jest issue suggesting that supplying the test folder could speed up tests, and it did, locally, from 3 seconds to 1 second; but made no change to how it runs in Docker. I would have left it at that and said that it's a Docker for Mac issue (especially because there is a speed issue with mounted volumes that is known) but I ran the same exact single test using mocha and it executed in milliseconds. The unit test is a simple expect(true).toBe(true) (or for mocha+chai expect(true).to.equal(true) and for some reason Jest, under Docker, executed in ~30 seconds vs mocha's less than 1 second.

Are there any configuration settings or optimizations that I could use to help speed up Jest? I would much rather use Jest over Mocha, but will obviously use the test platform that executes in a reasonable amount of time. This is my Jest configuration:

"jest": {
    "automock": false,
    "verbose": true,
    "testFileExtensions": [
      "js"
    ],
    "moduleFileExtensions": [
      "js",
      "json"
    ],
    "testDirectoryName": "spec/unit",
    "testPathIgnorePatterns": [
      "/node_modules/",
      "<rootDir>/lib/"
    ],
    "modulePathIgnorePatterns": [
      "/node_modules/",
      "<rootDir>/lib/"
    ]
  }
@cpojer

This comment has been minimized.

Copy link
Contributor

@cpojer cpojer commented Aug 10, 2016

Hmm, that is very weird. Is the file system access slow, that's really the only thing that should be different here between Jest and mocha? Jest crawls the file system from the root (package.json) and looks at all the files for static analysis. How many cores does the system report that it has? Can you try invoking Jest with -i?

@cpojer

This comment has been minimized.

Copy link
Contributor

@cpojer cpojer commented Aug 10, 2016

Another thing is that if you are using babel-jest, babel actually loads a ton of modules so if the fs is slow, this might contribute to that.

@aaronabramov

This comment has been minimized.

Copy link
Member

@aaronabramov aaronabramov commented Aug 10, 2016

i remember having similar kind of issues inside docker container. The problem was the slowness of fs access. I think it was related to mounting a directory through a virtual container

@aaronabramov

This comment has been minimized.

Copy link
Member

@aaronabramov aaronabramov commented Aug 10, 2016

found the issue docker/compose#631
not sure if it's still relevant

@aaronjensen

This comment has been minimized.

Copy link
Contributor

@aaronjensen aaronjensen commented Aug 11, 2016

@cpojer

This comment has been minimized.

Copy link
Contributor

@cpojer cpojer commented Aug 12, 2016

Yeah I'm gonna close this out because unfortunately it isn't actionable from our side :(

@cpojer cpojer closed this Aug 12, 2016
@mbrio

This comment has been minimized.

Copy link
Contributor Author

@mbrio mbrio commented Aug 12, 2016

Hi @cpojer, sorry I didn't mean to drop off like that, I did get a chance to run jest -i, I also installed native watchman and ran jest -i --watchman, neither of which sped up the processing of tests. As I mentioned in my original post I understood that there was a speed issue with Docker and mounted drives. I would like to point out that I have implemented mocha in conjunction with chokidar for watching files and still do not see the kind of slowdown I'm seeing with jest. Chokidar+mocha detects file changes in no more than 2 seconds and executed tests in under a second; jest takes over 30 seconds, sometimes up to a minute to detect a file change and executes the single test in a little over a second. Also, I'd like to note that I am not using babel-jest, I am using only the features of es6 that are available to node 6. I understand if this is something you are not looking to persue, but there does seem to be some sort of bottleneck in how jest watches files when compared to chokidar. Finally, as to your question about cores, my VM has 2 cores and 4gb of RAM.

@amkoehler

This comment has been minimized.

Copy link

@amkoehler amkoehler commented Jan 13, 2017

@mbrio I decided to switch over to mocha, and the tests run immediately within my docker container without an issue. No changes were made to my Dockerfile or docker-compose configuration. This seems like a Jest issue and not a Docker issue. I had no issues with running my Jests tests locally, only within the container.

@mbrio

This comment has been minimized.

Copy link
Contributor Author

@mbrio mbrio commented Jan 17, 2017

@amkoehler yeah, I think there's a significant difference in the libraries that Jest and Mocha use for watching filesystem changes. I assume that Mocha is less thorough, (and/or) less active, and/or less compatible with all filesystems. The speed issue is definitely an issue with Docker for Mac as I have friends who have slightly different configurations for running Docker on a Mac that do not see as big of an issue. With that said, since Docker for Mac is the official way to run Docker instances, it would be great for the developers of Jest to consider looking into how to be a bit more compatible with it.

@wmartins

This comment has been minimized.

Copy link

@wmartins wmartins commented Aug 19, 2017

I came to this issue lots of times when trying to figure it out why jest was slow.
Somehow, in my scenario, I was able to improve the performance. I'll just leave it here in case anyone is trying to figure it out too.

Scenario:

Building my application inside a docker container.

In my case, the container was based on Linux Alpine.

My Solution:

I changed the container base image to use Ubuntu instead of Alpine.
My tests started to run faster (from 3 ~ 4 min to 40s ~ 60s).

Hope this helps someone! :)

@alexgorbatchev

This comment has been minimized.

Copy link

@alexgorbatchev alexgorbatchev commented Dec 21, 2017

This page pretty much says that osxfs is slow and that's being worked on...

As of Dec'17, running jest can be considered unusable in my opinion. I'm seeing 28s time for 3 basic test files inside a container. Exact same thing on the host takes 0.4s. This can work for CI purposes, but not for developer env.

There are a few clunky workarounds for this, all of which in my opinion mostly defeat the point of running development env inside containers. docker-sync and d4m-nfs are considered to be currently best solutions for this. Personally I'm not interested in trying either because if I can't use docker pull as the only command to "setup" my dev environment, then I may as well not bother until osxfs is fixed.

@thymikee

This comment has been minimized.

Copy link
Collaborator

@thymikee thymikee commented Dec 21, 2017

@rogeliog is this something you could potentially find someone in Docker team to fix?

@jpbochi

This comment has been minimized.

Copy link

@jpbochi jpbochi commented Jul 13, 2018

enabling the delegated flag (eg. docker -v ${PWD}:${PWD}:delegated) can make jester considerably faster, but still not close to native. Docs about this flag at at https://docs.docker.com/docker-for-mac/osxfs-caching/

@amayun

This comment has been minimized.

Copy link

@amayun amayun commented Sep 7, 2018

I've investigated a bit, and it seems the source of slowness is this._crawl
It crawls whole file tree from the root of project, including node_modules - and due slow fs access from docker to osxfs it takes almost 2 minutes to walk whole file tree.
And for me solution was simply add "roots": ["./src"] to jest.config.js

@SimenB

This comment has been minimized.

Copy link
Collaborator

@SimenB SimenB commented Sep 13, 2018

That's great!
#6732 landed recently and fixed the roots support, so that's probably a viable solution for folks wanting to run things in docker now

@neilpalima

This comment has been minimized.

Copy link

@neilpalima neilpalima commented Oct 24, 2018

Is the fix already on the latest jest version? I am using the latest and running 3 tests on docker took 2+ mins.

@cdomigan

This comment has been minimized.

Copy link

@cdomigan cdomigan commented Nov 1, 2018

Using the new PnP feature of Yarn enabled Jest to run fast in Docker for me. It removes the node_modules folder meaning Jest doesn't have to crawl it.

@virasana

This comment has been minimized.

Copy link

@virasana virasana commented Dec 11, 2018

"roots": ["./src"] not supported for create-react-app (boo!)

@elliotlarson

This comment has been minimized.

Copy link

@elliotlarson elliotlarson commented Jan 28, 2019

I feel like this is still an issue. Running a single test file was taking me 17-20 seconds in a Docker container. After some Googling, I landed on this issue. I already have a jest roots config setup, like @amayun recommended. But, using the delegated approach recommended by @jpbochi allowed me to cut the run time down to 8 seconds. However, that's still a long time to wait for a single test to run. I hear that this is basically a Docker/Mac/Jest issue, and I get that there are Docker pieces that can't be addressed by the Jest team. Still though, this seems like a pretty common set of technologies to develop with. And, given that Mocha seems to run quickly in a Docker container leads me to believe that there could be something tweakable on the Jest side of things.

@jpbochi

This comment has been minimized.

Copy link

@jpbochi jpbochi commented Jan 28, 2019

I agree with @elliotlarson. Even if most of the blame is on Docker for Mac, I believe jest could optimize its scan for tests in ways that would help regardless of Docker.

@SimenB

This comment has been minimized.

Copy link
Collaborator

@SimenB SimenB commented Jan 28, 2019

Ideas and PRs are most welcome. It's, as far as I know, not an issue for anyone on the core team - so if anything is gonna happen here it'll have to come from the community

@jpbochi

This comment has been minimized.

Copy link

@jpbochi jpbochi commented Jan 28, 2019

@SimenB Fair enough. I only wished there was some incentive for the core team to try to reach more people. However cool jest might be, there are alternatives out there. And being super slow to run is a deal breaker.

@jasonwilliams

This comment has been minimized.

Copy link

@jasonwilliams jasonwilliams commented Jun 6, 2019

Linking to docker/for-mac#1358

@toryas

This comment has been minimized.

Copy link

@toryas toryas commented Jun 13, 2019

I had a similar problem, in a kubernete's pod the unit-test takes until 200 segs to done, then I found thins issue on Jest's GitHub.

So, I add this --maxWorkers=1 to jest script and then test-units just takes 14 segs to done

@SamanShafigh

This comment has been minimized.

Copy link

@SamanShafigh SamanShafigh commented Jul 9, 2019

I use create-react-app and I can use the --roots to fix the issue. This is they way I run my test using react-scripts.

./node_modules/.bin/react-scripts test --roots "${PWD}/src"

Note: I am using "react-scripts": "^2.1.5" and react-scripts test is documented that --roots accepts an array of strings but when I define it as an array like the following code it is not working.

./node_modules/.bin/react-scripts test --roots ["${PWD}/src"]

However if I pass it as string it will work. I don't know if I am doing a mistake or something

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.