New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incredibly slow (near unusable) on Docker container #1395

Closed
mbrio opened this Issue Aug 9, 2016 · 16 comments

Comments

Projects
None yet
@mbrio
Contributor

mbrio commented Aug 9, 2016

I've begun using Docker containers for many of my projects and have noticed how incredibly slow Jest runs in a Docker container when executing on a shared folder on OS X. Within Docker you can mount a local path and map it to a path within the container (e.g. ~/Source/projecta:/usr/src/app). When using this configuration my tests which consists of a single file with a single unit test takes over 30 seconds; but when run locally outside of Docker it takes 1 second. I'm using the command jest specs/unit; which I got from another Jest issue suggesting that supplying the test folder could speed up tests, and it did, locally, from 3 seconds to 1 second; but made no change to how it runs in Docker. I would have left it at that and said that it's a Docker for Mac issue (especially because there is a speed issue with mounted volumes that is known) but I ran the same exact single test using mocha and it executed in milliseconds. The unit test is a simple expect(true).toBe(true) (or for mocha+chai expect(true).to.equal(true) and for some reason Jest, under Docker, executed in ~30 seconds vs mocha's less than 1 second.

Are there any configuration settings or optimizations that I could use to help speed up Jest? I would much rather use Jest over Mocha, but will obviously use the test platform that executes in a reasonable amount of time. This is my Jest configuration:

"jest": {
    "automock": false,
    "verbose": true,
    "testFileExtensions": [
      "js"
    ],
    "moduleFileExtensions": [
      "js",
      "json"
    ],
    "testDirectoryName": "spec/unit",
    "testPathIgnorePatterns": [
      "/node_modules/",
      "<rootDir>/lib/"
    ],
    "modulePathIgnorePatterns": [
      "/node_modules/",
      "<rootDir>/lib/"
    ]
  }
@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Aug 10, 2016

Contributor

Hmm, that is very weird. Is the file system access slow, that's really the only thing that should be different here between Jest and mocha? Jest crawls the file system from the root (package.json) and looks at all the files for static analysis. How many cores does the system report that it has? Can you try invoking Jest with -i?

Contributor

cpojer commented Aug 10, 2016

Hmm, that is very weird. Is the file system access slow, that's really the only thing that should be different here between Jest and mocha? Jest crawls the file system from the root (package.json) and looks at all the files for static analysis. How many cores does the system report that it has? Can you try invoking Jest with -i?

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Aug 10, 2016

Contributor

Another thing is that if you are using babel-jest, babel actually loads a ton of modules so if the fs is slow, this might contribute to that.

Contributor

cpojer commented Aug 10, 2016

Another thing is that if you are using babel-jest, babel actually loads a ton of modules so if the fs is slow, this might contribute to that.

@aaronabramov

This comment has been minimized.

Show comment
Hide comment
@aaronabramov

aaronabramov Aug 10, 2016

Member

i remember having similar kind of issues inside docker container. The problem was the slowness of fs access. I think it was related to mounting a directory through a virtual container

Member

aaronabramov commented Aug 10, 2016

i remember having similar kind of issues inside docker container. The problem was the slowness of fs access. I think it was related to mounting a directory through a virtual container

@aaronabramov

This comment has been minimized.

Show comment
Hide comment
@aaronabramov

aaronabramov Aug 10, 2016

Member

found the issue docker/compose#631
not sure if it's still relevant

Member

aaronabramov commented Aug 10, 2016

found the issue docker/compose#631
not sure if it's still relevant

@aaronjensen

This comment has been minimized.

Show comment
Hide comment
Contributor

aaronjensen commented Aug 11, 2016

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Aug 12, 2016

Contributor

Yeah I'm gonna close this out because unfortunately it isn't actionable from our side :(

Contributor

cpojer commented Aug 12, 2016

Yeah I'm gonna close this out because unfortunately it isn't actionable from our side :(

@cpojer cpojer closed this Aug 12, 2016

@mbrio

This comment has been minimized.

Show comment
Hide comment
@mbrio

mbrio Aug 12, 2016

Contributor

Hi @cpojer, sorry I didn't mean to drop off like that, I did get a chance to run jest -i, I also installed native watchman and ran jest -i --watchman, neither of which sped up the processing of tests. As I mentioned in my original post I understood that there was a speed issue with Docker and mounted drives. I would like to point out that I have implemented mocha in conjunction with chokidar for watching files and still do not see the kind of slowdown I'm seeing with jest. Chokidar+mocha detects file changes in no more than 2 seconds and executed tests in under a second; jest takes over 30 seconds, sometimes up to a minute to detect a file change and executes the single test in a little over a second. Also, I'd like to note that I am not using babel-jest, I am using only the features of es6 that are available to node 6. I understand if this is something you are not looking to persue, but there does seem to be some sort of bottleneck in how jest watches files when compared to chokidar. Finally, as to your question about cores, my VM has 2 cores and 4gb of RAM.

Contributor

mbrio commented Aug 12, 2016

Hi @cpojer, sorry I didn't mean to drop off like that, I did get a chance to run jest -i, I also installed native watchman and ran jest -i --watchman, neither of which sped up the processing of tests. As I mentioned in my original post I understood that there was a speed issue with Docker and mounted drives. I would like to point out that I have implemented mocha in conjunction with chokidar for watching files and still do not see the kind of slowdown I'm seeing with jest. Chokidar+mocha detects file changes in no more than 2 seconds and executed tests in under a second; jest takes over 30 seconds, sometimes up to a minute to detect a file change and executes the single test in a little over a second. Also, I'd like to note that I am not using babel-jest, I am using only the features of es6 that are available to node 6. I understand if this is something you are not looking to persue, but there does seem to be some sort of bottleneck in how jest watches files when compared to chokidar. Finally, as to your question about cores, my VM has 2 cores and 4gb of RAM.

@amkoehler

This comment has been minimized.

Show comment
Hide comment
@amkoehler

amkoehler Jan 13, 2017

@mbrio I decided to switch over to mocha, and the tests run immediately within my docker container without an issue. No changes were made to my Dockerfile or docker-compose configuration. This seems like a Jest issue and not a Docker issue. I had no issues with running my Jests tests locally, only within the container.

amkoehler commented Jan 13, 2017

@mbrio I decided to switch over to mocha, and the tests run immediately within my docker container without an issue. No changes were made to my Dockerfile or docker-compose configuration. This seems like a Jest issue and not a Docker issue. I had no issues with running my Jests tests locally, only within the container.

@mbrio

This comment has been minimized.

Show comment
Hide comment
@mbrio

mbrio Jan 17, 2017

Contributor

@amkoehler yeah, I think there's a significant difference in the libraries that Jest and Mocha use for watching filesystem changes. I assume that Mocha is less thorough, (and/or) less active, and/or less compatible with all filesystems. The speed issue is definitely an issue with Docker for Mac as I have friends who have slightly different configurations for running Docker on a Mac that do not see as big of an issue. With that said, since Docker for Mac is the official way to run Docker instances, it would be great for the developers of Jest to consider looking into how to be a bit more compatible with it.

Contributor

mbrio commented Jan 17, 2017

@amkoehler yeah, I think there's a significant difference in the libraries that Jest and Mocha use for watching filesystem changes. I assume that Mocha is less thorough, (and/or) less active, and/or less compatible with all filesystems. The speed issue is definitely an issue with Docker for Mac as I have friends who have slightly different configurations for running Docker on a Mac that do not see as big of an issue. With that said, since Docker for Mac is the official way to run Docker instances, it would be great for the developers of Jest to consider looking into how to be a bit more compatible with it.

@wmartins

This comment has been minimized.

Show comment
Hide comment
@wmartins

wmartins Aug 19, 2017

I came to this issue lots of times when trying to figure it out why jest was slow.
Somehow, in my scenario, I was able to improve the performance. I'll just leave it here in case anyone is trying to figure it out too.

Scenario:

Building my application inside a docker container.

In my case, the container was based on Linux Alpine.

My Solution:

I changed the container base image to use Ubuntu instead of Alpine.
My tests started to run faster (from 3 ~ 4 min to 40s ~ 60s).

Hope this helps someone! :)

wmartins commented Aug 19, 2017

I came to this issue lots of times when trying to figure it out why jest was slow.
Somehow, in my scenario, I was able to improve the performance. I'll just leave it here in case anyone is trying to figure it out too.

Scenario:

Building my application inside a docker container.

In my case, the container was based on Linux Alpine.

My Solution:

I changed the container base image to use Ubuntu instead of Alpine.
My tests started to run faster (from 3 ~ 4 min to 40s ~ 60s).

Hope this helps someone! :)

@alexgorbatchev

This comment has been minimized.

Show comment
Hide comment
@alexgorbatchev

alexgorbatchev Dec 21, 2017

This page pretty much says that osxfs is slow and that's being worked on...

As of Dec'17, running jest can be considered unusable in my opinion. I'm seeing 28s time for 3 basic test files inside a container. Exact same thing on the host takes 0.4s. This can work for CI purposes, but not for developer env.

There are a few clunky workarounds for this, all of which in my opinion mostly defeat the point of running development env inside containers. docker-sync and d4m-nfs are considered to be currently best solutions for this. Personally I'm not interested in trying either because if I can't use docker pull as the only command to "setup" my dev environment, then I may as well not bother until osxfs is fixed.

alexgorbatchev commented Dec 21, 2017

This page pretty much says that osxfs is slow and that's being worked on...

As of Dec'17, running jest can be considered unusable in my opinion. I'm seeing 28s time for 3 basic test files inside a container. Exact same thing on the host takes 0.4s. This can work for CI purposes, but not for developer env.

There are a few clunky workarounds for this, all of which in my opinion mostly defeat the point of running development env inside containers. docker-sync and d4m-nfs are considered to be currently best solutions for this. Personally I'm not interested in trying either because if I can't use docker pull as the only command to "setup" my dev environment, then I may as well not bother until osxfs is fixed.

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Dec 21, 2017

Collaborator

@rogeliog is this something you could potentially find someone in Docker team to fix?

Collaborator

thymikee commented Dec 21, 2017

@rogeliog is this something you could potentially find someone in Docker team to fix?

@jpbochi

This comment has been minimized.

Show comment
Hide comment
@jpbochi

jpbochi Jul 13, 2018

enabling the delegated flag (eg. docker -v ${PWD}:${PWD}:delegated) can make jester considerably faster, but still not close to native. Docs about this flag at at https://docs.docker.com/docker-for-mac/osxfs-caching/

jpbochi commented Jul 13, 2018

enabling the delegated flag (eg. docker -v ${PWD}:${PWD}:delegated) can make jester considerably faster, but still not close to native. Docs about this flag at at https://docs.docker.com/docker-for-mac/osxfs-caching/

@amayun

This comment has been minimized.

Show comment
Hide comment
@amayun

amayun Sep 7, 2018

I've investigated a bit, and it seems the source of slowness is this._crawl
It crawls whole file tree from the root of project, including node_modules - and due slow fs access from docker to osxfs it takes almost 2 minutes to walk whole file tree.
And for me solution was simply add "roots": ["./src"] to jest.config.js

amayun commented Sep 7, 2018

I've investigated a bit, and it seems the source of slowness is this._crawl
It crawls whole file tree from the root of project, including node_modules - and due slow fs access from docker to osxfs it takes almost 2 minutes to walk whole file tree.
And for me solution was simply add "roots": ["./src"] to jest.config.js

@SimenB

This comment has been minimized.

Show comment
Hide comment
@SimenB

SimenB Sep 13, 2018

Collaborator

That's great!
#6732 landed recently and fixed the roots support, so that's probably a viable solution for folks wanting to run things in docker now

Collaborator

SimenB commented Sep 13, 2018

That's great!
#6732 landed recently and fixed the roots support, so that's probably a viable solution for folks wanting to run things in docker now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment