New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File access in mounted volumes extremely slow #77

Open
schmunk42 opened this Issue Aug 2, 2016 · 212 comments

Comments

Projects
None yet
@schmunk42

schmunk42 commented Aug 2, 2016

continued from https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/178

Expected behavior

File-system performance in the container not significantly slower than on the host or docker-machine.

Actual behavior

Running a composer (PHP) update in a container takes much longer than on the host. In the container we usually hit timeouts when updating huge git repos like twbs/bootstrap.

Information & Steps to reproduce the behavior

See link above

@Vanuan

This comment has been minimized.

Show comment
Hide comment
@Vanuan

Vanuan Sep 4, 2016

Same for npm install, bundle install, browserify, etc.

Vanuan commented Sep 4, 2016

Same for npm install, bundle install, browserify, etc.

@MartialGeek

This comment has been minimized.

Show comment
Hide comment
@MartialGeek

MartialGeek Sep 5, 2016

Hi

Ok, that's why a Symfony application is very slow in a Docker container!
I load a page in ~ 150 ms when I'm running my local Symfony server (on my host) VS ~ 2000 ms when the config files have been serialized in the cache directory within a Docker container.
It seems the IO in the vendors and the cache / logs directories are very slow...

Hi

Ok, that's why a Symfony application is very slow in a Docker container!
I load a page in ~ 150 ms when I'm running my local Symfony server (on my host) VS ~ 2000 ms when the config files have been serialized in the cache directory within a Docker container.
It seems the IO in the vendors and the cache / logs directories are very slow...

@lfv89

This comment has been minimized.

Show comment
Hide comment
@lfv89

lfv89 Sep 6, 2016

Same here with rails and its friends (rails, rake, rspec...)

Please let me know what information I can provide to help you guys out.

lfv89 commented Sep 6, 2016

Same here with rails and its friends (rails, rake, rspec...)

Please let me know what information I can provide to help you guys out.

@mduller

This comment has been minimized.

Show comment
Hide comment
@mduller

mduller Sep 7, 2016

Same here as well. We're looking into using IntelliJ IDEA and git on the Mac and compiling & running our product in a container, from the shared filesystem. Development cycles are considerably longer and our devs not happy with the proposed switch. Currently evaluating workarounds to still enable the switch to Docker and in particular Docker for Mac.

mduller commented Sep 7, 2016

Same here as well. We're looking into using IntelliJ IDEA and git on the Mac and compiling & running our product in a container, from the shared filesystem. Development cycles are considerably longer and our devs not happy with the proposed switch. Currently evaluating workarounds to still enable the switch to Docker and in particular Docker for Mac.

@wadjeroudi

This comment has been minimized.

Show comment
Hide comment
@wadjeroudi

wadjeroudi Sep 7, 2016

For those who look for a workaround and didn't read the whole thread, here is the solution :
https://github.com/EugenMayer/docker-sync

For those who look for a workaround and didn't read the whole thread, here is the solution :
https://github.com/EugenMayer/docker-sync

@ZZromanZZ

This comment has been minimized.

Show comment
Hide comment
@ZZromanZZ

ZZromanZZ Sep 7, 2016

Same with PHP (Symfony, Nette, Composer deps...). I can confirm https://github.com/EugenMayer/docker-sync is usable workaround.

Same with PHP (Symfony, Nette, Composer deps...). I can confirm https://github.com/EugenMayer/docker-sync is usable workaround.

@iwaffles

This comment has been minimized.

Show comment
Hide comment
@iwaffles

iwaffles Sep 8, 2016

We're experiencing this as well. We have our API dockerized and sometimes endpoints time out on our local machines (only in docker for mac). We're using Rails 5.

iwaffles commented Sep 8, 2016

We're experiencing this as well. We have our API dockerized and sometimes endpoints time out on our local machines (only in docker for mac). We're using Rails 5.

@MartialGeek

This comment has been minimized.

Show comment
Hide comment
@MartialGeek

MartialGeek Sep 9, 2016

In my team, some developers use a Mac, others use Linux... Docker sync is not the solution for me because the project configuration will not be the same for all environments. I hope that the OSXFS issues will be fixed quickly.

In my team, some developers use a Mac, others use Linux... Docker sync is not the solution for me because the project configuration will not be the same for all environments. I hope that the OSXFS issues will be fixed quickly.

@wadjeroudi

This comment has been minimized.

Show comment
Hide comment
@wadjeroudi

wadjeroudi Sep 9, 2016

@MartialGeek Off topic : https://github.com/EugenMayer/docker-sync/wiki/Keep-you-docker-compose.yml-portable
But you should create a makefile to handle the environment properly.

@MartialGeek Off topic : https://github.com/EugenMayer/docker-sync/wiki/Keep-you-docker-compose.yml-portable
But you should create a makefile to handle the environment properly.

@MartialGeek

This comment has been minimized.

Show comment
Hide comment
@MartialGeek

MartialGeek Sep 9, 2016

@wadjeroudi Thank you, I will read that ASAP ;)

@wadjeroudi Thank you, I will read that ASAP ;)

@petergombos

This comment has been minimized.

Show comment
Hide comment
@petergombos

petergombos Sep 13, 2016

Same issue here, takes +30 sec to run my babel build using a mounted volume vs. 5 sec using container's fs.

Same issue here, takes +30 sec to run my babel build using a mounted volume vs. 5 sec using container's fs.

@jordscream

This comment has been minimized.

Show comment
Hide comment
@jordscream

jordscream Sep 14, 2016

👍 Same issue with Git in docker, very slow ! Unusable compare to Docker Machine (with NFS)
Docker For Mac released too fast ! Need to be reviews !!!

👍 Same issue with Git in docker, very slow ! Unusable compare to Docker Machine (with NFS)
Docker For Mac released too fast ! Need to be reviews !!!

@jzahedieh

This comment has been minimized.

Show comment
Hide comment
@jzahedieh

jzahedieh Sep 14, 2016

Whole team is having issues with Magento, 3 minute page loads vs 8 seconds using docker-sync

Whole team is having issues with Magento, 3 minute page loads vs 8 seconds using docker-sync

@BenMcH

This comment has been minimized.

Show comment
Hide comment
@BenMcH

BenMcH Sep 15, 2016

I'm also experiencing this issue. Hitting the cache takes no less than 1 second for each file.

BenMcH commented Sep 15, 2016

I'm also experiencing this issue. Hitting the cache takes no less than 1 second for each file.

@apahne

This comment has been minimized.

Show comment
Hide comment
@apahne

apahne Sep 17, 2016

Will this issue eventually be addressed?

apahne commented Sep 17, 2016

Will this issue eventually be addressed?

@Vanuan

This comment has been minimized.

Show comment
Hide comment
@Vanuan

Vanuan Sep 17, 2016

Here's the latest reply from Docker Team:
https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/158

It's been quite a while since then. We all hope that they have something new to share.

Vanuan commented Sep 17, 2016

Here's the latest reply from Docker Team:
https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/158

It's been quite a while since then. We all hope that they have something new to share.

@joemewes

This comment has been minimized.

Show comment
Hide comment
@joemewes

joemewes Sep 17, 2016

Just checked on latest Docker for Mac update : Version 1.12.1 (build: 12133)

seems to be still an issue.

Just checked on latest Docker for Mac update : Version 1.12.1 (build: 12133)

seems to be still an issue.

@so0k

This comment has been minimized.

Show comment
Hide comment
@so0k

so0k Sep 19, 2016

Latest posts on the alternatives are interesting - https://forums.docker.com/t/alternatives-to-osxfs-performant-shares-under-osx/19565/14 - which points to https://github.com/cweagans/docker-bg-sync (minimal changes to compose file for developer allow a significant speed improvement for mounted source files)

so0k commented Sep 19, 2016

Latest posts on the alternatives are interesting - https://forums.docker.com/t/alternatives-to-osxfs-performant-shares-under-osx/19565/14 - which points to https://github.com/cweagans/docker-bg-sync (minimal changes to compose file for developer allow a significant speed improvement for mounted source files)

@joemewes

This comment has been minimized.

Show comment
Hide comment
@joemewes

joemewes Sep 19, 2016

Yeah, I've gone the UNISON approach (also mentioned a couple times in the original thread) : see here https://github.com/4AllDigital/DockerDrupal-lite/blob/master/docker-compose.yml#L7

using this container : https://github.com/4AllDigital/drupaldev-unison

if anyone needs an example to copy and want to use UNISON too.

Yeah, I've gone the UNISON approach (also mentioned a couple times in the original thread) : see here https://github.com/4AllDigital/DockerDrupal-lite/blob/master/docker-compose.yml#L7

using this container : https://github.com/4AllDigital/drupaldev-unison

if anyone needs an example to copy and want to use UNISON too.

@piotrekkaminski

This comment has been minimized.

Show comment
Hide comment
@piotrekkaminski

piotrekkaminski Sep 21, 2016

Same issue with Magento. Major performance issues. We are moving to Unison now as well.

Same issue with Magento. Major performance issues. We are moving to Unison now as well.

@BenMcH

This comment has been minimized.

Show comment
Hide comment
@BenMcH

BenMcH Sep 22, 2016

I just got around to testing the bg-sync workaround with our in-house rails app and it worked perfectly! Thanks for the suggestion @so0k

BenMcH commented Sep 22, 2016

I just got around to testing the bg-sync workaround with our in-house rails app and it worked perfectly! Thanks for the suggestion @so0k

@charlie-wasp

This comment has been minimized.

Show comment
Hide comment
@charlie-wasp

charlie-wasp Sep 25, 2016

@BenMcH Hello, I am trying to use bg-sync for rails app too, but I have a problems :) Can you direct me in the right direction? I opened the issue in the bg-sync repo - cweagans/docker-bg-sync#2

@BenMcH Hello, I am trying to use bg-sync for rails app too, but I have a problems :) Can you direct me in the right direction? I opened the issue in the bg-sync repo - cweagans/docker-bg-sync#2

@motin

This comment has been minimized.

Show comment
Hide comment
@motin

motin Sep 28, 2016

Anyone have some current performance stats with the latest stable and beta versions for some reproducible usage scenarios?
It would be great to understand if the performance is 10% of native or closer to 80% or whatever, to understand if a workaround is worth implementing, and if there is any noticeable difference between the latest stable and beta versions.

motin commented Sep 28, 2016

Anyone have some current performance stats with the latest stable and beta versions for some reproducible usage scenarios?
It would be great to understand if the performance is 10% of native or closer to 80% or whatever, to understand if a workaround is worth implementing, and if there is any noticeable difference between the latest stable and beta versions.

@Vanuan

This comment has been minimized.

Show comment
Hide comment
@Vanuan

Vanuan Sep 28, 2016

Though it also depends on the network latency, I find npm install a good test:

docker run \
    -v `pwd`:/src \
    -v `pwd`/tmp/node_modules/:/src/node_modules \
    -v `pwd`/tmp/npm_cache/:/root/ \
    -v `pwd`/tmp/tmp/:/tmp/ \
    node:4.4.3-slim \
    sh -c 'cd /src && npm config set loglevel info && npm install react babel webpack && echo "npm install completed..."'

The 2nd run, after caching, doesn't involve much networking so it would reveal the differences in disk usage.

Vanuan commented Sep 28, 2016

Though it also depends on the network latency, I find npm install a good test:

docker run \
    -v `pwd`:/src \
    -v `pwd`/tmp/node_modules/:/src/node_modules \
    -v `pwd`/tmp/npm_cache/:/root/ \
    -v `pwd`/tmp/tmp/:/tmp/ \
    node:4.4.3-slim \
    sh -c 'cd /src && npm config set loglevel info && npm install react babel webpack && echo "npm install completed..."'

The 2nd run, after caching, doesn't involve much networking so it would reveal the differences in disk usage.

@wadjeroudi

This comment has been minimized.

Show comment
Hide comment
@wadjeroudi

wadjeroudi Sep 30, 2016

FYI, we stopped using unison or rsync. We decided to switch the mac on virtualbox with https://github.com/adlogix/docker-machine-nfs
No more performance problem.

FYI, we stopped using unison or rsync. We decided to switch the mac on virtualbox with https://github.com/adlogix/docker-machine-nfs
No more performance problem.

@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom Apr 10, 2017

@daveisfera From what I can see this feature is like any other features added in latest version only, so this requires Docker 17.04 and (as @Vanuan ask about) Docker Compose 1.12.

So if you need to support older versions of Docker you'll need to either wait before you use this, or provide two set of compose configs.

andrerom commented Apr 10, 2017

@daveisfera From what I can see this feature is like any other features added in latest version only, so this requires Docker 17.04 and (as @Vanuan ask about) Docker Compose 1.12.

So if you need to support older versions of Docker you'll need to either wait before you use this, or provide two set of compose configs.

@bilby91

This comment has been minimized.

Show comment
Hide comment
@bilby91

bilby91 Apr 11, 2017

@yallop Is there any ETA for :delegated. I'm trying to solve a speed problem on writes. Rails asset compilation is SUPER slow between request when one asset changes.

bilby91 commented Apr 11, 2017

@yallop Is there any ETA for :delegated. I'm trying to solve a speed problem on writes. Rails asset compilation is SUPER slow between request when one asset changes.

@yallop

This comment has been minimized.

Show comment
Hide comment
@yallop

yallop Apr 11, 2017

Contributor

@bilby91: support for delegated is under development, but it's a little early to give a useful ETA.

We're always interested in reproducible test cases to help guide development. It's useful to know that rails asset compilation is currently slow, but it'd be even more useful if you could provide a more detailed description of your rails setup. We have a growing collection of such real world scenarios, and they're a big help in tracking down performance issues and measuring improvements.

The "What you can do" section of the osxfs documentation explains more about the information that's most useful, in case you have time to put together a more detailed report.

Contributor

yallop commented Apr 11, 2017

@bilby91: support for delegated is under development, but it's a little early to give a useful ETA.

We're always interested in reproducible test cases to help guide development. It's useful to know that rails asset compilation is currently slow, but it'd be even more useful if you could provide a more detailed description of your rails setup. We have a growing collection of such real world scenarios, and they're a big help in tracking down performance issues and measuring improvements.

The "What you can do" section of the osxfs documentation explains more about the information that's most useful, in case you have time to put together a more detailed report.

@daveisfera

This comment has been minimized.

Show comment
Hide comment
@daveisfera

daveisfera Apr 11, 2017

@andrerom I understand that this is a new feature, but what about people that want to support development on multiple OSes. RHEL 6 and 7 are going to be around for a long time still, so you're saying people will need to keep multiple configs around for years just so :cached can be specified on their Macs?

@andrerom I understand that this is a new feature, but what about people that want to support development on multiple OSes. RHEL 6 and 7 are going to be around for a long time still, so you're saying people will need to keep multiple configs around for years just so :cached can be specified on their Macs?

@daynejones

This comment has been minimized.

Show comment
Hide comment
@daynejones

daynejones Apr 12, 2017

Anecdotally, the hyperkit process consistently uses > 100% CPU and the osxfs process uses around 50%.

Using the new cached option for all of my mounts got this down to hyperkit using ~10% and osxfs using 0%.

I'm thrilled.

Anecdotally, the hyperkit process consistently uses > 100% CPU and the osxfs process uses around 50%.

Using the new cached option for all of my mounts got this down to hyperkit using ~10% and osxfs using 0%.

I'm thrilled.

@geerlingguy

This comment has been minimized.

Show comment
Hide comment
@geerlingguy

geerlingguy Apr 12, 2017

For the benefit of anyone testing this functionality right now, if you're mounting a volume in docker run on the CLI, pass the option along with other volume options at the end of the --volume command, like so:

docker run --volume=/local/path:/path/in/container:rw,cached [other args]

And note that you have to be running the current Edge build of Docker (if you're on stable, you should uninstall Docker first, download the Edge installer from the Docker website, then drag that version to your Applications folder and set it up.

(These things may be obvious to most... but to save people reading through the last 25 or so comments; that's the best way to test the new functionality at this time.)

I just tested with a fairly massive PHP-based project—there's a task in the CI process that downloads tons of Composer dependencies (always slow, even with fast IO), installs a Drupal application, then runs a series of Behat (functional) tests. The test suite usually takes 3-4 minutes on my MacBook Pro, 4-5 minutes on Travis CI in the native environment, and here's how it ran inside Docker on my Mac:

Before, not using cached

28 min, 42 sec

After adding cached

10 min, 8 sec

Speedup of 97%. 💯

(Reproduced 2x, completely rebuilt the environment (docker rm -f container) each run for both scenario.)

This workload should benefit even more once the delegated option is available, because all those Composer dependencies are written to the volume as well (and there are thousands of files written during this operation!).

For the benefit of anyone testing this functionality right now, if you're mounting a volume in docker run on the CLI, pass the option along with other volume options at the end of the --volume command, like so:

docker run --volume=/local/path:/path/in/container:rw,cached [other args]

And note that you have to be running the current Edge build of Docker (if you're on stable, you should uninstall Docker first, download the Edge installer from the Docker website, then drag that version to your Applications folder and set it up.

(These things may be obvious to most... but to save people reading through the last 25 or so comments; that's the best way to test the new functionality at this time.)

I just tested with a fairly massive PHP-based project—there's a task in the CI process that downloads tons of Composer dependencies (always slow, even with fast IO), installs a Drupal application, then runs a series of Behat (functional) tests. The test suite usually takes 3-4 minutes on my MacBook Pro, 4-5 minutes on Travis CI in the native environment, and here's how it ran inside Docker on my Mac:

Before, not using cached

28 min, 42 sec

After adding cached

10 min, 8 sec

Speedup of 97%. 💯

(Reproduced 2x, completely rebuilt the environment (docker rm -f container) each run for both scenario.)

This workload should benefit even more once the delegated option is available, because all those Composer dependencies are written to the volume as well (and there are thousands of files written during this operation!).

@barat

This comment has been minimized.

Show comment
Hide comment
@barat

barat Apr 13, 2017

So if I understand correctly, for Symfony app we still need to wait for delegate to make something like this:

volumes:
    - .:/symfony:rw,cached
    - ./app/cache:/symfony/app/cache:rw,delegated
    - ./vendor:/symfony/vendor:rw,delegated
    - /symfony/app/logs

Or maybe I'm missing something?
It's just my idea how to have vendors and cache on host (PHPStorm uses cache for Symfony integration, and Vendors are needed for autocomplete) and logs only in container (no need to have them on host if I can tail/grep them inside container)

Docker-sync is nice, but I read here that cached reduces CPU overhead on osxfs and hyperkit which is massive improvement for mbp13 owner ... this little thing can be really hot while docker is messing with file writes/reads ...

barat commented Apr 13, 2017

So if I understand correctly, for Symfony app we still need to wait for delegate to make something like this:

volumes:
    - .:/symfony:rw,cached
    - ./app/cache:/symfony/app/cache:rw,delegated
    - ./vendor:/symfony/vendor:rw,delegated
    - /symfony/app/logs

Or maybe I'm missing something?
It's just my idea how to have vendors and cache on host (PHPStorm uses cache for Symfony integration, and Vendors are needed for autocomplete) and logs only in container (no need to have them on host if I can tail/grep them inside container)

Docker-sync is nice, but I read here that cached reduces CPU overhead on osxfs and hyperkit which is massive improvement for mbp13 owner ... this little thing can be really hot while docker is messing with file writes/reads ...

@kricha

This comment has been minimized.

Show comment
Hide comment
@kricha

kricha Apr 13, 2017

does :cached works only with beta version d4m?

kricha commented Apr 13, 2017

does :cached works only with beta version d4m?

@jeanlaurent

This comment has been minimized.

Show comment
Hide comment
@jeanlaurent

jeanlaurent Apr 13, 2017

Member

@aLkRicha yes, it's called edge now, and only starting on Docker 17.04 CE. See the FAQ entry on Stable and Edge if you want to switch.

Member

jeanlaurent commented Apr 13, 2017

@aLkRicha yes, it's called edge now, and only starting on Docker 17.04 CE. See the FAQ entry on Stable and Edge if you want to switch.

@Vanuan

This comment has been minimized.

Show comment
Hide comment
@Vanuan

Vanuan Apr 14, 2017

@barat I think currently it should be like this

  volumes:
    - .:/symfony:ro,cached
    - app_cache_volume:/symfony/app/cache
    - vendor_volume:/symfony/vendor
    - logs_volume:/symfony/app/logs

volumes:
  app_cache_volume:
  vendor_volume:
  logs_volume:

Yes, if you need to look into vendor files in your IDE you should wait for delegated.

For now you can use duplicate vendor in both docker and out of docker with manual synchronization. So that PHPStorm uses local version and docker uses volume in VM.

Vanuan commented Apr 14, 2017

@barat I think currently it should be like this

  volumes:
    - .:/symfony:ro,cached
    - app_cache_volume:/symfony/app/cache
    - vendor_volume:/symfony/vendor
    - logs_volume:/symfony/app/logs

volumes:
  app_cache_volume:
  vendor_volume:
  logs_volume:

Yes, if you need to look into vendor files in your IDE you should wait for delegated.

For now you can use duplicate vendor in both docker and out of docker with manual synchronization. So that PHPStorm uses local version and docker uses volume in VM.

@norio-nomura

This comment has been minimized.

Show comment
Hide comment
@norio-nomura

norio-nomura Apr 15, 2017

I tried :cached on building Swift toolchain for Linux that source codes are placed in mac volume.
It seems that it certainly got faster, but the memory consumption of osxfs was so great that it affected other applications.
I will continue to use the combination of docker-machine and nfs.

I tried :cached on building Swift toolchain for Linux that source codes are placed in mac volume.
It seems that it certainly got faster, but the memory consumption of osxfs was so great that it affected other applications.
I will continue to use the combination of docker-machine and nfs.

@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom Apr 18, 2017

@andrerom I understand that this is a new feature, but what about people that want to support development on multiple OSes. RHEL 6 and 7 are going to be around for a long time still, so you're saying people will need to keep multiple configs around for years just so :cached can be specified on their Macs?

@daveisfera Well, yes and no. Docker is despite all the hype pushing for it's use early adopter stuff, it is currently not mature enough for early majority users to use it broadly yet, has some known stability issues and a fragmented market. RHEL 6 is even late majority users, they by nature don't want to test out new things, so don't make them. RHEL 7 however is supported by docker packages, so once they migrate there they can take advantage of this. In other words research usage of docker for future version of your own software, not for older versions. At least not under all possible supported platforms you have there, that won't blend.

@andrerom I understand that this is a new feature, but what about people that want to support development on multiple OSes. RHEL 6 and 7 are going to be around for a long time still, so you're saying people will need to keep multiple configs around for years just so :cached can be specified on their Macs?

@daveisfera Well, yes and no. Docker is despite all the hype pushing for it's use early adopter stuff, it is currently not mature enough for early majority users to use it broadly yet, has some known stability issues and a fragmented market. RHEL 6 is even late majority users, they by nature don't want to test out new things, so don't make them. RHEL 7 however is supported by docker packages, so once they migrate there they can take advantage of this. In other words research usage of docker for future version of your own software, not for older versions. At least not under all possible supported platforms you have there, that won't blend.

@asteinlein

This comment has been minimized.

Show comment
Hide comment
@asteinlein

asteinlein Apr 18, 2017

I got unknown option: cached when using this in a compose file when deploying to a stack (docker stack deploy foo --compose-file=docker-compose.yml (I'm on 17.06.0-ce-rc1-mac8 edge). When will that be supported?

I got unknown option: cached when using this in a compose file when deploying to a stack (docker stack deploy foo --compose-file=docker-compose.yml (I'm on 17.06.0-ce-rc1-mac8 edge). When will that be supported?

@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom Apr 18, 2017

@asteinlein did you set version to 3.2?`, at least that is the thing needed on docker-compose.


For PHP/Symfony users posted numbers for :cached with eZ Platform. TL;DR works, but for any heavy REST based app doing parallel requests needed to set SYMFONY_ENV=prod.

andrerom commented Apr 18, 2017

@asteinlein did you set version to 3.2?`, at least that is the thing needed on docker-compose.


For PHP/Symfony users posted numbers for :cached with eZ Platform. TL;DR works, but for any heavy REST based app doing parallel requests needed to set SYMFONY_ENV=prod.

@toooni

This comment has been minimized.

Show comment
Hide comment
@toooni

toooni Apr 18, 2017

@andrerom you may just change the cache and log dir in AppKernel.php to a non shareded dir (for dev):

    public function getCacheDir()
    {
        if (in_array($this->environment, ['dev', 'test'])) {
            return '/var/www/symfony/cache/'.$this->environment;
        }

        return parent::getCacheDir();
    }

    public function getLogDir()
    {
        if (in_array($this->environment, ['dev', 'test'])) {
            return '/var/www/symfony/logs';
        }

        return parent::getLogDir();
    }

toooni commented Apr 18, 2017

@andrerom you may just change the cache and log dir in AppKernel.php to a non shareded dir (for dev):

    public function getCacheDir()
    {
        if (in_array($this->environment, ['dev', 'test'])) {
            return '/var/www/symfony/cache/'.$this->environment;
        }

        return parent::getCacheDir();
    }

    public function getLogDir()
    {
        if (in_array($this->environment, ['dev', 'test'])) {
            return '/var/www/symfony/logs';
        }

        return parent::getLogDir();
    }
@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom Apr 18, 2017

@toooni I did test with /tmp/symfony/ folder for cache and log but got better response times without for some reason, but I'll have a second look when I can.

andrerom commented Apr 18, 2017

@toooni I did test with /tmp/symfony/ folder for cache and log but got better response times without for some reason, but I'll have a second look when I can.

@asteinlein

This comment has been minimized.

Show comment
Hide comment
@asteinlein

asteinlein Apr 18, 2017

@andrerom Thanks for the tip, but same error about unknown option: cached with the compose file version set to 3.2 when deploying as a stack, unfortunately.

@andrerom Thanks for the tip, but same error about unknown option: cached with the compose file version set to 3.2 when deploying as a stack, unfortunately.

@rivaros

This comment has been minimized.

Show comment
Hide comment
@rivaros

rivaros Apr 18, 2017

@asteinlein maybe still wrong version of docker. here's mine

ross@MAC-ROSS:~/projects/DockerPack $ docker version
Client:
 Version:      17.05.0-ce-rc1
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   2878a85
 Built:        Tue Apr 11 20:55:05 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.05.0-ce-rc1
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   2878a85
 Built:        Tue Apr 11 20:55:05 2017
 OS/Arch:      linux/amd64
 Experimental: true

Btw your docker-compose should start like:

version: "3"

rivaros commented Apr 18, 2017

@asteinlein maybe still wrong version of docker. here's mine

ross@MAC-ROSS:~/projects/DockerPack $ docker version
Client:
 Version:      17.05.0-ce-rc1
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   2878a85
 Built:        Tue Apr 11 20:55:05 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.05.0-ce-rc1
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   2878a85
 Built:        Tue Apr 11 20:55:05 2017
 OS/Arch:      linux/amd64
 Experimental: true

Btw your docker-compose should start like:

version: "3"
@rivaros

This comment has been minimized.

Show comment
Hide comment
@rivaros

rivaros Apr 18, 2017

Personally staying with inotify/rsync solutions because of gulp watchers/builders inside containers
So waiting for "delegate". :)

rivaros commented Apr 18, 2017

Personally staying with inotify/rsync solutions because of gulp watchers/builders inside containers
So waiting for "delegate". :)

@MetalArend

This comment has been minimized.

Show comment
Hide comment
@MetalArend

MetalArend Apr 23, 2017

@rivaros could you share that inotify/rsync solution?

@rivaros could you share that inotify/rsync solution?

@carn1x

This comment has been minimized.

Show comment
Hide comment
@carn1x

carn1x Apr 28, 2017

Is there any high level estimate for when :delegate might start to appear? Just wondering if I should try and get all my devs converted to something like docker-sync which has a lot more initial/ongoing overhead, especially for our devs who are just putting up with our decision to enforce docker rather than enthusiastically embracing it!

carn1x commented Apr 28, 2017

Is there any high level estimate for when :delegate might start to appear? Just wondering if I should try and get all my devs converted to something like docker-sync which has a lot more initial/ongoing overhead, especially for our devs who are just putting up with our decision to enforce docker rather than enthusiastically embracing it!

@sanbor

This comment has been minimized.

Show comment
Hide comment
@sanbor

sanbor Apr 28, 2017

sanbor commented Apr 28, 2017

@bilby91

This comment has been minimized.

Show comment
Hide comment
@bilby91

bilby91 Apr 28, 2017

Is there any solution for mac that haves a decent speed ? Running docker in vagrant solves the write speed issue ?

bilby91 commented Apr 28, 2017

Is there any solution for mac that haves a decent speed ? Running docker in vagrant solves the write speed issue ?

@andrerom

This comment has been minimized.

Show comment
Hide comment
@andrerom

andrerom Apr 29, 2017

@sanbor @bilby91 As found earlier up in this thread, there are two main choices if :cached does not solve it for you:

  • nfs either using Docker for Mac via d4m-nfs or with DockerToobox (cleanest, only thing you need to take care of is paths which gets different)
  • docker-sync or similar rsync/event based syncing (probably fastest, but you'll need to change your setup for it, and you might run into sync issues)

andrerom commented Apr 29, 2017

@sanbor @bilby91 As found earlier up in this thread, there are two main choices if :cached does not solve it for you:

  • nfs either using Docker for Mac via d4m-nfs or with DockerToobox (cleanest, only thing you need to take care of is paths which gets different)
  • docker-sync or similar rsync/event based syncing (probably fastest, but you'll need to change your setup for it, and you might run into sync issues)
@ianks

This comment has been minimized.

Show comment
Hide comment
@ianks

ianks May 3, 2017

Why does the docker for mac engine not just mount NFS?

ianks commented May 3, 2017

Why does the docker for mac engine not just mount NFS?

@aimakun

This comment has been minimized.

Show comment
Hide comment
@aimakun

aimakun May 3, 2017

@ianks There are several reasons for not using NFS. The issue is current one Docker for Mac using still have own problems. Give the team more time to solve that.

As @andrerom mentioned above, you still can use workarounds. I'm still using docker/toolbox and it's working fine.

Even Docker for Windows has long way to go, currently it still only available on Windows 10 pro.

aimakun commented May 3, 2017

@ianks There are several reasons for not using NFS. The issue is current one Docker for Mac using still have own problems. Give the team more time to solve that.

As @andrerom mentioned above, you still can use workarounds. I'm still using docker/toolbox and it's working fine.

Even Docker for Windows has long way to go, currently it still only available on Windows 10 pro.

@lox

This comment has been minimized.

Show comment
Hide comment
@lox

lox May 3, 2017

Could mods close this thread to prevent more comments like @ianks? Lots of us want to keep tabs on updates on the issue and the sheer volume of "use nfs or docker-sync" comments makes that very annoying.

lox commented May 3, 2017

Could mods close this thread to prevent more comments like @ianks? Lots of us want to keep tabs on updates on the issue and the sheer volume of "use nfs or docker-sync" comments makes that very annoying.

@luckydonald

This comment has been minimized.

Show comment
Hide comment
@luckydonald

luckydonald May 3, 2017

@lox, You shouldn't kill discussion however.
Don't be so grumpy 😃

luckydonald commented May 3, 2017

@lox, You shouldn't kill discussion however.
Don't be so grumpy 😃

@ianks

This comment has been minimized.

Show comment
Hide comment
@ianks

ianks May 4, 2017

@lox It was a question, not a statement.

ianks commented May 4, 2017

@lox It was a question, not a statement.

@lox

This comment has been minimized.

Show comment
Hide comment
@lox

lox May 4, 2017

Search on this page for NFS @ianks. If that doesn't clarify your question, we can provide more detail.

lox commented May 4, 2017

Search on this page for NFS @ianks. If that doesn't clarify your question, we can provide more detail.

@EugenMayer

This comment has been minimized.

Show comment
Hide comment
@EugenMayer

EugenMayer May 4, 2017

To add something for the discussion, i made some benchmarks with :cached and the newest edge release today https://github.com/EugenMayer/docker-sync/wiki/Performance-Tests-2017

In the end, :cached will not really help with 17s, thats even far below the fusion based shares with 12s, not to mention when you compare it with native_osxm which has only 0.20s, near hte native speed with 0.19s.

we are talking about :cached beeing 50x slower then a native docker container or alternatives

EugenMayer commented May 4, 2017

To add something for the discussion, i made some benchmarks with :cached and the newest edge release today https://github.com/EugenMayer/docker-sync/wiki/Performance-Tests-2017

In the end, :cached will not really help with 17s, thats even far below the fusion based shares with 12s, not to mention when you compare it with native_osxm which has only 0.20s, near hte native speed with 0.19s.

we are talking about :cached beeing 50x slower then a native docker container or alternatives

@herophuong

This comment has been minimized.

Show comment
Hide comment
@herophuong

herophuong May 4, 2017

@EugenMayer, :cached is used for improving read speed, and you're testing write speed. You will have to wait for :delegate for that benchmark.

@EugenMayer, :cached is used for improving read speed, and you're testing write speed. You will have to wait for :delegate for that benchmark.

@Vanuan

This comment has been minimized.

Show comment
Hide comment
@Vanuan

Vanuan May 5, 2017

Looks like cached/delegated wouldn't solve the issue. People wouldn't know when to apply what.

Vanuan commented May 5, 2017

Looks like cached/delegated wouldn't solve the issue. People wouldn't know when to apply what.

@EugenMayer

This comment has been minimized.

Show comment
Hide comment
@EugenMayer

EugenMayer May 5, 2017

For sure it would not, and not even because this implicates, that the container has the superior over the host - which is right the opposite what you need when you actually develop. The container will hardly ever surprise you with something "new interesting" - and if anything, its generated and reproducible.

Very very different for code changes, which are not reproducible and very hard to write again ( mentally alone ). Thats why docker-sync uses always "hosts wins" while still using the buffer. Losing code is just not funny - losing generated assets (in case it ever happens to be a conflict) is a no care.

so :delegate is just something you do never want to touch - even if i do get the general idea.

For sure it would not, and not even because this implicates, that the container has the superior over the host - which is right the opposite what you need when you actually develop. The container will hardly ever surprise you with something "new interesting" - and if anything, its generated and reproducible.

Very very different for code changes, which are not reproducible and very hard to write again ( mentally alone ). Thats why docker-sync uses always "hosts wins" while still using the buffer. Losing code is just not funny - losing generated assets (in case it ever happens to be a conflict) is a no care.

so :delegate is just something you do never want to touch - even if i do get the general idea.

@docker docker locked and limited conversation to collaborators May 5, 2017

@yallop

This comment has been minimized.

Show comment
Hide comment
@yallop

yallop May 5, 2017

Contributor

Subscribers to this issue might be interested to read the following blog post:

    User-guided caching in Docker for Mac

which explains how and when to use cached, along with some details about the implementation.

Since there are now significant improvements in the use cases that prompted this issue, we're closing the issue to further conversation. We'll continue to post occasional updates here when further performance improvements are available. There's a new, more focused issue open to track the progress of the caching implementation (i.e. the implementation of delegated, and further improvements to cached):

    File system performance improvements

If you'd like to follow developments closely, we recommend subscribing there. Thanks to everyone for contributing to this conversation!

Contributor

yallop commented May 5, 2017

Subscribers to this issue might be interested to read the following blog post:

    User-guided caching in Docker for Mac

which explains how and when to use cached, along with some details about the implementation.

Since there are now significant improvements in the use cases that prompted this issue, we're closing the issue to further conversation. We'll continue to post occasional updates here when further performance improvements are available. There's a new, more focused issue open to track the progress of the caching implementation (i.e. the implementation of delegated, and further improvements to cached):

    File system performance improvements

If you'd like to follow developments closely, we recommend subscribing there. Thanks to everyone for contributing to this conversation!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.