Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Synchronization stops. #517

Closed
elbsurfer opened this Issue Jan 3, 2018 · 159 comments

Comments

Projects
None yet
@elbsurfer
Copy link

elbsurfer commented Jan 3, 2018

Hello,
I use a pretty simple docker sync setup with these three files:

docker-compose-dev.yml

version: "2"
services:
  apache:
    volumes:
      - ./docker-config/vhost:/etc/apache2/sites-enabled/000-default.conf
      - rr-sync:/var/www/html:nocopy # nocopy is important

volumes:
  rr-sync:
    external: true

docker-compose.yml

version: '2'
services:
  apache:
    image: bylexus/apache-php7
    ports:
      - 80:80

  db:
    image: orchardup/mysql
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: rr

docker-sync.yml

version: "2"

options:
  verbose: true
syncs:
  rr-sync: # tip: add -sync and you keep consistent names as a convention
    src: './src'
    sync_excludes: ['.git']

Now it happens that the sync does start and I can visit the website on my local mashine. However when I change something nothing get's synced. When I then stop the synchronization and start the sync again with docker-sync-stack start, the latest changes are synced, but why is this not consistently the case and how I can change it?

I use MacOS Sierra 10.12.6 with the Docker version Version 17.09.0-ce-mac35 (19611) and unison sync. Thanks!

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jan 3, 2018

please try to reproduce this with https://github.com/EugenMayer/docker-sync-boilerplate

in the folder 'default'

@elbsurfer

This comment has been minimized.

Copy link
Author

elbsurfer commented Jan 3, 2018

Hello Eugen,
thanks for your quick response. I have used the default example and I see this:

Every 3s: cat /var/www/index.html                           2018-01-03 18:08:54
app-native-osx_1  |
app-native-osx_1  | inital, now change app/index.html and see if this will change me

When I change the file index.html, I do not see the change in the terminal. Sometimes I see a spinner in the terminal, but no change on the screen.

@elbsurfer

This comment has been minimized.

Copy link
Author

elbsurfer commented Jan 4, 2018

I restarted my Mac and tried the default example again - this one is working now. Also my other configuration seems to work now. I do not know what happened here before my restart...

@jameshalsall

This comment has been minimized.

Copy link

jameshalsall commented Jan 8, 2018

I am also seeing this issue, it works for a short time and then can stop randomly.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jan 8, 2018

are you using docker4mac edge?

@jameshalsall

This comment has been minimized.

Copy link

jameshalsall commented Jan 8, 2018

Yeah (Version 17.12.0-ce-mac45 (21669))

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jan 8, 2018

that should be your problem use stable

@jonaswouters

This comment has been minimized.

Copy link

jonaswouters commented Jan 9, 2018

@EugenMayer I'm guessing the problem lies with 17.12.0-ce, which is currently the new stable.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jan 9, 2018

i cannot work on those issues because all ppl give as information is 'it does' not work, fixing it by doing either nothing (just waiting), doing upgrades, some do downgrades, some do restarts

and nobody can make any sense about all this. there is litterellay no information or debuging in those reports. It's more a 'its broken, how to fix it/can somebody fix it for me'

the short answer to this is: no

Reasons vary from:

  • not reproducible on other systems and/or projects
  • way to much information is missing to make any assumption
  • no effort has been put into deducting or reducing to nail down or circle in
  • time. thats an free OSS project - get yourself involved or wait for the wonder

For example, i am running the latest docker4mac edge here:

image

no problems in either direction. Doing the same with the current stable, same result.
So if you expect me to magically getting involved, creating OSX VMs with your particular APFS/High Sierra and some project, you will wait forever.

Guys, take you time and finally start debugging that. You disagree on basically any matter, be it the d4m version, down or upgrading fixes it, be it a FS issue or a project issue. Get yourself a shell, use https://github.com/EugenMayer/docker-sync/wiki/8.-Strategies#native_osx-osx and
https://github.com/EugenMayer/docker-sync/wiki/8.1.-native_osx-sync-strategy-debugging-guide and start acting

We did not create this for the matter of too much time - rather you getting involved and being able to help yourself when you need it most with the timeframe you are in urge or not.

But just opening a new issue after issue with "it does not work / any longer" and thats it - will not do it.

@jonaswouters

This comment has been minimized.

Copy link

jonaswouters commented Jan 10, 2018

@EugenMayer The debugging guide did not help me sadly. And I don't think we disagree on anything, just confusion. The edge version became stable only very recently. That is why we have this edge/stable confusion.

I'm trying more stuff, but I've been debugging this for too long this night with no results. I'm now downgrading to the previous stable to confirm at least that it is the latest release.

As for replicating. After a docker-sync clean, and running it again, it works for me. But after restarting the stack, it no longer works.

I do agree that it is not your responsibility. This is OSS, and you've been very helpful so far. Thanks for that

@Krzysiaczek-at-theFoundry

This comment has been minimized.

Copy link

Krzysiaczek-at-theFoundry commented Jan 10, 2018

Yesterday, there was Docker for Mac update which I installed Version 17.12.0-ce-mac46 (21698) Stable and there was also an auto-update suggestion to 0.52. Until yesterday, I was working on docker-sync 0.46. It has half "failed" half "succeed" when I agreed to upgrade due to some permissions problems.

desktop-r9kn7ds:docker kuba$ docker-sync-stack start
     warning  There is an update (0.5.2) available (current version 0.4.6
). Please update before you continue
Shall I update docker-sync to 0.5.2 for you? yes
Updating installed gems
Updating docker-sync
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /Library/Ruby/Gems/2.0.0 directory.
     success  Successfully updated, please restart docker-sync and check the changelog at https://github.com/EugenMayer/docker-sync/wiki/5.-Changelog

So this looks like the beginning of the problems as in one line it says "success" just after "ERROR".

I've tried to give it a try and here are the results:

desktop-r9kn7ds:docker kuba$ docker-sync-stack start
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
       note:  You can also run docker-sync in the background with docker-sync --daemon
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
          ok  Starting unison for sync web-sync
          ok  web-sync container not running
          ok  starting web-sync container
     command  docker start web-sync && docker exec web-sync supervisorctl restart unison
          ok  starting initial sync of web-sync
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' web-sync
     command  docker inspect --format='{{(index (index .NetworkSettings.Ports "5000/tcp") 0).HostPort}}' web-sync
       error  Error getting mapped port, exit code 0
     message  Template parsing error: template: :1:3: executing "" at <index (index .Networ...>: error calling index: index of untyped nil
     command  unison -testserver /app_sync "socket://:"
/Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_strategy/unison.rb:191:in `block in start_container': Failed to start unison container in time, try to increase max_attempt in your configuration. See https://github.com/EugenMayer/docker-sync/wiki/2.-Configuration for more informations (RuntimeError)
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_strategy/unison.rb:187:in `loop'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_strategy/unison.rb:187:in `start_container'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_strategy/unison.rb:39:in `run'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_process.rb:87:in `run'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_manager.rb:100:in `block in run'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_manager.rb:99:in `each'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_manager.rb:99:in `run'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/tasks/stack/stack.thor:47:in `start'
	from /Library/Ruby/Gems/2.0.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
	from /Library/Ruby/Gems/2.0.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
	from /Library/Ruby/Gems/2.0.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
	from /Library/Ruby/Gems/2.0.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/bin/docker-sync-stack:15:in `<top (required)>'
	from /usr/local/bin/docker-sync-stack:23:in `load'
	from /usr/local/bin/docker-sync-stack:23:in `<main>'

I'm not sure if problem with port 5000 could be a source of the problem or maybe it is the result of half failed/broken upgrade installation?

Similar errors when stopping it:

desktop-r9kn7ds:docker kuba$ docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS               NAMES
82cde2d87f98        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh supe…"   4 weeks ago         Up 57 seconds       500/tcp             web-sync

desktop-r9kn7ds:docker kuba$ docker-sync stop
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
        info  Not running any precondition checks since you have no brew and that is unsupported. Is all up to you know.
          ok  Stopping sync container web-sync
/Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/watch_strategy/unison.rb:30:in `kill': no implicit conversion from nil to integer (TypeError)
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/watch_strategy/unison.rb:30:in `stop'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_process.rb:93:in `stop'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_manager.rb:135:in `block in stop'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_manager.rb:134:in `each'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/lib/docker-sync/sync_manager.rb:134:in `stop'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/tasks/sync/sync.thor:65:in `stop'
	from /Library/Ruby/Gems/2.0.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
	from /Library/Ruby/Gems/2.0.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
	from /Library/Ruby/Gems/2.0.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
	from /Library/Ruby/Gems/2.0.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
	from /Library/Ruby/Gems/2.0.0/gems/docker-sync-0.4.6/bin/docker-sync:14:in `<top (required)>'
	from /usr/local/bin/docker-sync:23:in `load'
	from /usr/local/bin/docker-sync:23:in `<main>'

I've decided to force it this time and run again full installation command gem install docker-sync. It failed with some permission error so I had to repeat it as sudoer (I'm not sure if this is a proper way?)

desktop-r9kn7ds:docker kuba$ gem install docker-sync
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /Library/Ruby/Gems/2.0.0 directory.
desktop-r9kn7ds:docker kuba$ sudo !!
sudo gem install docker-sync
Password:
Fetching: docker-compose-1.1.7.gem (100%)
Successfully installed docker-compose-1.1.7
Fetching: terminal-notifier-2.0.0.gem (100%)
Successfully installed terminal-notifier-2.0.0
Fetching: docker-sync-0.5.2.gem (100%)
Successfully installed docker-sync-0.5.2
Parsing documentation for docker-compose-1.1.7
Installing ri documentation for docker-compose-1.1.7
Parsing documentation for terminal-notifier-2.0.0
Installing ri documentation for terminal-notifier-2.0.0
Parsing documentation for docker-sync-0.5.2
Installing ri documentation for docker-sync-0.5.2
3 gems installed
desktop-r9kn7ds:docker kuba$ docker-sync -v
0.5.2

After this everything seemed to be fine, maybe even a little bit faster than usual, but only for a few minutes. The changes from the host have stopped to be populated into the container. It was confusing as I started to use docker-sync-stack start command to watch the log for sync output and I've seen something but I found that it works only in one direction from /app_sync to /host_sync.

I've stopped and restarted it and everything was fine again but just for a while.

After next restart, I wanted to measure the time but everything was fine and /host_sync to /app_syncwas working fine when I was saving the changes back and forth for the same file (and nothing else) for a few minutes.

Suddenly it stopped to sync and I realised that it was not time-related but it stopped to work after I started to use application and opposite direction sync has happened from /app_sync to /host_sync. After that host to the app has stopped to react to changes of the file.

I've followed your advice and wanted to provide some debugging details using directions from https://github.com/EugenMayer/docker-sync/wiki/8.1.-native_osx-sync-strategy-debugging-guide

Unfortunately I've got no output from the this debugging command which is the reason why there is no sync triggered. Basically the changes on the host files become undetected

desktop-r9kn7ds:web kuba$ echo $DEBUG_DOCKER_FILE
docroot/includes/commerce/DiscountManager.php
desktop-r9kn7ds:web kuba$ echo $DEBUG_DOCKER_SYNC
web-sync
desktop-r9kn7ds:web kuba$ diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/host_sync/$DEBUG_DOCKER_FILE")

Some time ago I've asked to provide instructions how to downgrade docker-sync to the previous version. I'm not familiar with this whole gem installation and as I guess it works with some remote URL to get recent version there should be some URLs provided with older one. I can't find anything on the wiki how to install from the downloaded zip files as well.

Can someone provide such details and update the Wiki, please?

@Krzysiaczek-at-theFoundry

This comment has been minimized.

Copy link

Krzysiaczek-at-theFoundry commented Jan 10, 2018

I've just downgraded DFM to Version 17.09.0-ce-mac35 (19611) and it seems to work with docker-sync 0.5.2 for now.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jan 10, 2018

@Krzysiaczek-at-theFoundry finally someone doing the whole round-robbin with a conclusion.

So we have a bug in the latest stable, but after some time. downgrading to 17.09.0-ce-mac35 stable helps - while using 0.5.2 - that sounds like a good solution.

One questions @Krzysiaczek-at-theFoundry are you using APFS?

@Krzysiaczek-at-theFoundry

This comment has been minimized.

Copy link

Krzysiaczek-at-theFoundry commented Jan 10, 2018

Nope, I'm still on (low) Sierra.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jan 10, 2018

same here, so beside your findings, AFPS could be still a variant

means anybody with AFPS and this issue who can verify that that downgrade helps him too?

@phillipsnick

This comment has been minimized.

Copy link

phillipsnick commented Jan 10, 2018

Also having the same issue here after updating to 17.12.0-cs-mac46. Spent almost an hour trying to downgrade Docker (with little success) so far to see if that resolves the issue.

@phillipsnick

This comment has been minimized.

Copy link

phillipsnick commented Jan 10, 2018

Downgraded to 17.09.0-ce-mac33 2017-10-03 (Stable) from https://docs.docker.com/docker-for-mac/release-notes/

Synchronisation working again with docker-sync 0.5.2

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jan 10, 2018

@phillipsnick great news, APFS / high sierra?

@jonaswouters

This comment has been minimized.

Copy link

jonaswouters commented Jan 10, 2018

I've also downgraded to 17.09.0-ce-mac33 and I've used it all day. It is definitely more stable, but I am able to break it by mounting the same docker-sync volume by 2 containers :) I'll skip the 2nd container tomorrow and see if it keeps working.

@kudos

This comment has been minimized.

Copy link

kudos commented Jan 10, 2018

I'm running 10.13 with APFS and Docker 17.12.0 and docker-sync 0.5.2.

I ran some tests and I'm finding the initial sync takes almost twice as long as it used to for the same codebase (~11 minutes versus ~6 minutes), and fs events are not being captured at all.

Even restarting after an initial sync takes just over 2 minutes, even though there's no changes to sync. I don't know how long it used to take, but it was quick enough that I never bothered measuring it.

@EugenMayer thanks for your work on this to date, and hopefully this is a helpful datapoint.

@brianmorton

This comment has been minimized.

Copy link

brianmorton commented Jan 10, 2018

@phillipsnick care to share how you successfully downgraded docker, and save the rest of us an hour?

@brianmorton

This comment has been minimized.

Copy link

brianmorton commented Jan 10, 2018

Alright I took the 20 minutes and figured it out myself. There are a couple of quick-ish ways.

The docker release notes now include links to older versions (after much arguing on github):
https://docs.docker.com/docker-for-mac/release-notes/

This 3rd party site is listing the releases:
https://jsok.github.io/docker-for-mac-versions/

If you know the build number you want, you can get the dmg yourself. For example, 21090 is for 17.09.1-ce-mac42 which is what I will be trying out right now to see if this fixes my issue. The URL pattern is:

https://download.docker.com/mac/stable/{{build}}/Docker.dmg

So for 21090 it is:
https://download.docker.com/mac/stable/21090/Docker.dmg

If you don't know the build number you want, somebody posted a handy little bash script that programmatically tries all build numbers to see if the build exists and lets you know:

channel=stable
for build in $(seq 21581 19125); do
  if curl -X HEAD -fsSL -I https://download.docker.com/mac/$channel/$build/Docker.dmg >/dev/null 2>/dev/null; then
    echo "Found build: $build"
    curl -X HEAD -fsSL -I https://download.docker.com/mac/$channel/$build/Docker.dmg
  fi
done

Change the sequence numbers on line 2 to define your range. You can set the first number to your currently broken build number (which you can find by using the menubar widget > about docker) and you can probably leave the second number alone unless you want a really old build.

I hope this helps others.

@brianmorton

This comment has been minimized.

Copy link

brianmorton commented Jan 10, 2018

Or if you use homebrew and you want the easy way:

brew cask uninstall docker
brew cask install https://raw.githubusercontent.com/caskroom/homebrew-cask/61f1d33be340e27b91f2a5c88da0496fc24904d3/Casks/docker.rb

The second command just installs the version of docker before the homebrew recipe was updated to include the latest, so you will still get build 21090 by doing this.

@phillipsnick

This comment has been minimized.

Copy link

phillipsnick commented Jan 10, 2018

@EugenMayer El Capitan 10.11.1 (I know old ha), won't be APFS due to not updating in so long.

docker-sync.yml

version: '2'

options:
  verbose: true
  max-attempt: 30
syncs:
  admin-sync:
    src: './admin'
    sync_strategy: 'native_osx'
    sync_userid: '33'

@brianmorton nice find with the 3rd party site. Any idea why theres so many different builds per version?

@kudos

This comment has been minimized.

Copy link

kudos commented Jan 11, 2018

@phillipsnick they most likely run a build for each merge to master.

@arjun810

This comment has been minimized.

Copy link

arjun810 commented Jan 11, 2018

Had the same issue after upgrading: syncing wasn't working. Worked after downgrading. On Sierra.

@erlandsona

This comment has been minimized.

Copy link

erlandsona commented Jan 11, 2018

Okay, I'll do my best to provide some extra context.

I've been having this issue when changing between git branches in other words when lots of files change at the same time.

Figuring out certain directories to add to sync_excludes seems to help but I have a hunch this is happening because of memory limits. I'm running the older docker4mac as proposed previously in this thread
OSX Sierra
Docker-Sync: 0.5.2
Docker4Mac:

  • Version 17.09.1-ce-mac42 (21090)
  • Channel: stable
  • 3176a6af01
  • CPU's: 4
  • Memory: 8gb

Following https://github.com/EugenMayer/docker-sync/wiki/8.1.-native_osx-sync-strategy-debugging-guide

I discovered the issue only affected one of the syncs for some reason. So I followed the steps provided and discovered this in the /app_sync. Logs:

❯ diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/app_sync/$DEBUG_DOCKER_FILE")
Files src/controllers/base/some_base_controller.js and /dev/fd/63 differ

Here's the output of docker exec "$DEBUG_DOCKER_SYNC" tail -n70 /tmp/unison.log
https://hastebin.com/oronuxupod.sql

Then I:

  • docker-sync-stack clean|started with the sync_args: "-copyonconflict -debug verbose"
  • git checkout develop
  • wait a second or two?
  • git checkout my-feature-branch

Latest output of docker exec "$DEBUG_DOCKER_SYNC" tail -n70 /tmp/unison.log
https://hastebin.com/ifitasapul.sql

That's about the best I have for now.
Let me know if there's anything else I can provide to help track down what might be causing this issue.

Cheers! 🍻

@SilberMa

This comment was marked as off-topic.

Copy link

SilberMa commented May 31, 2018

@derschatta I am using the native synchronization of d4m.

Maybe it is just a lucky punch.
I will test it with a symfony 3.4.8 project during the day and will give a report.

@nilbus

This comment has been minimized.

Copy link

nilbus commented May 31, 2018

Thanks for your reports @derschatta.

@SilberMa It’s good to know that there’s an alternative that is performant enough for some people, and I think that is well-understood now. 👍 This thread is for reporting on how docker-sync is working with recent versions of d4m. Please keep the comments relevant, per @EugenMayer’s requests.

@oachoor

This comment was marked as off-topic.

Copy link

oachoor commented May 31, 2018

I'm running a Symfony 3.4.11 project (MySQL, Redis, Elasticsearch, NPM-Watch) on Docker CE Edge 18.05.0-ce-mac66 (24545) since 4 days with :cached flag, the cpu-usage, speed and the synchronization are just tip-top.

@derschatta

This comment has been minimized.

Copy link

derschatta commented Jun 1, 2018

I tried running on :cached flag but there's still a huge difference in speed compared to using docker-sync (cached = 2s loading time vs. docker-sync: 0.5s loading time). It might be ok to work with it but actually I rather see docker-sync working as expected. For now mac66 works fine with my docker-sync setup 👍

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jun 1, 2018

My Specs: Sierra, raw, mac66 edge

Important, eventhough i run with raw image on Sierra, there are reports that this stuck in the start process. Maybe just upgrade to HS then


I am running mac66[edge] for 2 days now on a fairly big project and i can second that several things are changing:

TLTR: docker-sync seems to work very stable with mac66 edge for now

But there are a lot more changes:

  1. docker-compose is much more performant now and seems to have no timeouts anymore ( about 4 minute stall on commands after about 20 dc commands. This happens for anything, like dc ps or dc exec while at the same time docker ps/exec was working fine. All this thing are gone

  2. The rather big delay on shutting down a stack with CTRL+C has gone

  3. The big delay on doing dc down on bigger stack is gone

  4. the very long delay on creating service is entirely gone for bigger stack. Starts instantly as under linux

  5. The overall CPU usage on file IP like docker cp into the containers and from is significant lower

  6. The CPU usage overall seems for less laggy

  7. dc has been significanly updated including all the fixes and glitches of current linux-related stacks ( thats good for consistency )

  8. dc pull works a lot faster now, seems the same i/o benefit as seen above


Bottom line is, that mac66 edge can be really a good release for a switch. It is fixing this very issue for stopping sync but also includes significant improvments overall esp. in terms of performance


@derschatta @SilberMa @watermanio @MatthiasKuehneEllerhold please stop discussing any issue not related to docker-sync synchronization stops with post mac42 in here. I understand that you have important findings you want to share - create a new issue in docker-sync or d4m and share your insight, benchmarks and things - but stop taking the people in this issue as hostages for a different topic. I can comment on a new issue discussing in skipping docker-sync entirely, but i wont here.

@ktruehl

This comment has been minimized.

Copy link

ktruehl commented Jun 1, 2018

I've been testing mac66 edge for nearly a week now. It's not as bad as anything in between mac42 and mac66 edge, and indeed synchronization has only stopped once for me in the last week. However, regularly the Docker process starts hogging CPU. We're using docker for huge projects (Magento, and even worse Magento 2), and switching a branch or recompiling in Magento 2 can cause the Docker process to suddenly start using around 250% CPU (quad core i7 with hyperthreading and whatever that feature is called that overclocks one or two cores). I have noticed that with mac42, too, but much more rarely than with mac66 edge.

So, once the container CPU usage stayed high (I used docker stats to identify the guilty container) even after more than 30 minutes (yes, I gave unison plenty of time to finish syncing whatever it needed to sync), I checked out the processes within the affected container. Unison was running at full CPU. I restarted unison with supervisorctl and after the initialization the unison process went down to about 3% CPU usage. However, the Docker process was still using around 250% CPU. So I waited some more. No improvement. So I stopped the guilty container. And voilà, Docker CPU usage went down to normal. I restarted the affected container, and unfortunately the CPU usage went up to around 250% again. The only thing that fixes it, is docker-compose stop && docker-sync stop and then restarting docker (this is apparently not always necessary) and then docker-sync-stack start.

Conclusion for our setup: While I could use mac42 for days at end without having to completely restart Docker, I have to restart Docker at least once but usually several times a day with mac66 edge. So mac42 is still a lot better for me.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jun 1, 2018

@ktruehl thank you for the elaborated report. In general, branch switches, with our without docker-sync will cause troubles with mac42+ when you do not take care.

if, the next time, this happens again, try to connect to your xhyve VM:

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
2x enter

use top to identify which process is running wild. AFAIU its not unison of docker-sync since you restarted it, right. so it might be something in the xhyve VM ( which gets restarted when you restart docker ).

Probably /etc/init.d/020-containerd is for the docker-engine, i am not sure

Its an alpine vm but we cannot modify anything

apk add --update htop
ERROR: Unable to lock database: Read-only file system
ERROR: Failed to open apk database: Read-only file system

there is a docker-engine up and running (obviously) - but it is nicely transparent to us.

ps aux | grep dockerd
 1163 root       0:52 /usr/local/bin/dockerd -H unix:///var/run/docker.sock --config-file /run/config/docker/daemon.json --swarm-default-advertise-addr=eth0 --userland-proxy-path /usr/bin/vpnkit-expose-port

Maybe you can find at leat the service / binary this is cause by and we either can report this to d4m or we could try to gracefully restart it with a HUP or similar.

@derschatta

This comment has been minimized.

Copy link

derschatta commented Jun 1, 2018

@EugenMayer thanks for testing mac66 for yourself. Regarding discussing not related topics: I never discussed anything else than docker-sync on mac66. I emphasized the performance difference to make clear why I use docker-sync and want to find a way to use it with anything newer than mac42. Seems we found it :).

@grigoryosifov

This comment has been minimized.

Copy link

grigoryosifov commented Jun 6, 2018

If Docker mac66 continues to generate .qcow2 file after Reset to factory defaults, don't "hack" it by manually changing the .json file as described above! Upgrade your OS X instead to at least 10.13.4!

I've spent a lot of hours not understanding why there are so many success stories when I either got docker-sync missing a lot of files or randomly exiting containers with "Segmentation fault", with .raw enabled. Then finally I decided to upgrade OS X and it worked!

As you can see in this comment: docker/for-mac#2625 (comment) they had disabled .raw format by default for older OS X versions because of an APFS corruption bug with sparse files and then after the bug was fixed in OS X 10.13.4 they have re-enabled it.

Now I have working docker-sync so fingers crossed it'll be more stable than mac42 on our heavy Symfony project!

P.S. @EugenMayer, that might be misleading:

Important, even if you are on Sierra and not High Sierra you need to use .raw and not .qcow2 - its broken for non AFPS filesystem as well - just never use qcow2

Maybe you'd want to edit it a bit, as I expect most people to prefer reading your comments (at least I do).

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jun 7, 2018

@grigoryosifov thank you for the elaboration, really nice work gets done here nowdays! Interestingly i had this raw-image issue only on 1 of 4 of my devices - but i had it. Changed my comment above, better safe then sorry, right :)

@derschatta i maybe just got you wrong - excuse myself. Probably lost in the ether of comments. Sorry!

Mac67 is out with some promising additional fixes:

i have no real field test, so people do not just upgrade production instances. Test it first, add some comments. I expect 67 to be just a little polished 66, so no real issues added, rather some fixed

@maio1980

This comment has been minimized.

Copy link

maio1980 commented Jun 11, 2018

docker-edge-ma67-ps

docker-edge-ma67-cpu

Maybe the CPU problems don't was fixed... (Docker-EDGE-18.05.0-ce-mac67)

@nilbus

This comment has been minimized.

Copy link

nilbus commented Jun 12, 2018

Running successfully on 10.13.5 (High Sierra) using .raw (default) with D4M edge version mac67 since 6/8/2018, syncing a large (40k synced files) Rails app. 👍🎉

It works at least as well as mac42 did for me as an end user. I had not benchmarked mac42 CPU usage, so I can't compare. Docker is averaging maybe 25% CPU on 1 core with containers largely idle (according to ctop), but CPU usage does not change when I'm not using docker-sync. This thread is about sync performance though, not docker CPU usage (likely) unrelated to docker-sync.

I will post if I have any problems later.

@michalkleiner

This comment was marked as off-topic.

Copy link

michalkleiner commented Jun 12, 2018

Also running successfully on mac67 on 10.13.5 with .raw - even without docker-sync, the file events propagation is immediate and performance is perceived as native. Happy for this comment to be deleted as it's only vaguely docker-sync related. I'm so happy about the current status with mac67 that I had to share it as well.

sloria added a commit to sloria/dotfiles that referenced this issue Jun 16, 2018

Use docker-edge
For compatibility with docker-sync
EugenMayer/docker-sync#517 (comment)
@charrondev

This comment has been minimized.

Copy link

charrondev commented Jun 20, 2018

After this never working for me before, I finally have it working consistently with a large PHP + React app (~43k files), but only with the following setup.

  • MacOS 10.3.5 (High Sierra) This is the most important part. 10.3.3 did not work well.
  • Latest version of docker-sync 0.5.7.
  • Docker for Mac (mac67). This is currently an edge version, but I'm sure it will make it to stable shortly enough.
  • A .raw docker disk image. I'm not sure if it would work with the other format. Once this version of D4M hits stable, I'll have another member of my team with the old format try it out.
  • A re-created my docker-sync volume from scratch. Conflicts caused by previous syncing issues were breaking sync for me.
  • We are excluding node_modules and .git from syncing (We have ~20 repos total syncing which each have their own).

The latency of our app running is almost equal to native with this setup. We also run our unit tests in the container, and they run ~3x faster than without docker sync.

@norpan

This comment has been minimized.

Copy link

norpan commented Jun 21, 2018

I can confirm it works well, at least for a few hours, after upgrading to mac67.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jun 21, 2018

@charrondev you just have been a little quicker and those are great news! :

Indeed we did the same transition across the company the last 3 weeks.

  • we migrated to High Sierra APFS
  • edge mac67

The overall result was fairly impressive, the performance is a lot better overall with d4m, still docker-sync is a must and helps significant. We run docker-sync all day, yet we had only to restart it once over the last whole week - that is a very good result for now.

Its a huge project, over 50k files.

The reason we mirgrate to High Sierra were, that mac67 edge does not work in RAW with HFS, it can crash and stuck a very ugly way, destroy the whole d4m installation and does no longer allow you to start it now matter what you do. this happens with 1 of 2 boxes with Sierra, not with every box, but the impact was significant enough for us to switch. Especially our High Sierra pilot was very promising since the overall performance with APFS + raw is a huge boost

@fgrehm

This comment has been minimized.

Copy link

fgrehm commented Jun 21, 2018

Things are doing good on my end working on some of the same projects as @nilbus, but I'm on the Sierra + qcow + D4M 67 + docker-sync 0.5.7 combo. I can also confirm that it works after sleep / resume for 2 / 3 days in a row (which used to brake syncing for me in some old version > mac42)

@nilbus

This comment has been minimized.

Copy link

nilbus commented Jun 22, 2018

Today on day 10, sync started failing. It would sync the files (after a short delay) each time I started docker sync, but subsequent changes were not detected and synced. See my setup above.

Update: I found that changes were not being synced by Docker to the volume mount inside the sync container. Restarting Docker resolved the problem this time… I'm not sure what the problem was.

Edit: I was mistakenly looking in app_sync instead of host_sync, which means I know even less about what the state of things was. I'll know where to look next time if this happens again.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jul 22, 2018

I am closing this issue since I think we have a winner: we are working with d4m edge for weeks now, with a sync stop per week. also other reports here were pretty positive. I will open up a new issue to implement a notice that people should upgrade to the latest edge when the run d4m, so people do not get into trouble that often, without knowing that.

@EugenMayer EugenMayer closed this Jul 22, 2018

@brianmorton

This comment has been minimized.

Copy link

brianmorton commented Jul 23, 2018

What version is the edge that is working?

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Jul 23, 2018

edge 67 .. but you just would have to read 2 posts above.

@zedtux

This comment has been minimized.

Copy link

zedtux commented Aug 8, 2018

@EugenMayer you should consider updating the README.md to inform people about this, don't you?

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Aug 8, 2018

@zedtux not the readme, since it kind of content-less but the wiki for sure.

Mostly people do not read the docs anyhow, so my first action is rather a dependency test when starting docker-sync and/or a confirmation that "i know that.." .. this helps more and people have an easier time finding it.

Feel free to help with PRs if you find any free time right now, i am currently too busy, sorry

@zedtux

This comment has been minimized.

Copy link

zedtux commented Aug 8, 2018

Seems fair to me.

@derschatta

This comment has been minimized.

Copy link

derschatta commented Feb 25, 2019

Anyone still has any problems with this? For a while it worked for me but recently this issue came back. Can't tell yet with which version. But I tried an older version, I believe edge 18.05.0-ce-mac67 and the newest stable 2.0.0.3. With both synchronisation stops a some point. CPU load goes up and I can only get it back working by shutting down all containers, stopping docker-sync and start both up again. Then it works for a while and then stops again. Quite annoying.

I've upgraded Mac OS to Mojave recently but if I remember correctly the issue was already there before.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Feb 25, 2019

Everybody has this issue and it will never stop. Especially when using npm based projects where during watch/rebuilds a lot of files are created/removed, this one gets problematic.

In most cases, this has nothing to do with docker-sync at all, but with OSXFS getting stucked in the FS event queue, which then also stops events for unison in our docker image ( linux, so inode events ) and thus breaks syncing.

There is no fix to that - sometimes restarting docker sync helps, this means, that is an actual unison "too many events" issue, which can also happen for unison.

So what we have here:
a) OSXFS gets stuck -> docker for mac restart ist the only option, sometimes even only OS restart
b) unsion gets stuck -> restarting docker-sync with docker-sync stop && docker-sync clean && docker-sync start helps

There might be a way to auto-detect case b) since we already have monit ( AFAIK some already do that ) and then auto-heal itself.

We neither can avoid nor fix b) in general - it's not even sure it would need reimplemenation in Unison or its actually a OSXFS vs Inode event propagation related issue.

@EugenMayer

This comment has been minimized.

Copy link
Owner

EugenMayer commented Feb 25, 2019

created #646 to maybe follow up for a fix for b)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.