Not syncing both directions suddenly #410
Comments
|
Hi @nurtext, First, thanks for the detailed description Could you please add the output of |
|
I've ran into this intermittently since going to native_osx. There is nothing visible in In my local configs I'm now applying volumes to my containers that use docker-sync volumes like so: api:
volumes:
- /etc/localtime:/etc/localtime:ro # Fix TZ issues w/ docker-sync?
- api-app-sync:/srv/api/app:nocopy
- api-bin-sync:/srv/api/bin:nocopy
- api-src-sync:/srv/api/src:nocopy
- api-tests-sync:/srv/api/tests:nocopy
- api-web-sync:/srv/api/web:nocopy
- api-vendor-sync:/srv/api/vendor:nocopyI don't know how to reproduce the issue so I'll need to see overtime if this "fix" prevents the state from occurring. Update: This doesn't work. |
|
@codycraven so the issue was your time? (does it work now?) since timestamp compares are used to detect if a file has been changed, that could be cause |
|
@EugenMayer thus far I have not encountered sync failing since implementing this earlier today. My PR for my team was merged about an hour ago, so I should know by mid-week next week if this solved it (most of our team is traveling tomorrow and Monday is a holiday so I won't have a good sample until then). |
|
Very interesting, lets see how this turns out @codycraven - keep us posted. We do handle timezones: https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/sync_strategy/native_osx.rb#L86 in the sync container. So if your app container does not do that and your TZ differs, that is a good source for issues. This leaves us with the question, how we could solve this for the docker-sync user - seems like we can only do that by documentation? |
|
Ho, yes, this is very confusing → date && docker exec frontend-sync date && docker-compose run frontend date
Fri May 26 10:52:41 CEST 2017
Thu May 25 20:32:38 CEST 2017
Thu May 25 18:33:07 UTC 2017On MacOS Sierra 10.12.4. |
|
Thats a very common docker issue its basically there always, just not always visible. but since we compare timestamps on 2 different containers, that happens |
|
Hi guys! |
|
That is pretty odd, how does this get fixed by an Mac OS update .. were did you had this hint from? Which OS did you had before, El Capitan? |
|
Yes,10.11.4 |
|
@codycraven are you / your collegues on El Capitan? |
|
@EugenMayer I'm on Sierra. Looking at the native_osx.rb Docker run command is what gave me the idea to sync the time zone. Going a step further (if this doesn't resolve it) a colleague had the idea to sync the containers (VM) and host with NTP. |
|
Same timezone on host and all docker containers, problem still remains: Logfile: Edit: |
|
I discovered something that may help: my team leader (who's running the exact same Docker Compose and Docker Sync config than I do) is impacted by this problem and using @codycraven's workaround to share The only difference I can think of is that my containers and host seem totally desynchronized while his containers seem to have more consistent times: → date && docker exec frontend-sync date && docker-compose run frontend date
Mon 29 May 2017 16:49:06 BST
Mon May 29 16:49:06 BST 2017
Mon May 29 15:49:18 UTC 2017 |
|
@michaelbaudino are you using any "--prefer newer" syntax? |
|
Nope: we use the default. Here is our exact config: https://gist.github.com/michaelbaudino/1c83c253794e5b6bd0cc8fc47c1b4951 |
|
@michaelbaudino what about OS differences, are you running Sierra and he is running El Capitan? Still trying to understand the pattern here |
|
Nope, we are both on Sierra 10.12.4. Actually, we were, because I upgraded to 10.12.5 about 5 minutes ago |
|
@michaelbaudino when you added the TZ fix had you destroyed your old containers/volumes before bringing them back up or just stopped them? |
|
I just ran into this again on my machine. The time is synchronized perfectly so I continued digging. I checked the file I modified in my host within the docker-sync container and was astounded to find that the file within /host_sync/ was the old version that does not match what is on my host. Based on this, it appears that the volume mount into the docker-sync container is somehow breaking. Update |
|
I have exactly the same issue. I updated to the last version of Sierra and tried the hack to share localtime with no luck ... Tell me if I can help you by giving more info. |
|
Ok so what we have right now is that: a) the timezone does matter - but not for everyone. b) Sierra or el Capitan are not significant also - both could have this issue c) there is no pattern in the project configuration or anything too special Bottom line is, that we need to draw a tighter circle around the issue, it's far too broad right now. Suggestions: I) we write a wiki guide on how to debug issues or gather information - this is overdue anyway. So compare host/app any, check /tmp/unison II) we gather information from all people having this issue and all not having this one and see, were we have important differences - I would suggest a Google spreadsheet for this probably Anybody willing to help bootstrapping this? |
|
@EugenMayer Since I'm depending upon a working docker-sync for the next release of my project, I'm willing to help. Tell me what's up next and I'll do my best. |
|
I'm in, I can start work on a debug guide tomorrow (or contribute to it if someone else starts one). |
|
Some more debugging for you: Cleaned up everything: intermediate containers, docker-sync-stack, containers, etc. No change at all. Changed a file on host: no result.
Changed file got recognized: Tried to change a file afterwards: Same behavior and no more changes got recognized. Result of |
|
@nurtext @codycraven great. First - lets put in the guide in place, since that will be the first thing to ensure we are all doing reproductionable steps and are all on the same way - the only way to really find the actual issue pattern. Could you help with #415 @codycraven if you start the guide, i join you adding what i know about the containers, images, supervisord, debugging and things like that - please all chats about this further in #413 - i created a first outline of what we should do and why, feel free to actually be bold in any sense - i just emptied my mind there @michaelbaudino Mr. Spec, anything we could do with spec to formalize testing? I guess we would need to start with the bash thingy and it will take longer then we want to get this thing on the line, right? Created #414 for this - lets see Please try to spread the comments for the topics across the outer issues - the only thing we should do in this tickets is giving analysis results and give people a spot to "i am also effected" - and also advertise debugging steps / new things we want to know to get it fixed. Lets get our hands dirty on this - i see that is a serious issue here and we need to get this done ASAP, but i will need assistance to get it into a timeframe it suits you. |
|
Some more debugging according according to the guide in the wiki… Host Sync working: > diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/host_sync/$DEBUG_DOCKER_FILE")
> App Sync not working: > diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/app_sync/$DEBUG_DOCKER_FILE")
Files Resources/views/root.html.twig and /dev/fd/11 differLog is empty: > docker exec "$DEBUG_DOCKER_SYNC" tail -n70 /tmp/unison.log
> Missing directory: > docker container inspect --format '{{(index .Mounts 1).Source}}' "$DEBUG_DOCKER_SYNC"
/var/lib/docker/volumes/api-app-sync/_data
> docker exec "$DEBUG_DOCKER_SYNC" ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory
> ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directoryWill now try to reset d4m… |
|
No log means no unison is actually starting @nurtext, please see the supervisor logs for the reasons that this does not actually work or try to restart unison:
|
|
And I think we should have in mind, what projects we are doing. Because I'm working on symfony project, and I only sometimes have conflicts and had this no sync problem only maybe twice in 5 months. But frontend team working with same config, having this problem few times a week. They using Angular 1/2. |
EugenMayer
changed the title
|
Is this fixed? A few devs are starting to see docker-sync just stop. logs dont show any issue its just not picking up any changes. clean and start didnt seem to resolve then about 5min later it started working again |
|
@EugenMayer Your gist (https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2) seems not to work any more. I get "fatal: not a git repository (or any of the parent directories): .git". Is there an updates version? Thank you! |
|
Our team gave up. We went to bare metal dev env. Some have VMware. |
|
If you use the native delegated volumes, docker-sync isn't needed anymore as far as I'm concerned. I've had solid performance on Magento 2 (TONS of folders/files) with no delay or halts in syncing. |
|
closing this, use d4m edge to get it very robust, we have a about one sync stop per week at max |
|
@ddanier this is no longer needed with d4m edge |
|
I just started using docker-sync with latest versions of everything. Initial sync works but then nothing gets sync. Even with explicitly running When I tried to restart everything as suggested here. |
|
I would suggest opening a ticket, paste your configuration there and continue from there |
|
Maybe just to sum this up. Solution for this would be to either put docker4mac edge or use rsync as sync strategy? |
|
d4m will do it (in the meantime) Or to just understand how to configure your excludes and have a proper project layout mitigating the issues of d4m. Reduce the amount of files you
to only the parts you edit. |
|
Maybe I was to stupid and didn't install everything correctly, but I think I followed the installation instructions correctly. However, docker-sync never worked for me. I mount folder X from my Mac to docker-sync, and I'm running docker-sync with docker-machine. If I changed a file in folder X on my Mac, I would see it in the However, if I opened a shell in the docker container running docker-sync, and manually made a change in the After a long search it turns out that this particular behavior is related to a WONT-FIX bug in VirtualBox (see here and here). Basically, a change to a file that was initiated on my Mac does not result in a filesystem event within the container, and is therefore never picked up by Unison. Unison's documentation even explicitly states this (see documentation: "In particular, the Unix implementation does not compare the actual contents of files to their previous contents, but simply looks at each file's inode number and modtime; if neither of these have changed, then it concludes that the file has not been changed.") So, I'm now running a separate program inside the docker-sync container that regularly polls my I initially installed inotify-proxy to do this, but I didn't manage to get it to work properly, so I coded up my own version in PHP. It's plenty fast and seems to work as intended. Hope this helps someone here. If this whole story above is old news and there's existing code in docker-sync to deal with this, can you point me to it, and either update the documentation to point it out clearly, or make it the default behavior? As I interpret the Unison documentation ("considered unchanged if inode is the same") and the VirtualBox documentation ("we don't pass on inode changes") I don't think the existing setup could have ever worked for Docker running on VirtualBox. |
|
@infostreams this and more reasons are, why you should use OSXFS, thus d4m and not vbox/fusion, since the FS driver of OSXFS is better in those terms and fires all the events. There is no doubt that docker4mac and OSXFS has issues too, but i consider it the better alternative. Thanks for sharing your story in that detailed fashion though, very appreciated! |
|
@EugenMayer I don't think the documentation mentions anywhere that this, out of the box, doesn't work with docker-machine. If I had known that I could have resolved my issues by switching to docker4mac, I would have done that. As it is I spent a day or so on this problem. Can you update the documentation to say something along these lines? It would have saved me (and no doubt others) a lot of frustration. |
|
@infostreams checking the docs, i really have no topic about d4m vs docker-machine dedicated to this topic beside the tests under https://docker-sync.readthedocs.io/en/latest/miscellaneous/performance.html But we had this #346 ..so exactly your finding so indeed we could have saved some time for you. But that is how docs are, they are never complete. We have a topic about how to pick the right strategy at https://docker-sync.readthedocs.io/en/latest/advanced/sync-strategies.html and that currently suggest OSXFS eventhough it does not explain all the details, including those details - but following that guide would have saved you time too :) Anyway, we should probably add a "docker-machine vs d4m/d4w" topic - are you willing to do a PR for that? That would be awesome Update: it was there: https://docker-sync.readthedocs.io/en/latest/advanced/sync-strategies.html
under cons |
|
Hi, thanks for pointing out that reference. However, I completely missed that single sentence, and I wouldn't be surprised if many others missed it as well. I will open a pull request to add a single sentence to the installation instructions for Mac, because only saying "With native_osx we no longer have any host dependencies" is slightly misleading since it only works on one of the two ways to run Docker on Mac. Thanks for your great work! |
|
Hey guys, I have run into this issue recently. I've read the thread and it mentions it shouldn't happen with d4m edge anymore. But here we are, with d4m 2.2.3.0 edge and it still happens. Restart of my Mac fixed it. I think I didn't try restarting Docker itself though, could maybe help as well. |
|
@ostrolucky this is known and edge does not fix it ( or anything ). Either FSevents get stuck inside macOS ( macOS restart needs restart ) or inotify events inside the kernel of d4m ( docker restart might help ) and sometimes it's unison inside the sync container getting overheated by the amount of inotify events ( huge file change chunks ) which can be fixed by either restarting unison and sometimes in addition removing the lockfile of unison To be frank, this is unfixable by docker-sync - this is broken conceptually by macOS ( horrible FSevents implementation ), leading to a half baked OSXFS implementation on top of it, and in the end unison, which simply cannot handle fast changes of 20k files like an |
Good, this is just to make that transparent, because reading this thread it implies this was fixed in d4m edge. So good to make that clear that it indeed doesn't fix it.
Do you think mutagen can? If it does, perhaps we should write a strategy for it in docker-sync. Because TBH it's annoying having to exclude such folders with dependencies. Although we are a bit OT now, but there is no community forum or anything like that for docker-sync AFAIK. |
|
@ostrolucky there is no community forum and yes, the latter might be fixable by mutagen, but nobody really tried - see #603 for a issue about implementing mutagen. Feel free to join there and maybe even contribute with a PR or try to bring mutagen to those edge cases. I will close this thread with repeating the answer above so people now the "essence" - got your point, good idea |
|
d4m does not fix it - there are several layers of issues:
To be frank, this is unfixable by docker-sync - this is broken conceptually by macOS ( horrible FSevents implementation ), leading to a half baked OSXFS implementation on top of it, and in the end unison, which simply cannot handle fast changes of 20k files like an So there are cases where sync stops, many of them fall in the one mentioned above - and we can only try to mitigate those |
and others
Error/Feature Requestion/Docs
All docker container are up, but not syncing between /host_sync and /app_sync.
/app_sync to /host_sync is working.
Docker Driver
d4m
Sync strategy
native_osx
docker-compose.yml
docker-compose-dev.yml
docker-sync.yml
docker psoutputdocker-sync listoutputapi-app-sync On address :
api-bin-sync On address :
api-src-sync On address :
api-tests-sync On address :
api-web-sync On address :
api-vendor-sync On address :
OS
macOS 10.11.6
The text was updated successfully, but these errors were encountered: