Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.
Sign upNot syncing both directions suddenly #410
Comments
This comment has been minimized.
This comment has been minimized.
Hi @nurtext, First, thanks for the detailed description Could you please add the output of |
This comment has been minimized.
This comment has been minimized.
I've ran into this intermittently since going to native_osx. There is nothing visible in In my local configs I'm now applying volumes to my containers that use docker-sync volumes like so: api:
volumes:
- /etc/localtime:/etc/localtime:ro # Fix TZ issues w/ docker-sync?
- api-app-sync:/srv/api/app:nocopy
- api-bin-sync:/srv/api/bin:nocopy
- api-src-sync:/srv/api/src:nocopy
- api-tests-sync:/srv/api/tests:nocopy
- api-web-sync:/srv/api/web:nocopy
- api-vendor-sync:/srv/api/vendor:nocopy I don't know how to reproduce the issue so I'll need to see overtime if this "fix" prevents the state from occurring. Update: This doesn't work. |
This comment has been minimized.
This comment has been minimized.
@codycraven so the issue was your time? (does it work now?) since timestamp compares are used to detect if a file has been changed, that could be cause |
This comment has been minimized.
This comment has been minimized.
@EugenMayer thus far I have not encountered sync failing since implementing this earlier today. My PR for my team was merged about an hour ago, so I should know by mid-week next week if this solved it (most of our team is traveling tomorrow and Monday is a holiday so I won't have a good sample until then). |
This comment has been minimized.
This comment has been minimized.
Very interesting, lets see how this turns out @codycraven - keep us posted. We do handle timezones: https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/sync_strategy/native_osx.rb#L86 in the sync container. So if your app container does not do that and your TZ differs, that is a good source for issues. This leaves us with the question, how we could solve this for the docker-sync user - seems like we can only do that by documentation? |
This comment has been minimized.
This comment has been minimized.
Ho, yes, this is very confusing → date && docker exec frontend-sync date && docker-compose run frontend date
Fri May 26 10:52:41 CEST 2017
Thu May 25 20:32:38 CEST 2017
Thu May 25 18:33:07 UTC 2017 On MacOS Sierra 10.12.4. |
This comment has been minimized.
This comment has been minimized.
Thats a very common docker issue its basically there always, just not always visible. but since we compare timestamps on 2 different containers, that happens |
This comment has been minimized.
This comment has been minimized.
azamat163
commented
May 26, 2017
•
Hi guys! |
This comment has been minimized.
This comment has been minimized.
That is pretty odd, how does this get fixed by an Mac OS update .. were did you had this hint from? Which OS did you had before, El Capitan? |
This comment has been minimized.
This comment has been minimized.
azamat163
commented
May 26, 2017
•
Yes,10.11.4 |
This comment has been minimized.
This comment has been minimized.
@codycraven are you / your collegues on El Capitan? |
This comment has been minimized.
This comment has been minimized.
@EugenMayer I'm on Sierra. Looking at the native_osx.rb Docker run command is what gave me the idea to sync the time zone. Going a step further (if this doesn't resolve it) a colleague had the idea to sync the containers (VM) and host with NTP. |
This comment has been minimized.
This comment has been minimized.
Same timezone on host and all docker containers, problem still remains:
Logfile:
Edit: |
This comment has been minimized.
This comment has been minimized.
I discovered something that may help: my team leader (who's running the exact same Docker Compose and Docker Sync config than I do) is impacted by this problem and using @codycraven's workaround to share The only difference I can think of is that my containers and host seem totally desynchronized while his containers seem to have more consistent times: → date && docker exec frontend-sync date && docker-compose run frontend date
Mon 29 May 2017 16:49:06 BST
Mon May 29 16:49:06 BST 2017
Mon May 29 15:49:18 UTC 2017 |
This comment has been minimized.
This comment has been minimized.
@michaelbaudino are you using any "--prefer newer" syntax? |
This comment has been minimized.
This comment has been minimized.
Nope: we use the default. Here is our exact config: https://gist.github.com/michaelbaudino/1c83c253794e5b6bd0cc8fc47c1b4951 |
This comment has been minimized.
This comment has been minimized.
@michaelbaudino what about OS differences, are you running Sierra and he is running El Capitan? Still trying to understand the pattern here |
This comment has been minimized.
This comment has been minimized.
Nope, we are both on Sierra 10.12.4. Actually, we were, because I upgraded to 10.12.5 about 5 minutes ago |
This comment has been minimized.
This comment has been minimized.
@michaelbaudino when you added the TZ fix had you destroyed your old containers/volumes before bringing them back up or just stopped them? |
This comment has been minimized.
This comment has been minimized.
I just ran into this again on my machine. The time is synchronized perfectly so I continued digging. I checked the file I modified in my host within the docker-sync container and was astounded to find that the file within /host_sync/ was the old version that does not match what is on my host. Based on this, it appears that the volume mount into the docker-sync container is somehow breaking. Update |
This comment has been minimized.
This comment has been minimized.
masev
commented
May 30, 2017
I have exactly the same issue. I updated to the last version of Sierra and tried the hack to share localtime with no luck ... Tell me if I can help you by giving more info. |
This comment has been minimized.
This comment has been minimized.
Ok so what we have right now is that: a) the timezone does matter - but not for everyone. b) Sierra or el Capitan are not significant also - both could have this issue c) there is no pattern in the project configuration or anything too special Bottom line is, that we need to draw a tighter circle around the issue, it's far too broad right now. Suggestions: I) we write a wiki guide on how to debug issues or gather information - this is overdue anyway. So compare host/app any, check /tmp/unison II) we gather information from all people having this issue and all not having this one and see, were we have important differences - I would suggest a Google spreadsheet for this probably Anybody willing to help bootstrapping this? |
This comment has been minimized.
This comment has been minimized.
@EugenMayer Since I'm depending upon a working docker-sync for the next release of my project, I'm willing to help. Tell me what's up next and I'll do my best. |
This comment has been minimized.
This comment has been minimized.
I'm in, I can start work on a debug guide tomorrow (or contribute to it if someone else starts one). |
This comment has been minimized.
This comment has been minimized.
Some more debugging for you: Cleaned up everything: intermediate containers, docker-sync-stack, containers, etc. No change at all. Changed a file on host: no result.
Changed file got recognized:
Tried to change a file afterwards: Same behavior and no more changes got recognized. Result of
|
EugenMayer
referenced this issue
May 31, 2017
Closed
Quick Test: can we ensure sync works using rspec #414
This comment has been minimized.
This comment has been minimized.
@nurtext @codycraven great. First - lets put in the guide in place, since that will be the first thing to ensure we are all doing reproductionable steps and are all on the same way - the only way to really find the actual issue pattern. Could you help with #415 @codycraven if you start the guide, i join you adding what i know about the containers, images, supervisord, debugging and things like that - please all chats about this further in #413 - i created a first outline of what we should do and why, feel free to actually be bold in any sense - i just emptied my mind there @michaelbaudino Mr. Spec, anything we could do with spec to formalize testing? I guess we would need to start with the bash thingy and it will take longer then we want to get this thing on the line, right? Created #414 for this - lets see Please try to spread the comments for the topics across the outer issues - the only thing we should do in this tickets is giving analysis results and give people a spot to "i am also effected" - and also advertise debugging steps / new things we want to know to get it fixed. Lets get our hands dirty on this - i see that is a serious issue here and we need to get this done ASAP, but i will need assistance to get it into a timeframe it suits you. |
This comment has been minimized.
This comment has been minimized.
Some more debugging according according to the guide in the wiki… Host Sync working: > diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/host_sync/$DEBUG_DOCKER_FILE")
> App Sync not working: > diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/app_sync/$DEBUG_DOCKER_FILE")
Files Resources/views/root.html.twig and /dev/fd/11 differ Log is empty: > docker exec "$DEBUG_DOCKER_SYNC" tail -n70 /tmp/unison.log
> Missing directory: > docker container inspect --format '{{(index .Mounts 1).Source}}' "$DEBUG_DOCKER_SYNC"
/var/lib/docker/volumes/api-app-sync/_data
> docker exec "$DEBUG_DOCKER_SYNC" ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory
> ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory Will now try to reset d4m… |
This comment has been minimized.
This comment has been minimized.
No log means no unison is actually starting @nurtext, please see the supervisor logs for the reasons that this does not actually work or try to restart unison:
|
EugenMayer
closed this
Jun 1, 2017
This comment has been minimized.
This comment has been minimized.
mmrko
commented
Nov 9, 2017
Meanwhile, the rsync strategy works beautifully. Kudos to @EugenMayer & co. for their hard (and gratuitous) work! :) |
This comment has been minimized.
This comment has been minimized.
@mmrko yes, rsync is not affected by this, thats why having multiple strategies is one of our goals - pick the axe for your case. The problem with rsync is on the other side, CPU load, since it uses fswatcher to watch for file changes - and due to macOS being horrible in FS-Events, even that is just bad as it is. So its pretty bullet proof, but not entirely. But more or less, all strategies have their little edges, thats assured. But running drush for streight up 2 minutes for a I would love to remove docker-sync from this world, if it would not be needed .e.g d4m fixed it. That will be a great day for us all - no overhead, no glitches, no extra configuration - just docker. Well d4m is not just docker, its docker on a non linux kernel...and thats where we all start. |
Nov 9, 2017
Repository owner
locked and limited conversation to collaborators
This comment has been minimized.
This comment has been minimized.
Closed the thread for collaborators only so this thing does not get offtopic even more - way to important issue with way to valuable information, no need for all the other. Will reopen the issue in 1 week when things calmed down. If you feel in the need to express yourself about this, open a new issue in the queue so this one keeps the focus |
Nov 10, 2017
Repository owner
unlocked this conversation
EugenMayer
referenced this issue
Nov 12, 2017
Closed
High CPU usage and syncing stops with native OSX when files are being created and removed #497
This comment has been minimized.
This comment has been minimized.
Anybody having this issue, please see #497 and run the reproduction script, post your details. Also ensure you test it 2 times, one without my d4m fs fix and one with: https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2 Does the result change for you? This could really help use nailing down the issue. Why i am assuming that my fix helps is, that neither i have this issue often ( about 1 time in 3 months ) neither anybody of my coworkers - we are doing heavy development though, it means, one project most probably having 0.5 million files, 5 docker-sync endpoints starting at the same time and a stack which consume 8GB Ram + quiet some CPU. Still it hardly ever happens for us ( i think for most of us, it never happened at all). We have other projects, bigger Java projects, bigger ng4 stacks and so on, still it hardly ever happens for us. So there must be something we do that this does not happen. One thing i can tell is:
|
This comment has been minimized.
This comment has been minimized.
i am currently thinking about closing this issue and open a new one, extracting all the facts / informations we had here, also how to test owns system to provide informations so we can see pattern. I would link this issues as the source of deeper discussions. Any concerns about that? Main reason is, to consolidate the results and make it easier to understand how this issues shows up, how you can for now work around and what the reasons for this are, without reading so many posts. |
This comment has been minimized.
This comment has been minimized.
zedtux
commented
Nov 19, 2017
@EugenMayer I agree with you, it should also allow new comers to quicker understand the issue and where we stand. |
This comment has been minimized.
This comment has been minimized.
BunnyHolder
commented
Nov 28, 2017
And I think we should have in mind, what projects we are doing. Because I'm working on symfony project, and I only sometimes have conflicts and had this no sync problem only maybe twice in 5 months. But frontend team working with same config, having this problem few times a week. They using Angular 1/2.
|
EugenMayer
referenced this issue
Jan 10, 2018
Closed
Troubleshooting docker sync on high sierra #515
EugenMayer
changed the title
0.4.6 not syncing both directions on macOS 10.11.6
Not syncing both directions suddenly
Jan 12, 2018
Jan 12, 2018
This was referenced
This comment has been minimized.
This comment has been minimized.
b1alpha
commented
May 14, 2018
•
Is this fixed? A few devs are starting to see docker-sync just stop. logs dont show any issue its just not picking up any changes. clean and start didnt seem to resolve then about 5min later it started working again |
This comment has been minimized.
This comment has been minimized.
ddanier
commented
Jul 7, 2018
@EugenMayer Your gist (https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2) seems not to work any more. I get "fatal: not a git repository (or any of the parent directories): .git". Is there an updates version? Thank you! |
This comment has been minimized.
This comment has been minimized.
BunnyHolder
commented
Jul 9, 2018
•
Our team gave up. We went to bare metal dev env. Some have VMware. |
This comment has been minimized.
This comment has been minimized.
markshust
commented
Jul 9, 2018
If you use the native delegated volumes, docker-sync isn't needed anymore as far as I'm concerned. I've had solid performance on Magento 2 (TONS of folders/files) with no delay or halts in syncing. |
talya
referenced this issue
Jul 10, 2018
Closed
unison sync suddenly stops when app-sync writes many files #585
This comment has been minimized.
This comment has been minimized.
closing this, use d4m edge to get it very robust, we have a about one sync stop per week at max |
This comment has been minimized.
This comment has been minimized.
@ddanier this is no longer needed with d4m edge |
EugenMayer
closed this
Jul 22, 2018
This comment has been minimized.
This comment has been minimized.
ohcibi
commented
Nov 30, 2018
I just started using docker-sync with latest versions of everything. Initial sync works but then nothing gets sync. Even with explicitly running
When I tried to restart everything as suggested here. |
This comment has been minimized.
This comment has been minimized.
I would suggest opening a ticket, paste your configuration there and continue from there |
kamil-jakubowski
referenced this issue
Jan 28, 2019
Closed
Does not sync after few stop ands starts #633
This comment has been minimized.
This comment has been minimized.
adriadrop
commented
Feb 12, 2019
Maybe just to sum this up. Solution for this would be to either put docker4mac edge or use rsync as sync strategy? |
This comment has been minimized.
This comment has been minimized.
d4m will do it (in the meantime) Or to just understand how to configure your excludes and have a proper project layout mitigating the issues of d4m. Reduce the amount of files you
to only the parts you edit. |
nurtext commentedMay 24, 2017
Error/Feature Requestion/Docs
All docker container are up, but not syncing between /host_sync and /app_sync.
/app_sync to /host_sync is working.
Docker Driver
d4m
Sync strategy
native_osx
docker-compose.yml
docker-compose-dev.yml
docker-sync.yml
docker ps
outputdocker-sync list
outputapi-app-sync On address :
api-bin-sync On address :
api-src-sync On address :
api-tests-sync On address :
api-web-sync On address :
api-vendor-sync On address :
OS
macOS 10.11.6