Synchronization stops. #517
Comments
|
please try to reproduce this with https://github.com/EugenMayer/docker-sync-boilerplate in the folder 'default' |
|
Hello Eugen, When I change the file index.html, I do not see the change in the terminal. Sometimes I see a spinner in the terminal, but no change on the screen. |
|
I restarted my Mac and tried the default example again - this one is working now. Also my other configuration seems to work now. I do not know what happened here before my restart... |
|
I am also seeing this issue, it works for a short time and then can stop randomly. |
|
are you using docker4mac edge? |
|
Yeah ( |
|
that should be your problem use stable |
|
@EugenMayer I'm guessing the problem lies with 17.12.0-ce, which is currently the new stable. |
|
i cannot work on those issues because all ppl give as information is 'it does' not work, fixing it by doing either nothing (just waiting), doing upgrades, some do downgrades, some do restarts and nobody can make any sense about all this. there is litterellay no information or debuging in those reports. It's more a 'its broken, how to fix it/can somebody fix it for me' the short answer to this is: no Reasons vary from:
For example, i am running the latest docker4mac edge here:
no problems in either direction. Doing the same with the current stable, same result. Guys, take you time and finally start debugging that. You disagree on basically any matter, be it the d4m version, down or upgrading fixes it, be it a FS issue or a project issue. Get yourself a shell, use https://github.com/EugenMayer/docker-sync/wiki/8.-Strategies#native_osx-osx and We did not create this for the matter of too much time - rather you getting involved and being able to help yourself when you need it most with the timeframe you are in urge or not. But just opening a new issue after issue with "it does not work / any longer" and thats it - will not do it. |
|
@EugenMayer The debugging guide did not help me sadly. And I don't think we disagree on anything, just confusion. The edge version became stable only very recently. That is why we have this edge/stable confusion. I'm trying more stuff, but I've been debugging this for too long this night with no results. I'm now downgrading to the previous stable to confirm at least that it is the latest release. As for replicating. After a docker-sync clean, and running it again, it works for me. But after restarting the stack, it no longer works. I do agree that it is not your responsibility. This is OSS, and you've been very helpful so far. Thanks for that |
|
Yesterday, there was Docker for Mac update which I installed So this looks like the beginning of the problems as in one line it says "success" just after "ERROR". I've tried to give it a try and here are the results: I'm not sure if problem with port 5000 could be a source of the problem or maybe it is the result of half failed/broken upgrade installation? Similar errors when stopping it: I've decided to force it this time and run again full installation command After this everything seemed to be fine, maybe even a little bit faster than usual, but only for a few minutes. The changes from the host have stopped to be populated into the container. It was confusing as I started to use I've stopped and restarted it and everything was fine again but just for a while. After next restart, I wanted to measure the time but everything was fine and Suddenly it stopped to sync and I realised that it was not time-related but it stopped to work after I started to use application and opposite direction sync has happened I've followed your advice and wanted to provide some debugging details using directions from https://github.com/EugenMayer/docker-sync/wiki/8.1.-native_osx-sync-strategy-debugging-guide Unfortunately I've got no output from the this debugging command which is the reason why there is no sync triggered. Basically the changes on the host files become undetected Some time ago I've asked to provide instructions how to downgrade docker-sync to the previous version. I'm not familiar with this whole gem installation and as I guess it works with some remote URL to get recent version there should be some URLs provided with older one. I can't find anything on the wiki how to install from the downloaded zip files as well. Can someone provide such details and update the Wiki, please? |
|
I've just downgraded DFM to |
|
@Krzysiaczek-at-theFoundry finally someone doing the whole round-robbin with a conclusion. So we have a bug in the latest stable, but after some time. downgrading to One questions @Krzysiaczek-at-theFoundry are you using APFS? |
|
Nope, I'm still on (low) Sierra. |
|
same here, so beside your findings, AFPS could be still a variant means anybody with AFPS and this issue who can verify that that downgrade helps him too? |
|
Also having the same issue here after updating to |
|
Downgraded to Synchronisation working again with docker-sync 0.5.2 |
|
@phillipsnick great news, APFS / high sierra? |
|
I've also downgraded to 17.09.0-ce-mac33 and I've used it all day. It is definitely more stable, but I am able to break it by mounting the same docker-sync volume by 2 containers :) I'll skip the 2nd container tomorrow and see if it keeps working. |
|
I'm running 10.13 with APFS and Docker 17.12.0 and docker-sync 0.5.2. I ran some tests and I'm finding the initial sync takes almost twice as long as it used to for the same codebase (~11 minutes versus ~6 minutes), and fs events are not being captured at all. Even restarting after an initial sync takes just over 2 minutes, even though there's no changes to sync. I don't know how long it used to take, but it was quick enough that I never bothered measuring it. @EugenMayer thanks for your work on this to date, and hopefully this is a helpful datapoint. |
|
@phillipsnick care to share how you successfully downgraded docker, and save the rest of us an hour? |
|
Alright I took the 20 minutes and figured it out myself. There are a couple of quick-ish ways. The docker release notes now include links to older versions (after much arguing on github): This 3rd party site is listing the releases: If you know the build number you want, you can get the dmg yourself. For example, 21090 is for
So for 21090 it is: If you don't know the build number you want, somebody posted a handy little bash script that programmatically tries all build numbers to see if the build exists and lets you know: Change the sequence numbers on line 2 to define your range. You can set the first number to your currently broken build number (which you can find by using the menubar widget > about docker) and you can probably leave the second number alone unless you want a really old build. I hope this helps others. |
|
Or if you use homebrew and you want the easy way: The second command just installs the version of docker before the homebrew recipe was updated to include the latest, so you will still get build |
|
@EugenMayer El Capitan 10.11.1 (I know old ha), won't be APFS due to not updating in so long. docker-sync.yml @brianmorton nice find with the 3rd party site. Any idea why theres so many different builds per version? |
|
@phillipsnick they most likely run a build for each merge to master. |
|
Had the same issue after upgrading: syncing wasn't working. Worked after downgrading. On Sierra. |
|
Okay, I'll do my best to provide some extra context. I've been having this issue when changing between git branches in other words when lots of files change at the same time. Figuring out certain directories to add to
Following https://github.com/EugenMayer/docker-sync/wiki/8.1.-native_osx-sync-strategy-debugging-guide I discovered the issue only affected one of the syncs for some reason. So I followed the steps provided and discovered this in the /app_sync. Logs: Here's the output of Then I:
Latest output of That's about the best I have for now. Cheers! 🍻 |
|
Thanks for your reports @derschatta. @SilberMa It’s good to know that there’s an alternative that is performant enough for some people, and I think that is well-understood now. 👍 This thread is for reporting on how docker-sync is working with recent versions of d4m. Please keep the comments relevant, per @EugenMayer’s requests. |
This comment has been minimized.
This comment has been minimized.
|
I tried running on :cached flag but there's still a huge difference in speed compared to using docker-sync (cached = 2s loading time vs. docker-sync: 0.5s loading time). It might be ok to work with it but actually I rather see docker-sync working as expected. For now mac66 works fine with my docker-sync setup 👍 |
|
My Specs: Sierra, raw, mac66 edge Important, eventhough i run with raw image on Sierra, there are reports that this stuck in the start process. Maybe just upgrade to HS then I am running mac66[edge] for 2 days now on a fairly big project and i can second that several things are changing: TLTR: docker-sync seems to work very stable with But there are a lot more changes:
Bottom line is, that mac66 edge can be really a good release for a switch. It is fixing this very issue for stopping sync but also includes significant improvments overall esp. in terms of performance @derschatta @SilberMa @watermanio @MatthiasKuehneEllerhold please stop discussing any issue not related to docker-sync |
|
I've been testing mac66 edge for nearly a week now. It's not as bad as anything in between mac42 and mac66 edge, and indeed synchronization has only stopped once for me in the last week. However, regularly the Docker process starts hogging CPU. We're using docker for huge projects (Magento, and even worse Magento 2), and switching a branch or recompiling in Magento 2 can cause the Docker process to suddenly start using around 250% CPU (quad core i7 with hyperthreading and whatever that feature is called that overclocks one or two cores). I have noticed that with mac42, too, but much more rarely than with mac66 edge. So, once the container CPU usage stayed high (I used Conclusion for our setup: While I could use mac42 for days at end without having to completely restart Docker, I have to restart Docker at least once but usually several times a day with mac66 edge. So mac42 is still a lot better for me. |
|
@ktruehl thank you for the elaborated report. In general, branch switches, with our without docker-sync will cause troubles with mac42+ when you do not take care. if, the next time, this happens again, try to connect to your xhyve VM: use top to identify which process is running wild. AFAIU its not unison of docker-sync since you restarted it, right. so it might be something in the xhyve VM ( which gets restarted when you restart docker ). Probably Its an alpine vm but we cannot modify anything there is a docker-engine up and running (obviously) - but it is nicely transparent to us. Maybe you can find at leat the service / binary this is cause by and we either can report this to d4m or we could try to gracefully restart it with a HUP or similar. |
|
@EugenMayer thanks for testing mac66 for yourself. Regarding discussing not related topics: I never discussed anything else than docker-sync on mac66. I emphasized the performance difference to make clear why I use docker-sync and want to find a way to use it with anything newer than mac42. Seems we found it :). |
|
If Docker mac66 continues to generate I've spent a lot of hours not understanding why there are so many success stories when I either got docker-sync missing a lot of files or randomly exiting containers with "Segmentation fault", with As you can see in this comment: docker/for-mac#2625 (comment) they had disabled Now I have working docker-sync so fingers crossed it'll be more stable than mac42 on our heavy Symfony project! P.S. @EugenMayer, that might be misleading:
Maybe you'd want to edit it a bit, as I expect most people to prefer reading your comments (at least I do). |
|
@grigoryosifov thank you for the elaboration, really nice work gets done here nowdays! Interestingly i had this raw-image issue only on 1 of 4 of my devices - but i had it. Changed my comment above, better safe then sorry, right :) @derschatta i maybe just got you wrong - excuse myself. Probably lost in the ether of comments. Sorry! Mac67 is out with some promising additional fixes:
i have no real field test, so people do not just upgrade production instances. Test it first, add some comments. I expect 67 to be just a little polished 66, so no real issues added, rather some fixed |
|
Maybe the CPU problems don't was fixed... (Docker-EDGE-18.05.0-ce-mac67) |
|
Running successfully on 10.13.5 (High Sierra) using .raw (default) with D4M edge version mac67 since 6/8/2018, syncing a large (40k synced files) Rails app. 👍🎉 It works at least as well as mac42 did for me as an end user. I had not benchmarked mac42 CPU usage, so I can't compare. Docker is averaging maybe 25% CPU on 1 core with containers largely idle (according to ctop), but CPU usage does not change when I'm not using docker-sync. This thread is about sync performance though, not docker CPU usage (likely) unrelated to docker-sync. I will post if I have any problems later. |
This comment has been minimized.
This comment has been minimized.
For compatibility with docker-sync EugenMayer/docker-sync#517 (comment)
|
After this never working for me before, I finally have it working consistently with a large PHP + React app (~43k files), but only with the following setup.
The latency of our app running is almost equal to native with this setup. We also run our unit tests in the container, and they run ~3x faster than without docker sync. |
|
I can confirm it works well, at least for a few hours, after upgrading to |
|
@charrondev you just have been a little quicker and those are great news! : Indeed we did the same transition across the company the last 3 weeks.
The overall result was fairly impressive, the performance is a lot better overall with d4m, still docker-sync is a must and helps significant. We run docker-sync all day, yet we had only to restart it once over the last whole week - that is a very good result for now. Its a huge project, over 50k files. The reason we mirgrate to High Sierra were, that mac67 edge does not work in RAW with HFS, it can crash and stuck a very ugly way, destroy the whole d4m installation and does no longer allow you to start it now matter what you do. this happens with 1 of 2 boxes with Sierra, not with every box, but the impact was significant enough for us to switch. Especially our High Sierra pilot was very promising since the overall performance with APFS + raw is a huge boost |
|
Things are doing good on my end working on some of the same projects as @nilbus, but I'm on the Sierra + qcow + D4M 67 + docker-sync 0.5.7 combo. I can also confirm that it works after sleep / resume for 2 / 3 days in a row (which used to brake syncing for me in some old version > mac42) |
|
Today on day 10, sync started failing. It would sync the files (after a short delay) each time I started docker sync, but subsequent changes were not detected and synced. See my setup above. Update: I found that changes were not being synced by Docker to the volume mount inside the sync container. Restarting Docker resolved the problem this time… I'm not sure what the problem was. Edit: I was mistakenly looking in |
|
I am closing this issue since I think we have a winner: we are working with d4m edge for weeks now, with a sync stop per week. also other reports here were pretty positive. I will open up a new issue to implement a notice that people should upgrade to the latest edge when the run d4m, so people do not get into trouble that often, without knowing that. |
|
What version is the |
|
edge 67 .. but you just would have to read 2 posts above. |
|
@EugenMayer you should consider updating the |
|
@zedtux not the readme, since it kind of content-less but the wiki for sure. Mostly people do not read the docs anyhow, so my first action is rather a dependency test when starting docker-sync and/or a confirmation that "i know that.." .. this helps more and people have an easier time finding it. Feel free to help with PRs if you find any free time right now, i am currently too busy, sorry |
|
Seems fair to me. |
|
Anyone still has any problems with this? For a while it worked for me but recently this issue came back. Can't tell yet with which version. But I tried an older version, I believe edge 18.05.0-ce-mac67 and the newest stable 2.0.0.3. With both synchronisation stops a some point. CPU load goes up and I can only get it back working by shutting down all containers, stopping docker-sync and start both up again. Then it works for a while and then stops again. Quite annoying. I've upgraded Mac OS to Mojave recently but if I remember correctly the issue was already there before. |
|
Everybody has this issue and it will never stop. Especially when using npm based projects where during watch/rebuilds a lot of files are created/removed, this one gets problematic. In most cases, this has nothing to do with docker-sync at all, but with OSXFS getting stucked in the FS event queue, which then also stops events for unison in our docker image ( linux, so inode events ) and thus breaks syncing. There is no fix to that - sometimes restarting docker sync helps, this means, that is an actual unison "too many events" issue, which can also happen for unison. So what we have here: There might be a way to auto-detect case b) since we already have monit ( AFAIK some already do that ) and then auto-heal itself. We neither can avoid nor fix b) in general - it's not even sure it would need reimplemenation in Unison or its actually a OSXFS vs Inode event propagation related issue. |
|
created #646 to maybe follow up for a fix for b) |
|
One more possible reason: Example of config: Note: UID you have and GID you have to run next |
and others
Hello,
I use a pretty simple docker sync setup with these three files:
docker-compose-dev.yml
docker-compose.yml
docker-sync.yml
Now it happens that the sync does start and I can visit the website on my local mashine. However when I change something nothing get's synced. When I then stop the synchronization and start the sync again with docker-sync-stack start, the latest changes are synced, but why is this not consistently the case and how I can change it?
I use MacOS Sierra 10.12.6 with the Docker version Version 17.09.0-ce-mac35 (19611) and unison sync. Thanks!
The text was updated successfully, but these errors were encountered: