Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not syncing both directions suddenly #410

Closed
nurtext opened this issue May 24, 2017 · 99 comments
Closed

Not syncing both directions suddenly #410

nurtext opened this issue May 24, 2017 · 99 comments

Comments

@nurtext
Copy link

@nurtext nurtext commented May 24, 2017

Error/Feature Requestion/Docs

All docker container are up, but not syncing between /host_sync and /app_sync.
/app_sync to /host_sync is working.

Docker Driver

d4m

Sync strategy

native_osx

docker-compose.yml

version: '2'

services:
    api:
        container_name: api
        image: api
        build: .
        ports:
            - 8888:80
        volumes:
            - ./app:/srv/api/app
            - ./bin:/srv/api/bin
            - ./src:/srv/api/src
            - ./tests:/srv/api/tests
            - ./web:/srv/api/web
            - ./vendor:/srv/api/vendor
        environment:
            DATABASE_HOST: "127.0.0.1"

docker-compose-dev.yml

version: '2'

services:
    api:
        volumes:
            - api-app-sync:/srv/api/app:nocopy
            - api-bin-sync:/srv/api/bin:nocopy
            - api-src-sync:/srv/api/src:nocopy
            - api-tests-sync:/srv/api/tests:nocopy
            - api-web-sync:/srv/api/web:nocopy
            - api-vendor-sync:/srv/api/vendor:nocopy

volumes:
    api-app-sync:
        external: true

    api-bin-sync:
        external: true

    api-src-sync:
        external: true

    api-tests-sync:
        external: true

    api-web-sync:
        external: true

    api-vendor-sync:
        external: true

docker-sync.yml

version: '2'

options:
    verbose: true

syncs:
    api-app-sync:
        src: './app/'

    api-bin-sync:
        src: './bin/'

    api-src-sync:
        src: './src/'

    api-tests-sync:
        src: './tests/'

    api-web-sync:
        src: './web/'

    api-vendor-sync:
        src: './vendor/'

docker ps output

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                            NAMES
72ee4eb5a02b        api                              "docker-php-entryp..."   2 minutes ago       Up 2 minutes        9000/tcp, 0.0.0.0:8888->80/tcp   api
7ec87ccaa84f        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   15 minutes ago      Up 2 minutes        500/tcp                          api-vendor-sync
de22a140b897        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-web-sync
b1d749f9c992        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-tests-sync
62d9a55979a3        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-src-sync
0af351c80ed8        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-bin-sync
45d5f50d0659        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-app-sync

docker-sync list output

      ok  Found configuration at /Users/cedric/api/docker-sync.yml

api-app-sync On address :
api-bin-sync On address :
api-src-sync On address :
api-tests-sync On address :
api-web-sync On address :
api-vendor-sync On address :

OS

macOS 10.11.6

@michaelbaudino
Copy link
Contributor

@michaelbaudino michaelbaudino commented May 25, 2017

Hi @nurtext,

First, thanks for the detailed description 👍

Could you please add the output of docker logs api-web-sync ? (or whatever *-sync container you're trying to sync files with)

@codycraven
Copy link
Contributor

@codycraven codycraven commented May 25, 2017

I've ran into this intermittently since going to native_osx. There is nothing visible in docker logs that indicates a problem, however I suspect it stems from timestamps being out of sync between the host and the container.

In my local configs I'm now applying volumes to my containers that use docker-sync volumes like so:

    api:
        volumes:
            - /etc/localtime:/etc/localtime:ro # Fix TZ issues w/ docker-sync?
            - api-app-sync:/srv/api/app:nocopy
            - api-bin-sync:/srv/api/bin:nocopy
            - api-src-sync:/srv/api/src:nocopy
            - api-tests-sync:/srv/api/tests:nocopy
            - api-web-sync:/srv/api/web:nocopy
            - api-vendor-sync:/srv/api/vendor:nocopy

I don't know how to reproduce the issue so I'll need to see overtime if this "fix" prevents the state from occurring.

Update: This doesn't work.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 25, 2017

@codycraven so the issue was your time? (does it work now?)

since timestamp compares are used to detect if a file has been changed, that could be cause

@codycraven
Copy link
Contributor

@codycraven codycraven commented May 25, 2017

@EugenMayer thus far I have not encountered sync failing since implementing this earlier today.

My PR for my team was merged about an hour ago, so I should know by mid-week next week if this solved it (most of our team is traveling tomorrow and Monday is a holiday so I won't have a good sample until then).

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 26, 2017

Very interesting, lets see how this turns out @codycraven - keep us posted. We do handle timezones: https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/sync_strategy/native_osx.rb#L86 in the sync container.

So if your app container does not do that and your TZ differs, that is a good source for issues. This leaves us with the question, how we could solve this for the docker-sync user - seems like we can only do that by documentation?

@michaelbaudino
Copy link
Contributor

@michaelbaudino michaelbaudino commented May 26, 2017

Ho, yes, this is very confusing 😕

→ date && docker exec frontend-sync date && docker-compose run frontend date
Fri May 26 10:52:41 CEST 2017
Thu May 25 20:32:38 CEST 2017
Thu May 25 18:33:07 UTC 2017

On MacOS Sierra 10.12.4.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 26, 2017

Thats a very common docker issue its basically there always, just not always visible. but since we compare timestamps on 2 different containers, that happens

@azamat163
Copy link

@azamat163 azamat163 commented May 26, 2017

Hi guys!
I also encountered such a problem, but solved it with the help of the Mac OS update to 10.12.5. It helped me.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 26, 2017

That is pretty odd, how does this get fixed by an Mac OS update .. were did you had this hint from? Which OS did you had before, El Capitan?

@azamat163
Copy link

@azamat163 azamat163 commented May 26, 2017

Yes,10.11.4

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 26, 2017

@codycraven are you / your collegues on El Capitan?

@codycraven
Copy link
Contributor

@codycraven codycraven commented May 26, 2017

@EugenMayer I'm on Sierra. Looking at the native_osx.rb Docker run command is what gave me the idea to sync the time zone. Going a step further (if this doesn't resolve it) a colleague had the idea to sync the containers (VM) and host with NTP.

@nurtext
Copy link
Author

@nurtext nurtext commented May 29, 2017

Same timezone on host and all docker containers, problem still remains:

> date && docker exec api date && docker exec api-app-sync date
Mon May 29 11:39:34 CEST 2017
Mon May 29 11:39:34 CEST 2017
Mon May 29 11:39:34 CEST 2017

Logfile:

> docker logs api-app-sync
2017-05-29 11:29:05,061 CRIT Supervisor running as root (no user in config file)
2017-05-29 11:29:05,062 WARN Included extra file "/etc/supervisor.conf.d/supervisor.daemon.conf" during parsing
2017-05-29 11:29:05,097 INFO RPC interface 'supervisor' initialized
2017-05-29 11:29:05,097 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2017-05-29 11:29:05,097 INFO supervisord started with pid 1
2017-05-29 11:29:06,100 INFO spawned: 'unison' with pid 27
2017-05-29 11:29:07,271 INFO success: unison entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Edit:
Maybe this is another puzzle piece, but I doubt so: I recently moved from ubuntu:trusty-20170214 to php:7.1-fpm-alpine. Previous setup was working fine with native_osx sync

@michaelbaudino
Copy link
Contributor

@michaelbaudino michaelbaudino commented May 30, 2017

I discovered something that may help: my team leader (who's running the exact same Docker Compose and Docker Sync config than I do) is impacted by this problem and using @codycraven's workaround to share /etc/localtime fixed the issue for him (thanks for that, @codycraven, btw 🎩 ).

The only difference I can think of is that my containers and host seem totally desynchronized while his containers seem to have more consistent times:

→ date && docker exec frontend-sync date && docker-compose run frontend date
Mon 29 May 2017 16:49:06 BST
Mon May 29 16:49:06 BST 2017
Mon May 29 15:49:18 UTC 2017
@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 30, 2017

@michaelbaudino are you using any "--prefer newer" syntax?

@michaelbaudino
Copy link
Contributor

@michaelbaudino michaelbaudino commented May 30, 2017

Nope: we use the default. Here is our exact config: https://gist.github.com/michaelbaudino/1c83c253794e5b6bd0cc8fc47c1b4951

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 30, 2017

@michaelbaudino what about OS differences, are you running Sierra and he is running El Capitan?

Still trying to understand the pattern here

@michaelbaudino
Copy link
Contributor

@michaelbaudino michaelbaudino commented May 30, 2017

Nope, we are both on Sierra 10.12.4.

Actually, we were, because I upgraded to 10.12.5 about 5 minutes ago 😄

@codycraven
Copy link
Contributor

@codycraven codycraven commented May 30, 2017

@michaelbaudino when you added the TZ fix had you destroyed your old containers/volumes before bringing them back up or just stopped them?

@codycraven
Copy link
Contributor

@codycraven codycraven commented May 30, 2017

I just ran into this again on my machine. The time is synchronized perfectly so I continued digging.

I checked the file I modified in my host within the docker-sync container and was astounded to find that the file within /host_sync/ was the old version that does not match what is on my host.

Based on this, it appears that the volume mount into the docker-sync container is somehow breaking.

Update
This appears to be a false failure. I created a copy of my project in a new directory and ran it while debugging something. I failed to properly clean up afterwards as my sync container's bind mount was pointed at my copied directory instead of the one I was working on.

@masev
Copy link

@masev masev commented May 30, 2017

I have exactly the same issue. I updated to the last version of Sierra and tried the hack to share localtime with no luck ...

Tell me if I can help you by giving more info.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 31, 2017

Ok so what we have right now is that:

a) the timezone does matter - but not for everyone.

b) Sierra or el Capitan are not significant also - both could have this issue

c) there is no pattern in the project configuration or anything too special


Bottom line is, that we need to draw a tighter circle around the issue, it's far too broad right now. Suggestions:

I) we write a wiki guide on how to debug issues or gather information - this is overdue anyway. So compare host/app any, check /tmp/unison
. How to enter a snyc container, how to manually restart the inner unison project

II) we gather information from all people having this issue and all not having this one and see, were we have important differences - I would suggest a Google spreadsheet for this probably

Anybody willing to help bootstrapping this?

@nurtext
Copy link
Author

@nurtext nurtext commented May 31, 2017

@EugenMayer Since I'm depending upon a working docker-sync for the next release of my project, I'm willing to help. Tell me what's up next and I'll do my best.

@codycraven
Copy link
Contributor

@codycraven codycraven commented May 31, 2017

I'm in, I can start work on a debug guide tomorrow (or contribute to it if someone else starts one).

@nurtext
Copy link
Author

@nurtext nurtext commented May 31, 2017

Some more debugging for you:

Cleaned up everything: intermediate containers, docker-sync-stack, containers, etc. No change at all.

Changed a file on host: no result.

  1. docker exec -it api-app-sync /bin/bash
  2. ps aux
  3. kill -9 $PID_OF_UNISON

Changed file got recognized:

Contacting server...
Looking for changes
Reconciling changes
changed  ---->            Resources/views/root.html.twig
Propagating updates
UNISON 2.48.4 started propagating changes at 09:52:26.44 on 31 May 2017
[BGN] Updating file Resources/views/root.html.twig from /host_sync to /app_sync
[END] Updating file Resources/views/root.html.twig
UNISON 2.48.4 finished propagating changes at 09:52:26.45 on 31 May 2017
Saving synchronizer state
Synchronization complete at 09:52:26  (1 item transferred, 0 skipped, 0 failed)
Looking for changes

Tried to change a file afterwards: Same behavior and no more changes got recognized.

Result of ps aux:

PID   USER     TIME   COMMAND
    1 root       0:00 {supervisord} /usr/bin/python /usr/bin/supervisord
   16 root       0:00 tail -F /tmp/unison.log
   24 root       0:00 /bin/bash
   68 root       0:00 tail -f unison-stdout---supervisor-T9b2x9.log
   69 root       0:00 /bin/bash
   75 root       0:00 unison -prefer /host_sync -numericids -repeat watch -auto -batch /host_sync /app_sync -logfile /tmp/unison.log
   76 root       0:00 /usr/local/bin/unison-fsmonitor
   79 root       0:00 ps aux
@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented May 31, 2017

@nurtext @codycraven great. First - lets put in the guide in place, since that will be the first thing to ensure we are all doing reproductionable steps and are all on the same way - the only way to really find the actual issue pattern. Could you help with #415

@codycraven if you start the guide, i join you adding what i know about the containers, images, supervisord, debugging and things like that - please all chats about this further in #413 - i created a first outline of what we should do and why, feel free to actually be bold in any sense - i just emptied my mind there

@michaelbaudino Mr. Spec, anything we could do with spec to formalize testing? I guess we would need to start with the bash thingy and it will take longer then we want to get this thing on the line, right? Created #414 for this - lets see

Please try to spread the comments for the topics across the outer issues - the only thing we should do in this tickets is giving analysis results and give people a spot to "i am also effected" - and also advertise debugging steps / new things we want to know to get it fixed.

Lets get our hands dirty on this - i see that is a serious issue here and we need to get this done ASAP, but i will need assistance to get it into a timeframe it suits you.

@nurtext
Copy link
Author

@nurtext nurtext commented Jun 1, 2017

Some more debugging according according to the guide in the wiki…

Host Sync working:

> diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/host_sync/$DEBUG_DOCKER_FILE")
> 

App Sync not working:

> diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/app_sync/$DEBUG_DOCKER_FILE")
Files Resources/views/root.html.twig and /dev/fd/11 differ

Log is empty:

> docker exec "$DEBUG_DOCKER_SYNC" tail -n70 /tmp/unison.log
> 

Missing directory:

> docker container inspect --format '{{(index .Mounts 1).Source}}' "$DEBUG_DOCKER_SYNC"
/var/lib/docker/volumes/api-app-sync/_data
> docker exec "$DEBUG_DOCKER_SYNC" ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory
> ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory

Will now try to reset d4m…

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Jun 1, 2017

No log means no unison is actually starting @nurtext, please see the supervisor logs for the reasons that this does not actually work or try to restart unison:

supervisorctl restart unison

@EugenMayer EugenMayer closed this Jun 1, 2017
@OO00O0O
Copy link

@OO00O0O OO00O0O commented Nov 28, 2017

And I think we should have in mind, what projects we are doing. Because I'm working on symfony project, and I only sometimes have conflicts and had this no sync problem only maybe twice in 5 months. But frontend team working with same config, having this problem few times a week. They using Angular 1/2.

version: "2"
syncs:
  backend-sync:
    src: '../backend'
    sync_strategy: 'unison'
    sync_excludes: ['.git', '.idea']
    # sync_excludes: ['.git', '.idea', 'var/cache']
    sync_excludes_type: 'Path'
    watch_strategy: 'fswatch'

  frontend-sync:
    src: '../frontend'
    sync_strategy: 'unison'
    sync_excludes: ['.git', '.idea']
    sync_excludes_type: 'Path'
    watch_strategy: 'fswatch'
@EugenMayer EugenMayer changed the title 0.4.6 not syncing both directions on macOS 10.11.6 Not syncing both directions suddenly Jan 12, 2018
@b1alpha
Copy link

@b1alpha b1alpha commented May 14, 2018

Is this fixed? A few devs are starting to see docker-sync just stop. logs dont show any issue its just not picking up any changes. clean and start didnt seem to resolve then about 5min later it started working again

@ddanier
Copy link

@ddanier ddanier commented Jul 7, 2018

@EugenMayer Your gist (https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2) seems not to work any more. I get "fatal: not a git repository (or any of the parent directories): .git". Is there an updates version? Thank you!

@OO00O0O
Copy link

@OO00O0O OO00O0O commented Jul 9, 2018

Our team gave up. We went to bare metal dev env. Some have VMware.

@markshust
Copy link

@markshust markshust commented Jul 9, 2018

If you use the native delegated volumes, docker-sync isn't needed anymore as far as I'm concerned. I've had solid performance on Magento 2 (TONS of folders/files) with no delay or halts in syncing.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Jul 22, 2018

closing this, use d4m edge to get it very robust, we have a about one sync stop per week at max

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Jul 22, 2018

@ddanier this is no longer needed with d4m edge

@EugenMayer EugenMayer closed this Jul 22, 2018
@ohcibi
Copy link

@ohcibi ohcibi commented Nov 30, 2018

I just started using docker-sync with latest versions of everything. Initial sync works but then nothing gets sync. Even with explicitly running docker-sync sync. I just saw an error message:

unix:///tmp/supervisor.sock refused connection

When I tried to restart everything as suggested here.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Nov 30, 2018

I would suggest opening a ticket, paste your configuration there and continue from there

@adriadrop
Copy link

@adriadrop adriadrop commented Feb 12, 2019

Maybe just to sum this up. Solution for this would be to either put docker4mac edge or use rsync as sync strategy?

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Feb 12, 2019

d4m will do it (in the meantime)

Or to just understand how to configure your excludes and have a proper project layout mitigating the issues of d4m.

Reduce the amount of files you

  • watch
  • you sync

to only the parts you edit.

@infostreams
Copy link
Contributor

@infostreams infostreams commented Apr 6, 2020

Maybe I was to stupid and didn't install everything correctly, but I think I followed the installation instructions correctly. However, docker-sync never worked for me.

I mount folder X from my Mac to docker-sync, and I'm running docker-sync with docker-machine. If I changed a file in folder X on my Mac, I would see it in the host_sync/ folder, but unison never picked up the change, and the change never ended up in the app_sync/ folder, and thus never in my own docker container.

However, if I opened a shell in the docker container running docker-sync, and manually made a change in the host_sync/ folder from within the docker-sync container (e.g. by doing touch /host_sync/examplefile.txt), then unison would see that something changed, and copy it to the app_sync/ folder. So, the problem was with Unison, somehow.

After a long search it turns out that this particular behavior is related to a WONT-FIX bug in VirtualBox (see here and here). Basically, a change to a file that was initiated on my Mac does not result in a filesystem event within the container, and is therefore never picked up by Unison. Unison's documentation even explicitly states this (see documentation: "In particular, the Unix implementation does not compare the actual contents of files to their previous contents, but simply looks at each file's inode number and modtime; if neither of these have changed, then it concludes that the file has not been changed.")

So, I'm now running a separate program inside the docker-sync container that regularly polls my host_sync/ folder, and whenever it detects a change, it touches the file, which causes a filesystem event inside the container, which then causes Unison to pick up the change, which means my changes end up in my own container 🎉

I initially installed inotify-proxy to do this, but I didn't manage to get it to work properly, so I coded up my own version in PHP. It's plenty fast and seems to work as intended. Hope this helps someone here.

If this whole story above is old news and there's existing code in docker-sync to deal with this, can you point me to it, and either update the documentation to point it out clearly, or make it the default behavior? As I interpret the Unison documentation ("considered unchanged if inode is the same") and the VirtualBox documentation ("we don't pass on inode changes") I don't think the existing setup could have ever worked for Docker running on VirtualBox.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Apr 6, 2020

@infostreams this and more reasons are, why you should use OSXFS, thus d4m and not vbox/fusion, since the FS driver of OSXFS is better in those terms and fires all the events.

There is no doubt that docker4mac and OSXFS has issues too, but i consider it the better alternative.

Thanks for sharing your story in that detailed fashion though, very appreciated!

@infostreams
Copy link
Contributor

@infostreams infostreams commented Apr 6, 2020

@EugenMayer I don't think the documentation mentions anywhere that this, out of the box, doesn't work with docker-machine. If I had known that I could have resolved my issues by switching to docker4mac, I would have done that. As it is I spent a day or so on this problem. Can you update the documentation to say something along these lines? It would have saved me (and no doubt others) a lot of frustration.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Apr 7, 2020

@infostreams checking the docs, i really have no topic about d4m vs docker-machine dedicated to this topic beside the tests under https://docker-sync.readthedocs.io/en/latest/miscellaneous/performance.html

But we had this #346 ..so exactly your finding so indeed we could have saved some time for you.

But that is how docs are, they are never complete.

We have a topic about how to pick the right strategy at https://docker-sync.readthedocs.io/en/latest/advanced/sync-strategies.html and that currently suggest OSXFS eventhough it does not explain all the details, including those details - but following that guide would have saved you time too :)

Anyway, we should probably add a "docker-machine vs d4m/d4w" topic - are you willing to do a PR for that? That would be awesome

Update: it was there: https://docker-sync.readthedocs.io/en/latest/advanced/sync-strategies.html

It works under Docker for Mac only - missing file system events under vbox/fusion. See native_osx does not work with docker-machine vbox / fusion

under cons

@infostreams
Copy link
Contributor

@infostreams infostreams commented Apr 7, 2020

Hi, thanks for pointing out that reference. However, I completely missed that single sentence, and I wouldn't be surprised if many others missed it as well. I will open a pull request to add a single sentence to the installation instructions for Mac, because only saying "With native_osx we no longer have any host dependencies" is slightly misleading since it only works on one of the two ways to run Docker on Mac. Thanks for your great work!

@ostrolucky
Copy link
Contributor

@ostrolucky ostrolucky commented Apr 9, 2020

Hey guys, I have run into this issue recently. I've read the thread and it mentions it shouldn't happen with d4m edge anymore. But here we are, with d4m 2.2.3.0 edge and it still happens. Restart of my Mac fixed it. I think I didn't try restarting Docker itself though, could maybe help as well.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Apr 9, 2020

@ostrolucky this is known and edge does not fix it ( or anything ). Either FSevents get stuck inside macOS ( macOS restart needs restart ) or inotify events inside the kernel of d4m ( docker restart might help ) and sometimes it's unison inside the sync container getting overheated by the amount of inotify events ( huge file change chunks ) which can be fixed by either restarting unison and sometimes in addition removing the lockfile of unison

To be frank, this is unfixable by docker-sync - this is broken conceptually by macOS ( horrible FSevents implementation ), leading to a half baked OSXFS implementation on top of it, and in the end unison, which simply cannot handle fast changes of 20k files like an npm install would do

@ostrolucky
Copy link
Contributor

@ostrolucky ostrolucky commented Apr 9, 2020

this is known and edge does not fix it ( or anything )

Good, this is just to make that transparent, because reading this thread it implies this was fixed in d4m edge. So good to make that clear that it indeed doesn't fix it.

which simply cannot handle fast changes of 20k files

Do you think mutagen can? If it does, perhaps we should write a strategy for it in docker-sync. Because TBH it's annoying having to exclude such folders with dependencies. Although we are a bit OT now, but there is no community forum or anything like that for docker-sync AFAIK.

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Apr 9, 2020

@ostrolucky there is no community forum and yes, the latter might be fixable by mutagen, but nobody really tried - see #603 for a issue about implementing mutagen. Feel free to join there and maybe even contribute with a PR or try to bring mutagen to those edge cases.

I will close this thread with repeating the answer above so people now the "essence" - got your point, good idea

@EugenMayer
Copy link
Owner

@EugenMayer EugenMayer commented Apr 9, 2020

d4m does not fix it - there are several layers of issues:

  • Either FSevents get stuck inside macOS ( macOS restart needs restart ) \
  • or inotify events inside the linux kernel of d4m ( docker restart might help )
  • and sometimes it's unison inside the sync container getting overheated by the amount of inotify events ( huge file change chunks ) which can be fixed by either restarting unison and sometimes in addition removing the lockfile of unison

To be frank, this is unfixable by docker-sync - this is broken conceptually by macOS ( horrible FSevents implementation ), leading to a half baked OSXFS implementation on top of it, and in the end unison, which simply cannot handle fast changes of 20k files like an npm install would do.

So there are cases where sync stops, many of them fall in the one mentioned above - and we can only try to mitigate those

Repository owner locked as resolved and limited conversation to collaborators Apr 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet