Not syncing both directions suddenly

[nurtext]

Error/Feature Requestion/Docs

All docker container are up, but not syncing between /host_sync and /app_sync.
/app_sync to /host_sync is working.

Docker Driver

d4m

Sync strategy

native_osx

docker-compose.yml

version: '2'

services:
    api:
        container_name: api
        image: api
        build: .
        ports:
            - 8888:80
        volumes:
            - ./app:/srv/api/app
            - ./bin:/srv/api/bin
            - ./src:/srv/api/src
            - ./tests:/srv/api/tests
            - ./web:/srv/api/web
            - ./vendor:/srv/api/vendor
        environment:
            DATABASE_HOST: "127.0.0.1"

docker-compose-dev.yml

version: '2'

services:
    api:
        volumes:
            - api-app-sync:/srv/api/app:nocopy
            - api-bin-sync:/srv/api/bin:nocopy
            - api-src-sync:/srv/api/src:nocopy
            - api-tests-sync:/srv/api/tests:nocopy
            - api-web-sync:/srv/api/web:nocopy
            - api-vendor-sync:/srv/api/vendor:nocopy

volumes:
    api-app-sync:
        external: true

    api-bin-sync:
        external: true

    api-src-sync:
        external: true

    api-tests-sync:
        external: true

    api-web-sync:
        external: true

    api-vendor-sync:
        external: true

docker-sync.yml

version: '2'

options:
    verbose: true

syncs:
    api-app-sync:
        src: './app/'

    api-bin-sync:
        src: './bin/'

    api-src-sync:
        src: './src/'

    api-tests-sync:
        src: './tests/'

    api-web-sync:
        src: './web/'

    api-vendor-sync:
        src: './vendor/'

docker ps output

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                            NAMES
72ee4eb5a02b        api                              "docker-php-entryp..."   2 minutes ago       Up 2 minutes        9000/tcp, 0.0.0.0:8888->80/tcp   api
7ec87ccaa84f        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   15 minutes ago      Up 2 minutes        500/tcp                          api-vendor-sync
de22a140b897        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-web-sync
b1d749f9c992        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-tests-sync
62d9a55979a3        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-src-sync
0af351c80ed8        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-bin-sync
45d5f50d0659        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-app-sync

docker-sync list output

      ok  Found configuration at /Users/cedric/api/docker-sync.yml

api-app-sync On address :
api-bin-sync On address :
api-src-sync On address :
api-tests-sync On address :
api-web-sync On address :
api-vendor-sync On address :

OS

macOS 10.11.6

[michaelbaudino]

Hi @nurtext,

First, thanks for the detailed description 👍

Could you please add the output of docker logs api-web-sync ? (or whatever *-sync container you're trying to sync files with)

[codycraven]

I've ran into this intermittently since going to native_osx. There is nothing visible in docker logs that indicates a problem, however I suspect it stems from timestamps being out of sync between the host and the container.

In my local configs I'm now applying volumes to my containers that use docker-sync volumes like so:

    api:
        volumes:
            - /etc/localtime:/etc/localtime:ro # Fix TZ issues w/ docker-sync?
            - api-app-sync:/srv/api/app:nocopy
            - api-bin-sync:/srv/api/bin:nocopy
            - api-src-sync:/srv/api/src:nocopy
            - api-tests-sync:/srv/api/tests:nocopy
            - api-web-sync:/srv/api/web:nocopy
            - api-vendor-sync:/srv/api/vendor:nocopy

I don't know how to reproduce the issue so I'll need to see overtime if this "fix" prevents the state from occurring.

Update: This doesn't work.

[EugenMayer]

@codycraven so the issue was your time? (does it work now?)

since timestamp compares are used to detect if a file has been changed, that could be cause

[codycraven]

@EugenMayer thus far I have not encountered sync failing since implementing this earlier today.

My PR for my team was merged about an hour ago, so I should know by mid-week next week if this solved it (most of our team is traveling tomorrow and Monday is a holiday so I won't have a good sample until then).

[EugenMayer]

Very interesting, lets see how this turns out @codycraven - keep us posted. We do handle timezones: https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/sync_strategy/native_osx.rb#L86 in the sync container.

So if your app container does not do that and your TZ differs, that is a good source for issues. This leaves us with the question, how we could solve this for the docker-sync user - seems like we can only do that by documentation?

[michaelbaudino]

Ho, yes, this is very confusing 😕

→ date && docker exec frontend-sync date && docker-compose run frontend date
Fri May 26 10:52:41 CEST 2017
Thu May 25 20:32:38 CEST 2017
Thu May 25 18:33:07 UTC 2017

On MacOS Sierra 10.12.4.

[EugenMayer]

Thats a very common docker issue its basically there always, just not always visible. but since we compare timestamps on 2 different containers, that happens

[azamat163]

Hi guys!
I also encountered such a problem, but solved it with the help of the Mac OS update to 10.12.5. It helped me.

[EugenMayer]

That is pretty odd, how does this get fixed by an Mac OS update .. were did you had this hint from? Which OS did you had before, El Capitan?

[azamat163]

Yes,10.11.4

[EugenMayer]

@codycraven are you / your collegues on El Capitan?

[codycraven]

@EugenMayer I'm on Sierra. Looking at the native_osx.rb Docker run command is what gave me the idea to sync the time zone. Going a step further (if this doesn't resolve it) a colleague had the idea to sync the containers (VM) and host with NTP.

[nurtext]

Same timezone on host and all docker containers, problem still remains:

> date && docker exec api date && docker exec api-app-sync date
Mon May 29 11:39:34 CEST 2017
Mon May 29 11:39:34 CEST 2017
Mon May 29 11:39:34 CEST 2017

Logfile:

> docker logs api-app-sync
2017-05-29 11:29:05,061 CRIT Supervisor running as root (no user in config file)
2017-05-29 11:29:05,062 WARN Included extra file "/etc/supervisor.conf.d/supervisor.daemon.conf" during parsing
2017-05-29 11:29:05,097 INFO RPC interface 'supervisor' initialized
2017-05-29 11:29:05,097 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2017-05-29 11:29:05,097 INFO supervisord started with pid 1
2017-05-29 11:29:06,100 INFO spawned: 'unison' with pid 27
2017-05-29 11:29:07,271 INFO success: unison entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Edit:
Maybe this is another puzzle piece, but I doubt so: I recently moved from ubuntu:trusty-20170214 to php:7.1-fpm-alpine. Previous setup was working fine with native_osx sync

[michaelbaudino]

I discovered something that may help: my team leader (who's running the exact same Docker Compose and Docker Sync config than I do) is impacted by this problem and using @codycraven's workaround to share /etc/localtime fixed the issue for him (thanks for that, @codycraven, btw 🎩 ).

The only difference I can think of is that my containers and host seem totally desynchronized while his containers seem to have more consistent times:

→ date && docker exec frontend-sync date && docker-compose run frontend date
Mon 29 May 2017 16:49:06 BST
Mon May 29 16:49:06 BST 2017
Mon May 29 15:49:18 UTC 2017

[EugenMayer]

@michaelbaudino are you using any "--prefer newer" syntax?

[michaelbaudino]

Nope: we use the default. Here is our exact config: https://gist.github.com/michaelbaudino/1c83c253794e5b6bd0cc8fc47c1b4951

[EugenMayer]

@michaelbaudino what about OS differences, are you running Sierra and he is running El Capitan?

Still trying to understand the pattern here

[michaelbaudino]

Nope, we are both on Sierra 10.12.4.

Actually, we were, because I upgraded to 10.12.5 about 5 minutes ago 😄

[codycraven]

@michaelbaudino when you added the TZ fix had you destroyed your old containers/volumes before bringing them back up or just stopped them?

[codycraven]

I just ran into this again on my machine. The time is synchronized perfectly so I continued digging.

I checked the file I modified in my host within the docker-sync container and was astounded to find that the file within /host_sync/ was the old version that does not match what is on my host.

Based on this, it appears that the volume mount into the docker-sync container is somehow breaking.

Update
This appears to be a false failure. I created a copy of my project in a new directory and ran it while debugging something. I failed to properly clean up afterwards as my sync container's bind mount was pointed at my copied directory instead of the one I was working on.

[masev]

I have exactly the same issue. I updated to the last version of Sierra and tried the hack to share localtime with no luck ...

Tell me if I can help you by giving more info.

[EugenMayer]

Ok so what we have right now is that:

a) the timezone does matter - but not for everyone.

b) Sierra or el Capitan are not significant also - both could have this issue

c) there is no pattern in the project configuration or anything too special

---

Bottom line is, that we need to draw a tighter circle around the issue, it's far too broad right now. Suggestions:

I) we write a wiki guide on how to debug issues or gather information - this is overdue anyway. So compare host/app any, check /tmp/unison
. How to enter a snyc container, how to manually restart the inner unison project

II) we gather information from all people having this issue and all not having this one and see, were we have important differences - I would suggest a Google spreadsheet for this probably

Anybody willing to help bootstrapping this?

[nurtext]

@EugenMayer Since I'm depending upon a working docker-sync for the next release of my project, I'm willing to help. Tell me what's up next and I'll do my best.

[codycraven]

I'm in, I can start work on a debug guide tomorrow (or contribute to it if someone else starts one).

[nurtext]

Some more debugging for you:

Cleaned up everything: intermediate containers, docker-sync-stack, containers, etc. No change at all.

Changed a file on host: no result.

1. docker exec -it api-app-sync /bin/bash
2. ps aux
3. kill -9 $PID_OF_UNISON

Changed file got recognized:

Contacting server...
Looking for changes
Reconciling changes
changed  ---->            Resources/views/root.html.twig
Propagating updates
UNISON 2.48.4 started propagating changes at 09:52:26.44 on 31 May 2017
[BGN] Updating file Resources/views/root.html.twig from /host_sync to /app_sync
[END] Updating file Resources/views/root.html.twig
UNISON 2.48.4 finished propagating changes at 09:52:26.45 on 31 May 2017
Saving synchronizer state
Synchronization complete at 09:52:26  (1 item transferred, 0 skipped, 0 failed)
Looking for changes

Tried to change a file afterwards: Same behavior and no more changes got recognized.

Result of ps aux:

PID   USER     TIME   COMMAND
    1 root       0:00 {supervisord} /usr/bin/python /usr/bin/supervisord
   16 root       0:00 tail -F /tmp/unison.log
   24 root       0:00 /bin/bash
   68 root       0:00 tail -f unison-stdout---supervisor-T9b2x9.log
   69 root       0:00 /bin/bash
   75 root       0:00 unison -prefer /host_sync -numericids -repeat watch -auto -batch /host_sync /app_sync -logfile /tmp/unison.log
   76 root       0:00 /usr/local/bin/unison-fsmonitor
   79 root       0:00 ps aux

[EugenMayer]

@nurtext @codycraven great. First - lets put in the guide in place, since that will be the first thing to ensure we are all doing reproductionable steps and are all on the same way - the only way to really find the actual issue pattern. Could you help with #415

@codycraven if you start the guide, i join you adding what i know about the containers, images, supervisord, debugging and things like that - please all chats about this further in #413 - i created a first outline of what we should do and why, feel free to actually be bold in any sense - i just emptied my mind there

@michaelbaudino Mr. Spec, anything we could do with spec to formalize testing? I guess we would need to start with the bash thingy and it will take longer then we want to get this thing on the line, right? Created #414 for this - lets see

Please try to spread the comments for the topics across the outer issues - the only thing we should do in this tickets is giving analysis results and give people a spot to "i am also effected" - and also advertise debugging steps / new things we want to know to get it fixed.

Lets get our hands dirty on this - i see that is a serious issue here and we need to get this done ASAP, but i will need assistance to get it into a timeframe it suits you.

[nurtext]

Some more debugging according according to the guide in the wiki…

Host Sync working:

> diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/host_sync/$DEBUG_DOCKER_FILE")
> 

App Sync not working:

> diff -q "$DEBUG_DOCKER_FILE" <(docker exec "$DEBUG_DOCKER_SYNC" cat "/app_sync/$DEBUG_DOCKER_FILE")
Files Resources/views/root.html.twig and /dev/fd/11 differ

Log is empty:

> docker exec "$DEBUG_DOCKER_SYNC" tail -n70 /tmp/unison.log
> 

Missing directory:

> docker container inspect --format '{{(index .Mounts 1).Source}}' "$DEBUG_DOCKER_SYNC"
/var/lib/docker/volumes/api-app-sync/_data
> docker exec "$DEBUG_DOCKER_SYNC" ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory
> ls -la /var/lib/docker/volumes/api-app-sync/_data
No such file or directory

Will now try to reset d4m…

[EugenMayer]

No log means no unison is actually starting @nurtext, please see the supervisor logs for the reasons that this does not actually work or try to restart unison:

supervisorctl restart unison

[mmrko]

Meanwhile, the rsync strategy works beautifully. Kudos to @EugenMayer & co. for their hard (and gratuitous) work! :)

[EugenMayer]

@mmrko yes, rsync is not affected by this, thats why having multiple strategies is one of our goals - pick the axe for your case. The problem with rsync is on the other side, CPU load, since it uses fswatcher to watch for file changes - and due to macOS being horrible in FS-Events, even that is just bad as it is. So its pretty bullet proof, but not entirely.

But more or less, all strategies have their little edges, thats assured. But running drush for streight up 2 minutes for a drush cc all on the cli alone, instead of 3 seconds - is a much harder "edge" for most of us, i guess.

I would love to remove docker-sync from this world, if it would not be needed .e.g d4m fixed it. That will be a great day for us all - no overhead, no glitches, no extra configuration - just docker.

Well d4m is not just docker, its docker on a non linux kernel...and thats where we all start.

[EugenMayer]

Closed the thread for collaborators only so this thing does not get offtopic even more - way to important issue with way to valuable information, no need for all the other.

Will reopen the issue in 1 week when things calmed down. If you feel in the need to express yourself about this, open a new issue in the queue so this one keeps the focus

[EugenMayer]

Anybody having this issue, please see #497 and run the reproduction script, post your details.

Also ensure you test it 2 times, one without my d4m fs fix and one with: https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2

Does the result change for you? This could really help use nailing down the issue.

Why i am assuming that my fix helps is, that neither i have this issue often ( about 1 time in 3 months ) neither anybody of my coworkers - we are doing heavy development though, it means, one project most probably having 0.5 million files, 5 docker-sync endpoints starting at the same time and a stack which consume 8GB Ram + quiet some CPU. Still it hardly ever happens for us ( i think for most of us, it never happened at all). We have other projects, bigger Java projects, bigger ng4 stacks and so on, still it hardly ever happens for us.

So there must be something we do that this does not happen. One thing i can tell is:

• we are either using Sierra or Mountain Lion
• we are all using that fix above
• we are all having about 8-16GB RAM assigned to d4m
• at least 2-4 CPUs ( depending if it is a desktop or laptop )

[EugenMayer]

i am currently thinking about closing this issue and open a new one, extracting all the facts / informations we had here, also how to test owns system to provide informations so we can see pattern. I would link this issues as the source of deeper discussions.

Any concerns about that? Main reason is, to consolidate the results and make it easier to understand how this issues shows up, how you can for now work around and what the reasons for this are, without reading so many posts.

[zedtux]

@EugenMayer I agree with you, it should also allow new comers to quicker understand the issue and where we stand.

[BunnyHolder]

And I think we should have in mind, what projects we are doing. Because I'm working on symfony project, and I only sometimes have conflicts and had this no sync problem only maybe twice in 5 months. But frontend team working with same config, having this problem few times a week. They using Angular 1/2.

version: "2"
syncs:
  backend-sync:
    src: '../backend'
    sync_strategy: 'unison'
    sync_excludes: ['.git', '.idea']
    # sync_excludes: ['.git', '.idea', 'var/cache']
    sync_excludes_type: 'Path'
    watch_strategy: 'fswatch'

  frontend-sync:
    src: '../frontend'
    sync_strategy: 'unison'
    sync_excludes: ['.git', '.idea']
    sync_excludes_type: 'Path'
    watch_strategy: 'fswatch'

[b1alpha]

Is this fixed? A few devs are starting to see docker-sync just stop. logs dont show any issue its just not picking up any changes. clean and start didnt seem to resolve then about 5min later it started working again

[ddanier]

@EugenMayer Your gist (https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2) seems not to work any more. I get "fatal: not a git repository (or any of the parent directories): .git". Is there an updates version? Thank you!

[BunnyHolder]

Our team gave up. We went to bare metal dev env. Some have VMware.

[markshust]

If you use the native delegated volumes, docker-sync isn't needed anymore as far as I'm concerned. I've had solid performance on Magento 2 (TONS of folders/files) with no delay or halts in syncing.

[EugenMayer]

closing this, use d4m edge to get it very robust, we have a about one sync stop per week at max

[EugenMayer]

@ddanier this is no longer needed with d4m edge

[ohcibi]

I just started using docker-sync with latest versions of everything. Initial sync works but then nothing gets sync. Even with explicitly running docker-sync sync. I just saw an error message:

unix:///tmp/supervisor.sock refused connection

When I tried to restart everything as suggested here.

[EugenMayer]

I would suggest opening a ticket, paste your configuration there and continue from there

[adriadrop]

Maybe just to sum this up. Solution for this would be to either put docker4mac edge or use rsync as sync strategy?

[EugenMayer]

d4m will do it (in the meantime)

Or to just understand how to configure your excludes and have a proper project layout mitigating the issues of d4m.

Reduce the amount of files you

• watch
• you sync

to only the parts you edit.