8. Strategies

Eugen Mayer edited this page Nov 9, 2017 · 32 revisions

Sync strategies

The sync strategies depend on the OS, so not all strategies are available on all operating system

  • OSX: native_osx, unison, rsync
  • Windows: unison
  • Linux: native_linux, unison

native_osx (OSX)

DockerSync native_osx strategy overview

For advanced understanding, please read native_osx under the hood/concept

Native-OSX is a combination of two concepts, OSXFS only and Unison together. We use OSX to sync the file-system into a sync-container to /host_sync. In that sync container we sync from /host_sync to /app_sync using Unison. /app_sync is exposed as a named volume mount and consumed in the app. You ask yourself, why? Its fairly simple.

By having this extra layer of unison on linux, we detach the actual write/read performance of your application from the actual OSXFS performance - running at native speed. Still, we using OSXFS, a about up to 1 second delayed, to synchronize changes both ways. So we have a 2-way sync.

What is different to plain Unison you might ask. The first big issue with Unison is, how bad it performs under OSX due to the bad FS-Events of OSX, implemented in macfsevents and alikes. It uses a lot of CPU for watching files, it can lose some events and miss a sync - but also, it adds extra dependencies on the OSX hosts.

All that is eliminated by native_osx - we use Unison in the Linux container, where it performs great due to inotify-event based watchers.

pros

  • far more reliable due to a low-level implementation of the sync using OSXFS only
  • uses far less CPU to sync files from the host to the sync container - will handle a lot more files
  • no daemon to run, control, restart and sync on the OSX host. Makes sleep/hibernate/reboot much more robust
  • no dependencies on the OSX-Host at all
  • a lot easier installation since we just need gem install docker-sync and that on runs under system ruby. Anything else is in containers
  • it performs at native speed
  • it will be much easier to support Windows this way

cons

unison (OSX/Windows/Linux)

This strategy has the biggest drive to become the new default player out of the current technologies. It seems to work very well with huge codebases too. It generally is build to handle 2 way sync, so syncs back changes from the container to the host.

pros

  • offers 2 way sync *(please see unison-dualside why this is misleading here)
  • still very effective and works for huge projects
  • native speed in for the application

cons

  • can be unreliable with huge file counts ( > 30.000 ) and bad hardware ( events gets stuck )
  • the daemon on OSX needs extra care for sleep/hibernate.
  • Extra dependencies we need on OSX, in need to install unison and unox natively - brew dependencies

Initial startup delays with unison

On initial start, Unison sync needs to build a catalog of the files in the synced folder before sync begins. As a result, when syncing folders with large numbers of relatively large files (for example, 40k+ files occupying 4G of space) using unison, you may see a significant delay (even 20+ minutes) between the initial ok Starting unison message and the ok Synced message. This is not a bug. Performance in these situations can be improved by moving directories with a large number of files into a separate rsync strategy sync volume, and using unison only on directories where two-way sync is necessary.

rsync (OSX)

This strategy is probably the simplest one and probably the most robust one for now, while it has some limitations. rsync-syncs are known to be pretty fast, also working very efficient on huge codebases - no need to be scared of 20k files. rsync will easily rsync the changes ( diff ) very fast.

pros:

  • fast re-syncing huge codebases - sync diffs (faster then unison? proof?)
  • well tested and known to be robust
  • best handling of user-permission handling ( mapped into proper users in the app container )

cons

  • implements a one way sync only, means only changes of your codebase on the host are transferred to the app-container. Changes of the app-container are not synced back at all
  • Deleting files on the host does yet not delete them on the container, since we do not use --delete, see this discussion

Example: On the docker-sync-boilerplate

native_linux

Native linux is actually no real implementation, it just wraps docker-sync around the plain native docker mounts which do work perfectly on linux. So there are no sync-containers, no strategies or what so ever. This strategy is mainly used just to have the whole team use docker-sync, even on linux, to have the same interface. If you use default sync_strategy in the docker-sync.yml, under Linux, native_linux is picked automatically

Sync Flags or Options

You find the available options for each strategy in the the configuration

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.