What would happen if you took git, sed, rsync, and inotify, and you mushed and kneaded them together until smooth? You'd get gut-sync: real-time bi-directional folder synchronization.
I wrote gut so that I can edit source code locally on my desktop, have the programs I'm working on run either in the cloud or in a VM, and then I can seamlessly transition to a laptop -- with all of my changes synced continuously to each of these machines. I don't want to work with flakey remote filesystems, and I do want to have inotify (or kqueue or FSEvents or whatever) work correctly so that I can kick off builds just by hitting Save in my editor.
gut-sync solves this problem for me by using a modified version of git to synchronize changes between multiple (1 to N) systems in real-time. It's efficient and stable, written in Go, but it's sort of like a big shell script written in Go: all the heavy lifting is done by calling out to other utilities. It uses inotifywait (on Linux) or fswatch (on OSX) to listen for file changes, and then orchestrates calls to gut-commands (such as gut-add or gut-fetch) on each system in order to keep all systems up-to-date.
gut-sync has been tested on and between OSX and Ubuntu.
There are some other really awesome (perhaps more awesome for your use case) file sync tools out there. Here are the big differientiators (and may count as plusses or minuses for you) for gut-sync as compared to others:
- gut-sync uses git under the hood. (well, "gut", a machine-renamed version of git)
- If you already know git, or are an expert using git, then you can use that experience to configure, tweak, and explore/modify the gut-sync history.
- If you are not familiar at all with git, you may prefer another tool that exposes history/versions in a more user-friendly way.
- If you want to sync a lot of large files, or large rapidly-changing files, the overhead of using git (which never deletes history) may be too expensive.
- gut-sync communicates and deploys itself via SSH.
- If you already use SSH everywhere, then this means deploying and using gut-sync will be easy.
- If you don't use SSH, then another tool may be a better fit.
Here's a short list of some other tools you might want to check out:
- Syncthing: Really fantastic cross-platform open-source (MPLv2) sync daemon built around its own synchronization protocol. Runs a web GUI locally for easy setup.
- Unison, SparkleShare: Similar, open-source yet somewhat-abandoned sync tools.
- Finally, there are Dropbox, Google Drive, and many other similar hosted file sync services. These services store and transmit through a third party, which may incur latency and monetary cost in addition to reduced privacy/security.
If you have not a care for security, you could just cross your fingers and run this:
bash -c 'S="3bceab0bdc63b2dd7980161ae7d952ea821a23e693cb74961b0d41f61f557489";T="/tmp/gut.sh";set -e;wget -qO- "https://www.tillberg.us/c/$S/gut-1.0.3.sh">$T; echo "$S $T"|shasum -a256 -c-;bash $T;rm $T'
This will download and install the correct gut
Go binary to /usr/local/bin/gut
. It verifies
the SHA256 sum of the script it downloads, and then in turn in the SHA256 sum of the binary it
subsequently downloads & installs, but it doesn't verify the integrity of the author.
But shrug, right?
To install from source, first you'll need the go compiler installed (v1.4 or later). The Go install documentation is a good place to start if you haven't set up Go already.
After you have Go installed, you also need to set your GOPATH. You probably just want to
use the defaults: (add these to your .profile
/.bash_profile
to persist to new shell sessions)
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
With that all configured, to install gut-sync into $GOPATH/bin, run:
go get github.com/tillberg/gut
By default gut-sync will download (and verify) pre-built binaries for gut-commands.
You can optionally build these from source; much of the process is actually automated
but requires the same dependencies (build-essentials, autoconf, etc) required to build
git from source. To do so, use --build-deps
, i.e. gut sync --build-deps ...
.
Let's say that you want to create a pair of linked folders, ~/work locally and ~/work2 on my.server.com. Fire up a terminal and run something like this:
$ gut sync ~/work username@my.server.com:~/work
This command sets up a gut repo locally in ~/work and clones it to your ~/work2 directory on my.server.com, then starts watching the filesystem on both ends for changes. When a change is made, gut-sync commits the change and then merges it to the other server.
Open up a second terminal and make gut do some work:
$ cd ~/work
$ git clone https://github.com/tillberg/gut.git
$ cd gut
$ rm util_test.go
$ git add . --all
$ git commit -m 'made all tests pass'
Then hop onto the other host and take a look at what's there.
$ cd ~/work2/gut
$ git log --stat
# ... <- You should see the commit you just made
$ gut log --stat
# ... <- You should see *all* the file changes recorded here, including inside ~/work2/gut/.git/
To exclude files from gut-sync, use .gutignore files just as you'd use .gitignore over in git-world.
gut-sync connects to the ssh agent specified by SSH_AUTH_SOCK
and uses the experimental
golang.org/x/crypto/ssh SSH client. This does not, for example, read any settings
in ~/.ssh/config, and it may differ in a number of other ways from using the ssh
OpenSSH client,
such as Username settings and Hostname aliases. For many, this will work just fine (as it does
on all of my systems). If it doesn't work for you, though, please create issues and/or PRs with
as much detail as you can provide about where it breaks down (thanks!).
If you see this message, it means you've run out of inotify watch slots. You can increase this limit
temporarily by writing to /proc/sys/fs/inotify/max_user_watches
, or permanently by modifying the
fs.inotify.max_user_watches
sysctl property. See
this great page about inotify max_user_watches on the guard/listen
project for tips and additional details.
Alternately, you could reduce the total number of directories inside the folder you're synchronizing.
In addition to removing folders you don't wish to sync, some options include running git gc
inside
less-used repositories, removing unused node_modules
dependencies (which tend to span a large number
of directories), and more generally scanning the output of find /path/to/gut/repo -type d
for cases
where a large number of directories is being used.
Note that inotifywait/fswatch don't exclude .gutignore
d paths from being wired up for change
notifications, which would be a great way to cut down on noise and watch-slot consumption from large
directory hierarchies which we're not synchronizing, anyway.
I've done some implementation work for Windows (and had a fully-functioning Python implementation before porting to Go -- it's definitely feasible), and so if you're interested in either using that or helping to implement, open up an issue for discussion and/or tag #4.
The reason it's necessary to use a modified version of git, and not git itself, is that stock git will refuse to traverse into .git folders, which is critical to using gut-sync to synchronize folders containing git repos. Other than the name-change, though, gut is the same as git.
You can use gut just like you'd use git, if you want:
$ gut init
Initialized empty Gut repository in /tmp/test/.gut/
$ touch README
$ gut add README
$ gut commit -m 'First gut commit'
[master (root-commit) f216bb4] First gut commit
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 README
This means that you can use the various gut(git) commands, e.g gut log -p
,
gut show HEAD
, gut log --stat
, and even gutk
(it's installed at
~/.guts/gut-build/bin/gutk
) to examine the history of whatever gut-sync
does. So if and when gut-sync screws something up, you might (might) be
able to repair the damage by referencing the gut history and/or doing a
hard-reset to an older version.
You'll probably have a tough time speaking to remote git repos, though. Github, for one, doesn't support gut-receive-pack. :)
$ gut push -u origin master
Invalid command: 'gut-receive-pack 'tillberg/test.git''
You appear to be using ssh to clone a git:// URL.
Make sure your core.gitProxy config option and the
GIT_PROXY_COMMAND environment variable are NOT set.
fatal: Could not read from remote repository.