sftpclone

A tool for cloning/syncing a local directory tree with an SFTP server.

Features

• Keep in sync a local directory tree with a specified folder of an SFTP server.
• Update symbolic links as needed and keep files consistent.
• Automatic tilde expansion/handling on the SFTP server.
• Public key authentication.
• ssh_config entries compatibility.
• Syncing exclusion patterns.
• Compatible with both Python 2 and Python 3.

Install

You can install sftpclone by using pip:

$ pip install sftpclone --user

Note: Sometimes building required dependencies in user mode doesn't work. In that case, you'd need to use sudo and to remove the --user flag. Alternatively, you could make use of a virtualenv.

Alternatively, you can clone this repository and then launch:

$ git clone https://github.com/unbit/sftpclone
$ cd sftpclone
$ python setup.py install

In both cases, you'll find the sftpclone script in your path.

Usage

usage: sftpclone [-h] [-k private-key-path]
                 [-l {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}] [-p PORT]
                 [-f] [-a] [-c ssh config path] [-n known_hosts path] [-d]
                 [-e exclude-from-file-path] [-t] [-o]
                 [-r --create-remote-directory]
                 local-path user[:password]@hostname:remote-path

Where, for each command line argument:

• local-path: The path of the local folder. This path must exists and can contain ~ (we use tilde expansion).
• sftp-url: It specifies the remote SFTP url having the form: [user[:password]@]hostname:remote-path. Both the password and the user field can be omitted. If you omit the former then you should specify a private key identity file. If you omit the latter then the current user is automatically used. The hostname can refer to a element of your ssh_config file. If the remote path contains ~, then it will be expanded to the default folder in which the user begins her SFTP session.
• [h]elp: show the help message and exit.
• private-[k]ey-path: the path to your private identity file. Set it if you are not using password authentication. It automatically defaults to ~/.ssh/id_rsa and can be used more than once.
• [l]ogging: set the log level (ERROR by default).
• [p]ort: SSH remote port (defaults to 22).
• [f]ix-symlinks: if you have absolute symlinks pointing to your synced directory, they will remain consistent on the remote server: i.e., they will have an absolute path that reflect the path of the cloned directory on the server. Useful for cluster configurations.
• ssh-[a]gent: enable ssh-agent support. Any private-[k]ey-path argument will be ignored.
• ssh-[c]onfig-path: in the sftp-url's hostname you can specify an entry of your ssh_config file. If you are using a non-standard path, you can set it here.
• k[n]own_hosts path: path to your known_hosts file. Default to ~/.ssh/known_hosts.
• [d]isable-known-hosts: disable remote fingerprint check against local known_host file.
• [e]xclude-from-file-path: the path to a file containing a list of patterns. Each file matched by these pattern will be ignored (not synced).
• do-not-dele[t]e: do not delete remote files that are missing from the local directory.
• all[o]w-unknown: do not ask for confirmation before connecting to unknown hosts.
• -create-[r]emote-directory: Create remote base directory if missing.

Warning: be sure to select a proper remote folder. The synchronization process will indeed delete any file that doesn't exist in the local folder (unless you turn the -t option on).

ssh_config compatibility

The hostname in the sftp-url parameter can be a valid entry in a ssh_config file. Specifically, your entry should have relevant parameters such as:

• HostName
• User
• Port
• IdentityFile
• ProxyCommand

Any value not found will fallback to the CLI arguments. Anyway, you have to set the IdentityFile field, otherwise authentication will try to fallback to ~/.ssh/id_rsa and could not work. The first hostname matching the pattern is chosen (in the ssh_config way).

known_hosts checking

By default sftpclone will match the remote host fingerprint against the one contained in your ~/.ssh/known_hosts file. If this file doesn't exists on your machine, you can specify a different path by using the -n option. Furthermore, you can disable the check with the -d flag. Unknown hosts will require the user to authorize the connection. Please note that, even after authorization, the known_host file won't be modified.

Exclude list

It takes inspiration from the rsync/tar --exclude-from flag.

You can specify among your command line arguments a file containing a list of patterns, one per each line. All those files that match any pattern will not be synced with the SFTP server.

Lines beginning with ; or # are ignored.

Each pattern is considered relative to the syncing directory. As a consequence, leading / are ignored.

Example

; This will exclude any file or directory beginning with foo
foo*
; This will exclude any file foo in a subdir of the directory bar.
bar/*/foo

Programmatic usage

You can find some examples of programmatic usage inside the examples directory.

Testing

This project uses nose for testing. In addition, on Python 2 you'll need the mock module (part of Python standard lib from 3.3). In both cases, you can install test requirements with:

$ pip install -r test_requirements.txt

Then, You can launch the test suite by using, from the project root directory:

$ nosetests
$ python setup.py test # alternatively