Support basic SFTP operations

[nchammas]

Following from the discussion here, this is a request for SFTP support in AsyncSSH.

I don't have any concrete proposal at this time, but I thought it would be useful to create this issue for tracking purposes.

[ronf]

A first cut of SFTP client support is now available in the 'develop' branch. It's a reasonably complete implementation of an SFTP version 3 client, but it does not yet support any extensions. Those will be added over time.

The beginnings of a framework for adding SFTP server support is also present here and support for that will come in future commits.

Support for later SFTP protocol versions may also be added, though it appears that those are not yet supported by OpenSSH, which is what I'm doing my testing against so far.

Note: This implementation attempts to follow the SFTP standard with regard to the order of arguments for the FXP_SYMLINK message. However, it has a workaround to automatically detect OpenSSH and paramiko peers and reverse these arguments to be compatible with them. Other implementations can easily be added by name if they have this problem.

Feedback on the new API is welcome. Full documentation for it is present in this checkin.

[nchammas]

Exciting! I love how quickly AsyncSSH is adding great features like this!

Sometime this weekend or next week I'll play around with this new API and send some feedback your way.

One thing I'm interested in trying out is testing parts of the SFTP API with Hypothesis, a property-based testing library which seems like it would be a good fit for testing something like an SFTP client. I'll keep you posted!

[ronf]

Thanks, sounds great. :)

By the way, I haven't checked in an official example yet, but here's one to get you started:

import asyncio, asyncssh, sys

@asyncio.coroutine
def run_client():
    with (yield from asyncssh.connect('localhost')) as conn:
        with (yield from conn.start_sftp_client()) as client:
            yield from client.chdir('/tmp')
            yield from client.put('testfile')

try:
    asyncio.get_event_loop().run_until_complete(run_client())
except (OSError, asyncssh.Error) as exc:
    sys.exit('SFTP operation failed: ' + str(exc))

[nchammas]

Sweet. This already looks great!

[ronf]

Just checked in another update here in the "develop" branch. The latest version supports recursive transfers on the get(), put(), and copy() methods, so you can copy an entire directory tree in a single call. It also now properly preserves symlinks when doing a copy, if you tell it to.

Up next I'm thinking about adding mget(), mput(), and mcopy() methods that allow you to pass in a glob expression to copy a set of files matching a wildcard pattern.

[ronf]

An initial version of SFTP server support is now available. A simple example of this is:

import asyncio, asyncssh, sys

@asyncio.coroutine
def start_server():
    yield from asyncssh.listen('', 8022, server_host_keys=['ssh_host_key'],
                               authorized_client_keys='ssh_user_ca',
                               sftp_factory=True)

loop = asyncio.get_event_loop()

try:
    loop.run_until_complete(start_server())
except (OSError, asyncssh.Error) as exc:
    sys.exit('Error starting server: ' + str(exc))

loop.run_forever()

If you want to customize the SFTP server behavior, the code would look something like:

import asyncio, asyncssh, sys 

class MySFTPServer(asyncssh.SFTPServer):
    def __init__(self, conn):
        super().__init__(conn, chroot=b'/tmp')

@asyncio.coroutine
def start_server():
    yield from asyncssh.listen('', 8022, server_host_keys=['ssh_host_key'],
                               authorized_client_keys='ssh_user_ca',
                               sftp_factory=MySFTPServer)

loop = asyncio.get_event_loop()

try:
    loop.run_until_complete(start_server())
except (OSError, asyncssh.Error) as exc:
    sys.exit('Error starting server: ' + str(exc))

loop.run_forever()

This sets up an SFTP listener and restricts accesses to only files within the /tmp directory. This SFTPServer subclass could also be extended to override other methods such as open, close, read, write, etc. to add whatever additional logic was needed by the application.

Full documentation is available in the the develop branch. You can also access this documentation at http://asyncssh.readthedocs.org/en/develop/api.html#sftp-support.

Comments welcome! I'll be looking to migrate this over into the master branch in another week or so if no major problems or concerns are reported.

[nchammas]

@ronf - I've been a bit behind on trying this out. I still intend to, but will need another week or two to get to it.

It's really nice to see full SFTP support coming together like this!

[ronf]

@nchammas No problem. Whenever you get a chance, let me know how it goes.

I've now updated the README and examples in the documentation to include some basic SFTP client and server information, and also simplified some of the other examples by taking advantage of recent improvements. The updated docs can be read at:

http://asyncssh.readthedocs.org/en/develop/index.html

[ronf]

This is now migrated to master and released as part of asyncssh version 1.1.0!

[nchammas]

I tried out some basic get and put operations, and they look good to me. The API is also simple, which is great.

I even got some tests setup using Hypothesis like I mentioned I wanted to try.

Check it out!

import asyncio
import asyncssh

from hypothesis import given
import hypothesis.strategies as st

PATH_PREFIX = '/some/path'


@asyncio.coroutine
def put_file(a, b):
    with (yield from asyncssh.connect('localhost')) as conn:
        with (yield from conn.start_sftp_client()) as sftp:
            yield from sftp.put(localpaths=a, remotepath=b)


def run_put_file(a, b):
    (asyncio
        .get_event_loop()
        .run_until_complete(put_file(a, b)))


@asyncio.coroutine
def get_file(a, b):
    with (yield from asyncssh.connect('localhost')) as conn:
        with (yield from conn.start_sftp_client()) as sftp:
            yield from sftp.get(remotepaths=a, localpath=b)

def run_get_file(a, b):
    (asyncio
        .get_event_loop()
        .run_until_complete(get_file(a, b)))


@given(st.binary(max_size=(2 * 1024)))
def test_put_preserves_file(f):
    before_path = PATH_PREFIX + '/before'
    after_path = PATH_PREFIX + '/after'

    with open(before_path, 'wb') as f1:
        f1.write(f)

    run_put_file(before_path, after_path)

    with open(before_path, 'rb') as f1, open(after_path, 'rb') as f2:
        assert f1.read() == f2.read()


@given(st.binary(max_size=(2 * 1024)))
def test_get_preserves_file(f):
    before_path = PATH_PREFIX + '/before'
    after_path = PATH_PREFIX + '/after'

    with open(before_path, 'wb') as f1:
        f1.write(f)

    run_get_file(before_path, after_path)

    with open(before_path, 'rb') as f1, open(after_path, 'rb') as f2:
        assert f1.read() == f2.read()

I ran these tests using pytest as follows:

python3 -m pytest asyncssh-hypothesis-tests.py

What these tests do is generate random 2 KB files and then make sure that put()-ing or get()-ing them preserves the contents of the file. You can configure how many iterations it goes through; I think it defaults to a few hundred, which took a couple of minutes on my laptop.

It's pretty neat! I found no bugs this way, which isn't surprising given how simple the test is. But perhaps this example will give you ideas for some more nefarious tests. :)

Hypothesis comes with many built-in data generators like the binary generator I used above, and you can write your own generators too.

Anyway, thumbs up from me on the basics of the SFTP client. 👍

[ronf]

Thanks for the feedback, Nicholas. If you do run into any problems, or you think of something that isn't there yet, let me know!

Also, thanks for the link to Hypothesis and your example. It looks like a great way to add some randomness to unit tests, and the idea that it'll automatically try to find the simplest input that fails is a pretty cool concept! I also really like that it seems like it can be inserted into almost any of the unit test frameworks. I'll have to look at it more closely when I start putting more of my own tests together.

There's a bit more internal code cleanup I want to do now that I've got most of the core functionality in place, but after that I expect better unit testing will be my focus for a while.