Tests fail

[djmitche]

• Django Remote Submission version: master
• Django version: latest
• Python version: 3.5
• Operating System: Ubuntu 16.04

Description

Running the tests fails. I'm sure something in my SSH setup is missing, so this is probably going to come down to a doc update as to how to set up what i didn't set up!

What I Did

I started a standard Ubuntu openssh-server install, with a user 'worker' and a password. No ssh keys are set up (this is in a clean docker container). My .env is:

TEST_PYTHON_ARGUMENTS=-u
TEST_PYTHON_PATH=/usr/bin/python3
TEST_REMOTE_DIRECTORY=/home/worker
TEST_REMOTE_FILENAME=foo.py
TEST_REMOTE_PASSWORD=foo#123
TEST_REMOTE_USER=worker
TEST_SERVER_HOSTNAME=127.0.0.1
TEST_SERVER_PORT=22

Result summary:

tests/test_models.py::test_server_string_representation PASSED
tests/test_models.py::test_job_string_representation PASSED
tests/test_models.py::test_log_string_representation PASSED
tests/test_models.py::test_result_string_representation PASSED
tests/test_tasks.py::test_submit_job_normal_usage[True] PASSED
tests/test_tasks.py::test_submit_job_normal_usage[False] PASSED
tests/test_tasks.py::test_submit_job_multiple_streams[True] PASSED
tests/test_tasks.py::test_submit_job_multiple_streams[False] PASSED
tests/test_tasks.py::test_submit_job_failure[True] PASSED
tests/test_tasks.py::test_submit_job_failure[False] PASSED
tests/test_tasks.py::test_submit_job_log_policy_log_total[True] PASSED
tests/test_tasks.py::test_submit_job_log_policy_log_total[False] PASSED
tests/test_tasks.py::test_submit_job_log_policy_log_none[True] PASSED
tests/test_tasks.py::test_submit_job_log_policy_log_none[False] PASSED
tests/test_tasks.py::test_submit_job_timeout[True] PASSED
tests/test_tasks.py::test_submit_job_timeout[False] PASSED
tests/test_tasks.py::test_submit_job_modified_files[True] PASSED
tests/test_tasks.py::test_submit_job_modified_files[False] FAILED
tests/test_tasks.py::test_submit_job_modified_files_positive_pattern[True] PASSED
tests/test_tasks.py::test_submit_job_modified_files_positive_pattern[False] FAILED
tests/test_tasks.py::test_submit_job_modified_files_negative_pattern[True] PASSED
tests/test_tasks.py::test_submit_job_modified_files_negative_pattern[False] FAILED
tests/test_tasks.py::test_submit_job_deploy_key[True] FAILED
tests/test_tasks.py::test_submit_job_deploy_key[False] FAILED
tests/test_tasks.py::test_delete_key_old_way FAILED
tests/test_tasks.py::test_deploy_and_delete_key FAILED

And the first exception:

========================================================================================================================== FAILURES ===========================================================================================================================
____________________________________________________________________________________________________________ test_submit_job_modified_files[False] ____________________________________________________________________________________________________________

env = Env(server_hostname='127.0.0.1', server_port=22, remote_directory='/home/worker', remote_filename='foo.py', remote_user='worker', remote_password='******', python_path='/usr/bin/python3', python_arguments=['-u']), job = <Job: 1-job-title>
wrapper_cls = <class 'django_remote_submission.remote.RemoteWrapper'>

    @pytest.mark.django_db
    @pytest.mark.job_program('''\
    from __future__ import print_function
    import time
    import sys
    for i in range(5):
        with open('{}.txt'.format(i), 'w') as f:
            print('line: {}'.format(i), file=f)
        time.sleep(0.1)
    ''')
    def test_submit_job_modified_files(env, job, wrapper_cls):
        from django_remote_submission.models import Job, Log, Result
        from django_remote_submission.tasks import submit_job_to_server, LogPolicy
        import re
    
        results = submit_job_to_server(job.pk, env.remote_password,
                                       wrapper_cls=wrapper_cls)
    
>       assert len(results) == 5
E       assert 0 == 5
E        +  where 0 = len({})

tests/test_tasks.py:423: AssertionError
-------------------------------------------------------------------------------------------------------------------- Captured stdout setup --------------------------------------------------------------------------------------------------------------------
DEBUG:environ.environ:Read environment variables from: /django-remote-submission/.env
DEBUG:environ.environ:get 'TEST_SERVER_HOSTNAME' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_SERVER_PORT' casted as '<class 'int'>' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_REMOTE_DIRECTORY' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_REMOTE_FILENAME' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_REMOTE_USER' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_REMOTE_PASSWORD' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_PYTHON_PATH' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_PYTHON_ARGUMENTS' casted as '<class 'list'>' with default '<NoValue>'
-------------------------------------------------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------------------------------------------------
INFO:django_remote_submission.remote:Connecting user worker to 127.0.0.1 with password.
DEBUG:paramiko.transport:starting thread (client mode): 0x94d5a0f0
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.2.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_7.2p2)
DEBUG:paramiko.transport:kex algos:['curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'rsa-sha2-512', 'rsa-sha2-256', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['chacha20-poly1305@openssh.com', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] server encrypt:['chacha20-poly1305@openssh.com', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] client mac:['umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: ecdh-sha2-nistp256
DEBUG:paramiko.transport:HostKey agreed: ssh-ed25519
DEBUG:paramiko.transport:Cipher agreed: aes128-ctr
DEBUG:paramiko.transport:MAC agreed: hmac-sha2-256
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Adding ssh-ed25519 host key for 127.0.0.1: b'c629794f404e0cb071153c17cd672cc4'
DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Authentication (password) successful!
DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes
DEBUG:paramiko.transport:Received global request "hostkeys-00@openssh.com"
DEBUG:paramiko.transport:Rejecting "hostkeys-00@openssh.com" global request from server.
DEBUG:paramiko.transport:[chan 0] Max packet out: 32768 bytes
DEBUG:paramiko.transport:Secsh channel 0 opened.
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
INFO:paramiko.transport.sftp:[chan 0] Opened sftp connection (server version 3)
DEBUG:paramiko.transport.sftp:[chan 0] stat(b'/home/worker')
DEBUG:paramiko.transport.sftp:[chan 0] normalize(b'/home/worker')
DEBUG:paramiko.transport.sftp:[chan 0] open(b'/home/worker/foo.py', 'wt')
DEBUG:paramiko.transport.sftp:[chan 0] open(b'/home/worker/foo.py', 'wt') -> b'00000000'
DEBUG:paramiko.transport.sftp:[chan 0] close(b'00000000')
exec_command(command='cd /home/worker && /usr/bin/python3 -u foo.py')
DEBUG:paramiko.transport:[chan 1] Max packet in: 32768 bytes
DEBUG:paramiko.transport:[chan 1] Max packet out: 32768 bytes
DEBUG:paramiko.transport:Secsh channel 1 opened.
DEBUG:paramiko.transport:[chan 1] Sesch channel 1 request ok
DEBUG:paramiko.transport:[chan 1] EOF received (1)
DEBUG:paramiko.transport.sftp:[chan 0] listdir(b'/home/worker/.')
DEBUG:paramiko.transport:[chan 1] EOF sent (1)
<SFTPAttributes: [ size=34 uid=1000 gid=1000 mode=0o40700 atime=1502843554 mtime=1502843554 ]>
<SFTPAttributes: [ size=189 uid=1000 gid=1000 mode=0o100664 atime=1502845059 mtime=1502845058 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844437 mtime=1502844433 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844437 mtime=1502844434 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844437 mtime=1502844435 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844438 mtime=1502844436 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844438 mtime=1502844437 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
INFO:paramiko.transport.sftp:[chan 0] sftp session closed.
DEBUG:paramiko.transport:[chan 0] EOF sent (0)
-------------------------------------------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------------------------------------------
INFO :: Connecting user worker to 127.0.0.1 with password.