Issues with library testing

[smarek]

Environment OS X, x64, 10.9.4 (not relevant probably)
Python version 2.7.8

1. Missing documentation on how to run tests. Should we use run.sh or run-separate.sh ?
2. python setup.py test doesn't react, should execute given tests after in-place build
3. Running ./run-separate.sh gives no output on tests success/failure, example output:

> ./run-separate.sh 
Starting: test_basic.py
Starting: test_bip32_speed.py
Starting: test_debuglink.py
Starting: test_msg_applysettings.py
Starting: test_msg_changepin.py
Starting: test_msg_cipherkeyvalue.py
Starting: test_msg_clearsession.py
Starting: test_msg_decryptmessage.py
Starting: test_msg_encryptmessage.py
Starting: test_msg_estimatetxsize.py
Starting: test_msg_getaddress.py
Starting: test_msg_getentropy.py
Starting: test_msg_getpublickey.py
Starting: test_msg_loaddevice.py
Starting: test_msg_ping.py
Starting: test_msg_recoverydevice.py
Starting: test_msg_resetdevice.py
Starting: test_msg_signmessage.py
Starting: test_msg_signtx.py
Starting: test_msg_simplesigntx.py
Starting: test_msg_verifymessage.py
Starting: test_msg_wipedevice.py
Starting: test_protect_call.py
Starting: test_protection_levels.py
Starting: test_zerosig.py

4. Running ./run.sh in tests folder, gives 99 errors like this:

======================================================================
ERROR: test_two_zero_signature (test_zerosig.TestZeroSig)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/{REPOSITORY_PATH}/python-trezor/tests/common.py", line 9, in setUp
    self.debug_transport = config.DEBUG_TRANSPORT(*config.DEBUG_TRANSPORT_ARGS, **config.DEBUG_TRANSPORT_KWARGS)
  File "../trezorlib/transport_socket.py", line 16, in __init__
    super(SocketTransportClient, self).__init__(device, *args, **kwargs)
  File "../trezorlib/transport.py", line 14, in __init__
    self._open()
  File "../trezorlib/transport_socket.py", line 20, in _open
    self.socket.connect(self.device)
  File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
gaierror: [Errno 8] nodename nor servname provided, or not known

----------------------------------------------------------------------

[prusnak]

tests directory contains unit test for TREZOR device so they should not be a part of setup.py

Also you see errors because you have no TREZOR device connected.

[smarek]

Yes, but they should respond to python setup.py test, see https://pythonhosted.org/an_example_pypi_project/setuptools.html#using-setup-py

It was connected, so that means that tests fail on OSX as of now

run-separate.sh shouldn't return any OK/FAIL status for each test?

[prusnak]

Do you have hidapi and mnemonic python packages installed? Is hidapi able to open the device (permissions)?

run-separate outputs into $testname.out and $testname.err files.

[slush0]

@prusnak tests require debuglink, correct? That said, nothing in /tests/ will run against stock firmware...

[smarek]

@prusnak I have, setup.py installed them as dependencies and install succeeded.
Don't know about permissions, can you add some script to check device connectivity and/or file (or other) permissions, so I can tell you debug info?

I have lots of out/err files, want any specific ones?

@slush0 if that's correct, it should be mentioned that tests are not to test device accesibility, state and state of library against platform, probably remove the tests from public repository and replace them with tests that user/developer can run against stock firmware?

[prusnak]

Does running the script as root work? If yes then my guess would be that you need to do something similar like udev rules on Linux (https://github.com/trezor/trezord/blob/master/release/linux/trezor.rules for inspiration).

No unit tests can be run on stock firmware, because they need to know internal data of TREZOR that stock firmware never shares for obvious reasons.