Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Travis may not report failure if a test fails #4825

Closed
zjs opened this issue Jun 11, 2017 · 2 comments
Closed

Travis may not report failure if a test fails #4825

zjs opened this issue Jun 11, 2017 · 2 comments
Labels
area: code health area: testing has pr

Comments

@zjs
Copy link
Collaborator

zjs commented Jun 11, 2017

Tests in certbot-dns-digitalocean appeared to be passing on Python 3 because Travis was reporting those runs as green.

However, until #4821 one test contained calls to a method that does not exist in Python 3.

These failures were being hidden by nose errors which prevented all tests from being run (without reporting failure?!).

Example: https://travis-ci.org/certbot/certbot/jobs/241386053

Obtaining file:///home/travis/build/certbot/certbot/certbot-dns-digitalocean
Requirement already satisfied: zope.interface==4.1.3 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 28))
Requirement already satisfied: mock==2.0.0 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 29))
Requirement already satisfied: acme==0.16.0.dev0 in ./acme (from certbot-dns-digitalocean==0.16.0.dev0)
Requirement already satisfied: certbot==0.16.0.dev0 in /home/travis/build/certbot/certbot (from certbot-dns-digitalocean==0.16.0.dev0)
Collecting python-digitalocean>=1.11 (from certbot-dns-digitalocean==0.16.0.dev0)
Requirement already satisfied: setuptools>=1.0 in ./.tox/py36/lib/python3.6/site-packages (from certbot-dns-digitalocean==0.16.0.dev0)
Requirement already satisfied: six==1.10.0 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 23))
Requirement already satisfied: pbr==1.8.1 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 16))
Requirement already satisfied: cryptography==1.8.2 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 7))
Requirement already satisfied: pyOpenSSL==16.2.0 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 17))
Requirement already satisfied: pyRFC3339==1.0 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 19))
Requirement already satisfied: pytz==2015.7 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 21))
Requirement already satisfied: requests==2.12.1 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 22))
Requirement already satisfied: ConfigArgParse==0.10.0 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 5))
Requirement already satisfied: configobj==5.0.6 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 6))
Requirement already satisfied: parsedatetime==2.1 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 15))
Requirement already satisfied: zope.component==4.2.2 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 26))
Requirement already satisfied: cffi==1.4.2 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 4))
Requirement already satisfied: packaging==16.8 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 14))
Requirement already satisfied: asn1crypto==0.22.0 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 3))
Requirement already satisfied: idna==2.5 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 10))
Requirement already satisfied: zope.event==4.1.0 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 27))
Requirement already satisfied: pycparser==2.14 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 2))
Requirement already satisfied: pyparsing==2.1.8 in ./.tox/py36/lib/python3.6/site-packages (from -c /tmp/tmp.nOoN6GPb3H (line 18))
Installing collected packages: python-digitalocean, certbot-dns-digitalocean
  Running setup.py develop for certbot-dns-digitalocean
Successfully installed certbot-dns-digitalocean python-digitalocean-1.11
test_cleanup (certbot_dns_digitalocean.dns_digitalocean_test.AuthenticatorTest) ... ok
test_get_chall_pref (certbot_dns_digitalocean.dns_digitalocean_test.AuthenticatorTest) ... ok
test_more_info (certbot_dns_digitalocean.dns_digitalocean_test.AuthenticatorTest) ... ok
test_parser_arguments (certbot_dns_digitalocean.dns_digitalocean_test.AuthenticatorTest) ... ok
test_perform (certbot_dns_digitalocean.dns_digitalocean_test.AuthenticatorTest) ... ok
nose.plugins.multiprocess: ERROR: Worker 1 error running test or returning results
Traceback (most recent call last):
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 717, in __runner
    resultQueue.put((ix, test_addr, test.tasks, batch(result)))
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 689, in batch
    errors = [(TestLet(c), err) for c, err in result.errors]
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 689, in <listcomp>
    errors = [(TestLet(c), err) for c, err in result.errors]
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 165, in __init__
    self._id = case.id()
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/case.py", line 86, in id
    return self.test.id()
TypeError: 'int' object is not callable
Process Process-3:
Traceback (most recent call last):
  File "/opt/python/3.6.1/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/opt/python/3.6.1/lib/python3.6/unittest/case.py", line 601, in run
    testMethod()
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 717, in __runner
    resultQueue.put((ix, test_addr, test.tasks, batch(result)))
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 689, in batch
    errors = [(TestLet(c), err) for c, err in result.errors]
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 689, in <listcomp>
    errors = [(TestLet(c), err) for c, err in result.errors]
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 165, in __init__
    self._id = case.id()
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/case.py", line 86, in id
    return self.test.id()
TypeError: 'int' object is not callable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/python/3.6.1/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/opt/python/3.6.1/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 652, in runner
    keyboardCaught, shouldStop, loaderClass, resultClass, config)
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 749, in __runner
    resultQueue.put((ix, test_addr, test.tasks, batch(result)))
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 689, in batch
    errors = [(TestLet(c), err) for c, err in result.errors]
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 689, in <listcomp>
    errors = [(TestLet(c), err) for c, err in result.errors]
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/plugins/multiprocess.py", line 165, in __init__
    self._id = case.id()
  File "/home/travis/build/certbot/certbot/.tox/py36/lib/python3.6/site-packages/nose/case.py", line 86, in id
    return self.test.id()
TypeError: 'int' object is not callable
----------------------------------------------------------------------
Ran 5 tests in 100.307s
OK

In this case the only bug was in test code, but it seems like we should try to understand what's causing the test failures to be hidden here to make sure real bugs don't slip in undetected in the future.
@zjs zjs mentioned this issue Jun 11, 2017
Merged
@bmw
Copy link
Member

bmw commented Jun 12, 2017

We've unfortunately seen this before (see #4050). I added https://github.com/certbot/certbot/blob/master/tests/letstest/scripts/test_sdists.sh to catch the problem in the future, but that test doesn't run on Python 3 which we didn't support when that test is written. To solve this we should either:

  1. Stop using nose and switch to py.test or plain unittest. nose is unmaintained and we have other problems with the package (see Bug running unit tests in parallel on Python 2.6 #4377).
  2. Update test_sdists.sh linked above to also test on Python 3. This can only be done on some of the tests in the test farm as not all systems used have Python 3.3+ available.

Ideally, we should probably do both.

@bmw bmw mentioned this issue Nov 29, 2017
Merged
@bmw bmw added the has pr label Nov 29, 2017
@bmw
Copy link
Member

bmw commented Dec 1, 2017

Fixed in #5282.

@bmw bmw closed this as completed Dec 1, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: code health area: testing has pr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zjs @bmw