failed tests

[carlwgeorge]

I co-maintain the python-jwt package in Fedora for PyJWT. The test suite is now failing in Fedora 30 and I'm not sure why. The failing tests are:

• TestJWS.test_decode_no_algorithms_verify_signature_false
• TestJWT.test_decode_no_algorithms_verify_false

Expected Result

Tests to pass (just like they did in Fedora 29).

Actual Result

_______________________________________ TestJWS.test_decode_no_algorithms_verify_signature_false _______________________________________

self = <tests.test_api_jws.TestJWS object at 0x7f03249037b8>, jws = <jwt.api_jws.PyJWS object at 0x7f0324903128>

    def test_decode_no_algorithms_verify_signature_false(self, jws):
        example_secret = 'secret'
        example_jws = (
            b'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.'
            b'aGVsbG8gd29ybGQ.'
            b'SIr03zM64awWRdPrAM_61QWsZchAtgDV3pphfHPPWkI'
        )
    
        try:
            pytest.deprecated_call(
                jws.decode, example_jws, key=example_secret,
>               options={'verify_signature': False},
            )
E           Failed: DID NOT WARN. No warnings of type (<class 'DeprecationWarning'>, <class 'PendingDeprecationWarning'>) was emitted. The list of emitted warnings is: [].

tests/test_api_jws.py:295: Failed
____________________________________________ TestJWT.test_decode_no_algorithms_verify_false ____________________________________________

self = <tests.test_api_jwt.TestJWT object at 0x7f0324d987f0>, jwt = <jwt.api_jwt.PyJWT object at 0x7f0324d8c748>
payload = {'claim': 'insanity', 'exp': 1541389632, 'iss': 'jeff'}

    def test_decode_no_algorithms_verify_false(self, jwt, payload):
        secret = 'secret'
        jwt_message = jwt.encode(payload, secret)
    
        try:
            pytest.deprecated_call(
>               jwt.decode, jwt_message, secret, verify=False,
            )
E           Failed: DID NOT WARN. No warnings of type (<class 'DeprecationWarning'>, <class 'PendingDeprecationWarning'>) was emitted. The list of emitted warnings is: [].

tests/test_api_jwt.py:517: Failed

Full output: https://gist.github.com/carlwgeorge/563597c80e3324101d754efac99d14a1

Reproduction Steps

yum install python3-{devel,setuptools,cryptography,pytest,pytest-cov}
wget https://files.pythonhosted.org/packages/source/P/PyJWT/PyJWT-1.6.4.tar.gz
tar -xf PyJWT-1.6.4.tar.gz
cd PyJWT-1.6.4
pytest-3 -v

System Information

$ python -m jwt.help

$ python3 -m jwt.help
{
  "cryptography": {
    "version": "2.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.1"
  },
  "platform": {
    "release": "4.18.16-300.fc29.x86_64",
    "system": "Linux"
  },
  "pyjwt": {
    "version": "1.6.4"
  }
}

[jpadilla]

I recently ran into something similar so I pinned test dependencies. b616107

[carlwgeorge]

I went through and ran the tests with various pytest versions from 3.6.4 (the version in Fedora 29) and 3.10.0 (latest). They pass fine up until 3.7.4, then it gets messy.

• 3.6.4 through 3.7.4: 173 passed, 14 skipped, 1 xfailed
• 3.8.0 through 3.8.2: 173 passed, 14 skipped, 1 xfailed, 143 warnings
• 3.9.1 through 3.10.0: 2 failed, 171 passed, 14 skipped, 1 xfailed, 143 warnings

[glaubitz]

We're using the patch from NixOS/nixpkgs#49602 in openSUSE now and it fixes the testsuite.

[jpadilla]

Aiming to release a new update before the week ends.

[jpadilla]

Released v1.7.0 last week.

[dotlambda]

It's said to see you're viewing pinning the pytest version as a solution to this.

[jpadilla]

@dotlambda I can't do anything about it with a comment like that though.

What would you recommend we should do exactly? Stop pinning? Pin a specified range so it doesn't happen again? Add a compat layer to handle various versions of a test dependency?

[glaubitz]

@jpadilla I would suggest using the patch from NixOS/nixpkgs#49602.

Pinning users to an old version is a bad idea, in particular when users are getting their Python packages through their Linux distribution.

[jpadilla]

@glaubitz That I can do, thanks. Wouldn't not pinning(at least a range) mean it's more likely for tests to break like they did at a later point? Current pytest version is now 4.0.1

[glaubitz]

Pinning is fine as long as you're not pinning to a fixed version and you're depending on a reasonably new version which distributions would ship in their unstable or rolling release distributions. In this case, the version you pinned on was relatively old looking at the number of releases pytest made since 3.7.0 was released. So, it's expected for Linux vendors to package versions later than 3.7.0 in their unstable or rolling release distributions.

It's reasonable to pin using a range if you expect the API to break after a certain version number of your dependency. So, you can definitely use a range. Just don't depend on something old.

[jpadilla]

@glaubitz thanks for the thorough explanation! Here's what I did: b65e1ac

[jpadilla]

https://github.com/jpadilla/pyjwt/releases/tag/1.7.1