Flaky tests in openedx/core/lib/tests/test_jwt.py tests with jwt.exceptions.ExpiredSignatureError

[robrap]

There are some flaky tests related to our JWTs (i.e. passed on re-run) in edx-platform.

• Here's a link the failing log.
• Because this is JWT related, I'm hesitant to follow the ~/Flaky+Test+Process and remove these until they are fixed. So, for now this ticket is for fixing the issue.
• There is no guarantee that the code itself isn't flaky, and not the test. But typically that is not the case.
• This was seen at least 3 times. Once above, and @wgu-taylor-payne mentioned seeing it a couple of times.

Other notes:

• Does test order matter, and is order random or fixed in CI right now?

Here is the list of failures:

FAILED openedx/core/lib/tests/test_jwt.py::TestSign::test_create_jwt - jwt.exceptions.ExpiredSignatureError: Signature has expired
FAILED openedx/core/lib/tests/test_jwt.py::TestSign::test_create_jwt_with_claims - jwt.exceptions.ExpiredSignatureError: Signature has expired
FAILED openedx/core/lib/tests/test_jwt.py::TestUnpack::test_missing_expired_lms_user_id - jwt.exceptions.ExpiredSignatureError: Signature has expired
FAILED openedx/core/lib/tests/test_jwt.py::TestUnpack::test_unpack_jwt - jwt.exceptions.ExpiredSignatureError: Signature has expired
FAILED openedx/core/lib/tests/test_jwt.py::TestUnpack::test_unpack_jwt_with_claims - jwt.exceptions.ExpiredSignatureError: Signature has expired
FAILED openedx/core/lib/tests/test_jwt.py::TestUnpack::test_unpack_token_with_invalid_user - jwt.exceptions.ExpiredSignatureError: Signature has expired

Here is an example failure:

___________________________ TestSign.test_create_jwt ___________________________

self = <openedx.core.lib.tests.test_jwt.TestSign testMethod=test_create_jwt>

    def test_create_jwt(self):
        token = create_jwt(test_user_id, test_timeout, {}, test_now)
    
>       decoded = unpack_and_verify(token)

openedx/core/lib/tests/test_jwt.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
openedx/core/lib/jwt.py:89: in unpack_and_verify
    decoded = jwt.decode(
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/jwt/api_jwt.py:365: in decode
    decoded = self.decode_complete(
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/jwt/api_jwt.py:272: in decode_complete
    self._validate_claims(
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/jwt/api_jwt.py:405: in _validate_claims
    self._validate_exp(payload, now, leeway)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <jwt.api_jwt.PyJWT object at 0x7f9ec75c7440>
payload = {'exp': 1776957758, 'iat': 1776956758, 'iss': 'token-test-issuer', 'lms_user_id': 121, ...}
now = 1776957784.86637, leeway = 0

    def _validate_exp(
        self,
        payload: dict[str, Any],
        now: float,
        leeway: float,
    ) -> None:
        try:
            exp = int(payload["exp"])
        except ValueError:
            raise DecodeError(
                "Expiration Time claim (exp) must be an integer."
            ) from None
    
        if exp <= (now - leeway):
>           raise ExpiredSignatureError("Signature has expired")
E           jwt.exceptions.ExpiredSignatureError: Signature has expired

/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/jwt/api_jwt.py:508: ExpiredSignatureError

[snakefood3232]

It seems the flaky tests are related to ExpiredSignatureError in openedx/core/lib/tests/test_jwt.py. I'd handle this by reviewing the logic around JWT expiration and ensuring test fixtures use a consistent and controlled time, maybe using mocking for time-sensitive operations. Can have a PR addressing this in two days. I've built similar JWT handling features and fixed related test flakiness in the past, so this aligns well with my experience.

[feanil]

@snakefood3232 feel free to take a look and make a PR if you think you have a solution.

[wgu-taylor-payne]

@wgu-jesse-stewart created a fix for this in #38464. Capturing the time in each test method rather than at import time.

This could be one way to solve the issue, if we don't go the mock route.

[robrap]

@wgu-taylor-payne: What is the mock route? Would freeze_time be appropriate, and is that what you meant?

[wgu-taylor-payne]

@robrap Yes, freeze_time is what I had in mind. There are three approaches worth comparing:

1. Current fix in #38464 (per-test time()) — calls get_test_now() (which returns int(time())) at the start of each test method instead of capturing it once at module import. Eliminates the flakiness window between import and execution, but the tests still use the real clock — test_now is a different value every run.

2. freeze_time — freezes time.time(), datetime.now(), and other time sources to a fixed value for the duration of the test. The token's iat, exp, and the verification clock all agree deterministically, every run. expected_full_token could go back to being a plain constant. Already used extensively across the codebase.

3. unittest.mock.patch — patches time directly without an extra dependency: @patch('openedx.core.lib.tests.test_jwt.time', return_value=1736942400). Same determinism as freeze_time, but only freezes the specific time() call you patch — it won't catch other time sources (datetime.now(), etc.) if they're used downstream. More surgical, less ergonomic.

All three fix the flakiness. The practical difference is reproducibility: approaches 2 and 3 produce the exact same token on every run, making failures easier to debug. Between those two, freeze_time is broader and more readable; mock.patch is lighter if you only need to control time.time(). The per-test time() approach (1) is correct but non-deterministic.

Either of the deterministic routes is a bit cleaner, but 1 is sufficient if we just want the flakiness gone.

[robrap]

Thanks @wgu-taylor-payne. I appreciate your response.

Personally, I'd prefer non-flaky to flaky, and I'm not planning on working on this, so if we need to choose between option 1 and flaky tests, let's go for option 1. If you want to look into 2 or 3, I wouldn't stop you, but I don't think it is blocking.

[wgu-taylor-payne]

@snakefood3232 I had some time today to implement something so put up a PR for this, FYI. Thanks for being willing to help out though.