VCR shouldn't record 500, 408, 429, when running challenges

[waynehamadi]

Duplicates

• I have searched the existing issues

Summary 💡

Reminder. VCR is used to record HTTP interactions, so that when the same HTTP request is about to be made, we use VCR instead.

This makes tests cheaper and faster, without much drawbacks, except maintaining VCR ;-)

Currently VCR saves the 500, 408, 429.
This means if we get rate limited while recording a challenge, it will behave like that the next time.

TODO: there is a challenge decorator, find a way to change VCR's behavior so that it stops recording anything that's not a 200, if a function uses this decorator.

Not to do: do not ignore all non 200 error in VCR, some of our integration tests need the 500 errors.

Examples 🌈

No response

Motivation 🔦

No response

[erik-megarad]

I opened a PR, #4469, which adds the change.

Unfortunately, due to the way that request filtering in pytest-vcr works we don't have access to the pytest context so we don't know if the challenge marker is set. Instead, this applies filtering to all tests in the challenges module. I think this is the only way to do it.

[waynehamadi]

@erik-megarad Great thank you!