--ff option doesn't work in pytest

[peterbe]

Summary
I'm running pytest kuma/users/tests/test_views.py -x --ff and it fails on one of the tests. That's fine and expected. It fails on the 29th test. Meaning, it worked on 28 other tests. That's fine.
But when I run it a second time the --ff option should start with the test that failed the last time first. That makes it easy to re-run and re-run without having to fail for all those tests that work. But it keeps running all other tests first. Again.

Steps To Reproduce (STR)
See above. However, all I did was inject a assert 0 somewhere in kuma/users/views.py which correctly breaks at least one test.
It might be different when there's an assert 0 somewhere in a test function inside kuma/users/tests/test_views.py.

Actual behavior
See above.

Expected behavior
That it should run the test that failed first

Additional context
pytest prints information about test running when it starts. This is what I see:

kuma@825b0c877641:/app$ pytest kuma/users/tests/test_views.py -x --ff
================================================ test session starts =================================================
platform linux -- Python 3.8.0, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
django: settings: kuma.settings.testing (from ini)
rootdir: /app, inifile: pytest.ini
plugins: cov-2.8.1, django-3.8.0, requests-mock-1.7.0, metadata-1.8.0, base-url-1.4.1, rerunfailures-8.0, celery-4.4.0
collected 70 items
run-last-failure: rerun previous 1 failure first

kuma/users/tests/test_views.py ............................F

[peterbe]

To iterate, what I would expect the stdout to be is something like this:

....
kuma/users/tests/test_views.py F

and stop there.

[peterbe]

I don't know if --ff belongs to the rerunfailures plugin but it smells like it would be the plugin.

[peterbe]

CC @callahad

[callahad]

--failed-first (--ff) is a standard part of pytest. I'll take a look at this tomorrow, but I wonder if you want --stepwise (--sw), which says it will "exit on test failure and continue from last failing test next time."

So instead of -x --ff try just --sw?

[peterbe]

THanks! I'll definitely try that instead because it sounds like a match made in heaven.

[peterbe]

I tried --sw and it says "continue from last failing test next time" but no matter how many times I repeat the command, it runs the first good 14 tests over and over.

Sorry for the messy screenshot.

At least it seems to successfully save the right things to .pytest_cache

So, you'd think that the next time I run pytest kuma/users/tests/test_views.py --sw it would start with "kuma/users/tests/test_views.py::KumaGitHubTests::test_signup_username_edit_event_tracking" which is the last test to have failed.

[callahad]

@peterbe This works for me in a simpler case. I inserted an assert False after line 25 in kuma/api/tests/test_signal_handlers.py.

When I initially run with --sw I see output at the very beginning of the test run like:

collected 1972 items                                                                                                                                                                                       
stepwise: no previously failed tests, not skipping.

kuma/api/tests/test_signal_handlers.py .F

Then when I run it again, I see:

collected 1972 items / 1 deselected / 1971 selected                                                                                                                                                        
stepwise: skipping 1 already passed items.

kuma/api/tests/test_signal_handlers.py F

Note the first, passing test was successfully skipped and didn't print a dot.

My exact invocation is docker-compose exec web pytest --sw kuma

Similarly, using -x --ff I see:

collected 1972 items                                                                                                                                                                                       
run-last-failure: rerun previous 1 failure first

kuma/api/tests/test_signal_handlers.py F

[callahad]

I see the same thing you do if I put an assert False at the top of test_signup_username_edit_event_tracking in kuma/users/tests/test_views.py... so something else is interfering with the test ordering.

[callahad]

Adding -v reveals this test order:

collected 72 items                                                                                                                                                                                         
run-last-failure: rerun previous 1 failure first

kuma/users/tests/test_views.py::test_old_profile_url_gone PASSED
kuma/users/tests/test_views.py::test_user_detail_view PASSED
kuma/users/tests/test_views.py::test_my_user_page PASSED
kuma/users/tests/test_views.py::test_bug_698971 PASSED
kuma/users/tests/test_views.py::test_user_edit PASSED
kuma/users/tests/test_views.py::test_my_user_edit PASSED
kuma/users/tests/test_views.py::test_user_edit_beta PASSED
kuma/users/tests/test_views.py::test_user_edit_websites PASSED
kuma/users/tests/test_views.py::test_bug_698126_l10n PASSED
kuma/users/tests/test_views.py::test_user_edit_github_is_public PASSED
kuma/users/tests/test_views.py::test_404_logins[DOMAIN] PASSED
kuma/users/tests/test_views.py::test_404_logins[WIKI_HOST] PASSED
kuma/users/tests/test_views.py::test_404_already_logged_in[DOMAIN] PASSED
kuma/users/tests/test_views.py::test_404_already_logged_in[WIKI_HOST] PASSED
kuma/users/tests/test_views.py::test_delete_user_login_always_required PASSED
kuma/users/tests/test_views.py::test_delete_user_not_allowed PASSED
kuma/users/tests/test_views.py::test_delete_user_with_no_revisions PASSED
kuma/users/tests/test_views.py::test_delete_user_no_revisions_misc_related PASSED
kuma/users/tests/test_views.py::test_delete_user_donate_attributions PASSED
kuma/users/tests/test_views.py::test_delete_user_keep_attributions PASSED
kuma/users/tests/test_views.py::test_delete_user_no_revisions_but_attachment_revisions_donate PASSED
kuma/users/tests/test_views.py::test_send_recovery_email[good_email] PASSED
kuma/users/tests/test_views.py::test_send_recovery_email[bad_email] PASSED
kuma/users/tests/test_views.py::test_recover_valid PASSED
kuma/users/tests/test_views.py::test_invalid_token_fails PASSED
kuma/users/tests/test_views.py::test_invalid_uid_fails PASSED
kuma/users/tests/test_views.py::test_signin_landing PASSED
kuma/users/tests/test_views.py::test_signin_landing_multi_auth_disabled PASSED
kuma/users/tests/test_views.py::KumaGitHubTests::test_signup_username_edit_event_tracking FAILED
Destroying test database for alias 'default' ('test_developer_mozilla_org')...

So what do those tests have in common that our failing test does not?

[callahad]

Looks like it's running every single top-level test in that file before dropping down into the KumaGitHubTests class.

Why, I do not yet know.

[callahad]

This is due to pytest-django's re-ordering of test items.

See pytest-dev/pytest-django#214

Commenting out the pytest_collection_modifyitems function in pytest-django's plugin.py resolves this issue.

I wonder if it's possible to tell pytest in which order to evaluate plugins, so that the --ff re-ordering always happens last?

[callahad]

Decorating that function with @pytest.hookimpl(tryfirst=True) makes things work as expected. Not sure if that's the right way to do things in pytest, but it works. I'll raise an upstream issue.

[callahad]

Upstream issue at pytest-dev/pytest-django#819

[callahad]

Closing this here; it's not our issue and I don't see an obvious way to address it in our codebase, at least not from skimming the pytest docs on plugins.