What is the preferred way for editors/tools to interact with pytest?

[brettcannon]

We are just starting floating the idea around of rewriting out testing support code for the Python extension for VS Code. As such that means making we hook into pytest appropriately.

So my question is what does "appropriate" hooking look like to the pytest team? I.e. what is the preferred mechanism for test discovery (e.g. populate a test explorer), and to execute a test(s) (e.g. when a user runs a test from a test explorer)? And for whatever the answer is, what versions of pytest would be supported (I'm honestly more concerned to know this to tell people what they would need to upgrade to instead of requiring something like Python 2 support)?

[RonnyPfannschmidt]

Here a sparse version, i hope i find time to elaborate later

An ide probably best integrates into pytest by having a pythonpath injected starter library that runs pytest.main with a own plugin object added

This in turn can take actual full control over whatever pytest does

I strongly recommend keeping us in the loop about apis /internals used

I would recommend reasonably recent pytest

But i suspect for part of the use cases it can easily work with 4.x

I strongly recommend 6.x as starting point and choose lower versions only if absolutely required

[brettcannon]

I'm assuming the node ID is what you're passing back into the main() call (e.g. "test.py::Test::test_method")?

[brettcannon]

And does -pno:terminal turn off all terminal-related plug-ins and their output?

[nicoddemus]

I'm assuming the node ID is what you're passing back into the main() call (e.g. "test.py::Test::test_method")?

Yes, you can see that in the second pytest.main call.

And does -pno:terminal turn off all terminal-related plug-ins and their output?

Exactly. 👍

[brettcannon]

I just noticed that the execution example of pytest.main(["--collect-only", "-pno:terminal", "test.py::Test::test_method"], plugins=[MyPlugin()]) still specifies --collect-only even though it's for test execution. Is that on purpose or a copy-and-paste mistake?

[nicoddemus]

Copy-and-paste mistake, sorry, I've fixed the example.

[Lothiraldan]

Hello, sorry for intruding in the discussion but I have been working on something that could fit your need: https://github.com/Lothiraldan/litf. It is meant to be something similar to https://microsoft.github.io/language-server-protocol/. I already have a pytest plugin for it that seems to work so far but likely requires more work: https://github.com/Lothiraldan/pytest-litf

For example if you install pytest and pytest-litf (pip install pytest pytest-litf) and run the following commands inside the directory https://github.com/Lothiraldan/pytest-litf/tree/master/example_dir, you will get:

$ pytest-litf '{"collect-only": true}}'

pytest-litf '{"collect-only": true}'
collecting ... 
{"_type": "litf_start", "litf_version": "0.0.1"}
{"_type": "session_start", "test_number": 32}
{"_type": "test_collection", "line": 8, "file": "test_class.py", "test_name": "TestClassPassing.test_passing", "id": "test_class.py::TestClassPassing::()::test_passing"}
{"_type": "test_collection", "line": 19, "file": "test_class.py", "test_name": "TestClassFailing.test_failing", "id": "test_class.py::TestClassFailing::()::test_failing"}
{"_type": "test_collection", "line": 30, "file": "test_class.py", "test_name": "TestClassError.test_error", "id": "test_class.py::TestClassError::()::test_error"}
{"_type": "test_collection", "line": 41, "file": "test_class.py", "test_name": "TestClassFailingAndErrorTeardown.test_error", "id": "test_class.py::TestClassFailingAndErrorTeardown::()::test_error"}
{"_type": "test_collection", "line": 52, "file": "test_class.py", "test_name": "TestClassErrorSetup.test_passing", "id": "test_class.py::TestClassErrorSetup::()::test_passing"}
{"_type": "test_collection", "line": 63, "file": "test_class.py", "test_name": "TestClassErrorSetupAndTeardown.test_passing", "id": "test_class.py::TestClassErrorSetupAndTeardown::()::test_passing"}
{"_type": "test_collection", "line": 74, "file": "test_class.py", "test_name": "TestClassErrorTeardown.test_passing", "id": "test_class.py::TestClassErrorTeardown::()::test_passing"}
{"_type": "test_collection", "line": 4, "file": "test_func.py", "test_name": "test_success", "id": "test_func.py::test_success"}
{"_type": "test_collection", "line": 8, "file": "test_func.py", "test_name": "test_fails", "id": "test_func.py::test_fails"}
{"_type": "test_collection", "line": 12, "file": "test_func.py", "test_name": "test_fixtures[0]", "id": "test_func.py::test_fixtures[0]"}
{"_type": "test_collection", "line": 12, "file": "test_func.py", "test_name": "test_fixtures[1]", "id": "test_func.py::test_fixtures[1]"}
{"_type": "test_collection", "line": 12, "file": "test_func.py", "test_name": "test_fixtures[2]", "id": "test_func.py::test_fixtures[2]"}
{"_type": "test_collection", "line": 17, "file": "test_func.py", "test_name": "test_error", "id": "test_func.py::test_error"}
{"_type": "test_collection", "line": 9, "file": "test_module_setup_teardown.py", "test_name": "test_passing", "id": "test_module_setup_teardown.py::test_passing"}
{"_type": "test_collection", "line": 13, "file": "test_module_setup_teardown.py", "test_name": "test_failing", "id": "test_module_setup_teardown.py::test_failing"}
{"_type": "test_collection", "line": 24, "file": "test_module_setup_teardown.py", "test_name": "TestClassPassing.test_passing", "id": "test_module_setup_teardown.py::TestClassPassing::()::test_passing"}
{"_type": "test_collection", "line": 35, "file": "test_module_setup_teardown.py", "test_name": "TestClassFailing.test_failing", "id": "test_module_setup_teardown.py::TestClassFailing::()::test_failing"}
{"_type": "test_collection", "line": 4, "file": "test_skip.py", "test_name": "test_skip_function", "id": "test_skip.py::test_skip_function"}
{"_type": "test_collection", "line": 10, "file": "test_skip.py", "test_name": "TestSkipCall.test_skip_method", "id": "test_skip.py::TestSkipCall::()::test_skip_method"}
{"_type": "test_collection", "line": 17, "file": "test_skip.py", "test_name": "TestSkipClass.test_skipped_1", "id": "test_skip.py::TestSkipClass::()::test_skipped_1"}
{"_type": "test_collection", "line": 20, "file": "test_skip.py", "test_name": "TestSkipClass.test_skipped_2", "id": "test_skip.py::TestSkipClass::()::test_skipped_2"}
{"_type": "test_collection", "line": 4, "file": "test_slow.py", "test_name": "test_slow_passing", "id": "test_slow.py::test_slow_passing"}
{"_type": "test_collection", "line": 4, "file": "test_std.py", "test_name": "test_stdout", "id": "test_std.py::test_stdout"}
{"_type": "test_collection", "line": 8, "file": "test_std.py", "test_name": "test_stderr", "id": "test_std.py::test_stderr"}
{"_type": "test_collection", "line": 19, "file": "test_std.py", "test_name": "TestClassStdout.test_stdout", "id": "test_std.py::TestClassStdout::()::test_stdout"}
{"_type": "test_collection", "line": 30, "file": "test_std.py", "test_name": "TestClassStdoutSetup.test_stdout", "id": "test_std.py::TestClassStdoutSetup::()::test_stdout"}
{"_type": "test_collection", "line": 41, "file": "test_std.py", "test_name": "TestClassStdoutAllPhases.test_stdout", "id": "test_std.py::TestClassStdoutAllPhases::()::test_stdout"}
{"_type": "test_collection", "line": 52, "file": "test_std.py", "test_name": "TestClassFailing.test_stderr", "id": "test_std.py::TestClassFailing::()::test_stderr"}
{"_type": "test_collection", "line": 8, "file": "test_unittest.py", "test_name": "TestStringMethods.test_isupper", "id": "test_unittest.py::TestStringMethods::test_isupper"}
{"_type": "test_collection", "line": 5, "file": "test_unittest.py", "test_name": "TestStringMethods.test_upper", "id": "test_unittest.py::TestStringMethods::test_upper"}
{"_type": "test_collection", "line": 1, "file": "directory/test_file_2.py", "test_name": "test_success", "id": "directory/test_file_2.py::test_success"}
{"_type": "test_collection", "line": 1, "file": "directory/test_func.py", "test_name": "test_success", "id": "directory/test_func.py::test_success"}
{"_type": "session_end", "total_duration": 0.06476974487304688, "passed": 0, "failed": 0, "error": 0, "skipped": 0}

And without the collect only:

pytest-litf '{}'
collecting ... 
{"_type": "litf_start", "litf_version": "0.0.1"}
{"_type": "session_start", "test_number": 32}
{"_type": "test_result", "file": "test_class.py", "line": 8, "test_name": "TestClassPassing.test_passing", "duration": 0.002009868621826172, "durations": {"setup": 0.0008170604705810547, "call": 0.000560760498046875, "teardown": 0.0006320476531982422}, "outcome": "passed", "id": "test_class.py::TestClassPassing::()::test_passing", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_class.py", "line": 19, "test_name": "TestClassFailing.test_failing", "duration": 0.0017082691192626953, "durations": {"setup": 0.0007331371307373047, "call": 0.0007903575897216797, "teardown": 0.00018477439880371094}, "outcome": "failed", "id": "test_class.py::TestClassFailing::()::test_failing", "stdout": "", "stderr": "", "error": {"humanrepr": "self = <test_class.TestClassFailing object at 0x7f3a8a968610>\n\n    def test_failing(self):\n>       assert False\nE       assert False\n\ntest_class.py:20: AssertionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_class.py", "line": 30, "test_name": "TestClassError.test_error", "duration": 0.00043463706970214844, "durations": {"setup": 0.00016069412231445312, "call": 0.00013875961303710938, "teardown": 0.00013518333435058594}, "outcome": "failed", "id": "test_class.py::TestClassError::()::test_error", "stdout": "", "stderr": "", "error": {"humanrepr": "self = <test_class.TestClassError object at 0x7f3a8a900910>\n\n    def test_error(self):\n>       1 / 0\nE       ZeroDivisionError: division by zero\n\ntest_class.py:31: ZeroDivisionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_class.py", "line": 41, "test_name": "TestClassFailingAndErrorTeardown.test_error", "duration": 0.0007061958312988281, "durations": {"setup": 0.0001914501190185547, "call": 0.0002789497375488281, "teardown": 0.0002357959747314453}, "outcome": "failed", "id": "test_class.py::TestClassFailingAndErrorTeardown::()::test_error", "stdout": "", "stderr": "", "error": {"humanrepr": "self = <test_class.TestClassFailingAndErrorTeardown object at 0x7f3a8a900f50>\n\n    def test_error(self):\n>       assert False\nE       assert False\n\ntest_class.py:42: AssertionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_class.py", "line": 52, "test_name": "TestClassErrorSetup.test_passing", "duration": 0.0002980232238769531, "durations": {"setup": 0.0001780986785888672, "teardown": 0.00011992454528808594}, "outcome": "failed", "id": "test_class.py::TestClassErrorSetup::()::test_passing", "stdout": "", "stderr": "", "error": {"humanrepr": "self = <test_class.TestClassErrorSetup object at 0x7f3a8a900750>, method = <bound method TestClassErrorSetup.test_passing of <test_class.TestClassErrorSetup object at 0x7f3a8a900750>>\n\n    def setup_method(self, method):\n>       1 / 0\nE       ZeroDivisionError: division by zero\n\ntest_class.py:47: ZeroDivisionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_class.py", "line": 63, "test_name": "TestClassErrorSetupAndTeardown.test_passing", "duration": 0.00026154518127441406, "durations": {"setup": 0.00014472007751464844, "teardown": 0.00011682510375976562}, "outcome": "failed", "id": "test_class.py::TestClassErrorSetupAndTeardown::()::test_passing", "stdout": "", "stderr": "", "error": {"humanrepr": "self = <test_class.TestClassErrorSetupAndTeardown object at 0x7f3a8a92eb10>, method = <bound method TestClassErrorSetupAndTeardown.test_passing of <test_class.TestClassErrorSetupAndTeardown object at 0x7f3a8a92eb10>>\n\n    def setup_method(self, method):\n>       1 / 0\nE       ZeroDivisionError: division by zero\n\ntest_class.py:58: ZeroDivisionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_class.py", "line": 74, "test_name": "TestClassErrorTeardown.test_passing", "duration": 0.0003600120544433594, "durations": {"setup": 0.00013017654418945312, "call": 0.0001087188720703125, "teardown": 0.00012111663818359375}, "outcome": "failed", "id": "test_class.py::TestClassErrorTeardown::()::test_passing", "stdout": "", "stderr": "", "error": {"humanrepr": "self = <test_class.TestClassErrorTeardown object at 0x7f3a8a921e10>, method = <bound method TestClassErrorTeardown.test_passing of <test_class.TestClassErrorTeardown object at 0x7f3a8a921e10>>\n\n    def teardown_method(self, method):\n>       1 / 0\nE       ZeroDivisionError: division by zero\n\ntest_class.py:72: ZeroDivisionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_func.py", "line": 4, "test_name": "test_success", "duration": 0.0003304481506347656, "durations": {"setup": 0.00012350082397460938, "call": 0.00011157989501953125, "teardown": 9.5367431640625e-05}, "outcome": "passed", "id": "test_func.py::test_success", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_func.py", "line": 8, "test_name": "test_fails", "duration": 0.0003731250762939453, "durations": {"setup": 0.000118255615234375, "call": 0.0001380443572998047, "teardown": 0.00011682510375976562}, "outcome": "failed", "id": "test_func.py::test_fails", "stdout": "", "stderr": "", "error": {"humanrepr": "def test_fails():\n>       assert False\nE       assert False\n\ntest_func.py:9: AssertionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_func.py", "line": 12, "test_name": "test_fixtures[0]", "duration": 0.00039124488830566406, "durations": {"setup": 0.0001857280731201172, "call": 9.751319885253906e-05, "teardown": 0.00010800361633300781}, "outcome": "passed", "id": "test_func.py::test_fixtures[0]", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_func.py", "line": 12, "test_name": "test_fixtures[1]", "duration": 0.0004830360412597656, "durations": {"setup": 0.0001609325408935547, "call": 0.00019860267639160156, "teardown": 0.00012350082397460938}, "outcome": "failed", "id": "test_func.py::test_fixtures[1]", "stdout": "", "stderr": "", "error": {"humanrepr": "number = 1\n\n    @pytest.mark.parametrize(\"number\", list(range(3)))\n    def test_fixtures(number):\n>       assert number % 2 == 0\nE       assert (1 % 2) == 0\n\ntest_func.py:14: AssertionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_func.py", "line": 12, "test_name": "test_fixtures[2]", "duration": 0.00046324729919433594, "durations": {"setup": 0.00019407272338867188, "call": 0.000118255615234375, "teardown": 0.00015091896057128906}, "outcome": "passed", "id": "test_func.py::test_fixtures[2]", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_func.py", "line": 17, "test_name": "test_error", "duration": 0.0006124973297119141, "durations": {"setup": 0.00016021728515625, "call": 0.00017523765563964844, "teardown": 0.0002770423889160156}, "outcome": "failed", "id": "test_func.py::test_error", "stdout": "", "stderr": "", "error": {"humanrepr": "def test_error():\n>       1 / 0\nE       ZeroDivisionError: division by zero\n\ntest_func.py:18: ZeroDivisionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_module_setup_teardown.py", "line": 9, "test_name": "test_passing", "duration": 0.0006561279296875, "durations": {"setup": 0.00042366981506347656, "call": 0.00012421607971191406, "teardown": 0.00010824203491210938}, "outcome": "passed", "id": "test_module_setup_teardown.py::test_passing", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_module_setup_teardown.py", "line": 13, "test_name": "test_failing", "duration": 0.0004029273986816406, "durations": {"setup": 0.00011515617370605469, "call": 0.00015616416931152344, "teardown": 0.0001316070556640625}, "outcome": "failed", "id": "test_module_setup_teardown.py::test_failing", "stdout": "", "stderr": "", "error": {"humanrepr": "def test_failing():\n>       assert False\nE       assert False\n\ntest_module_setup_teardown.py:14: AssertionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_module_setup_teardown.py", "line": 24, "test_name": "TestClassPassing.test_passing", "duration": 0.00032711029052734375, "durations": {"setup": 0.00012946128845214844, "call": 9.703636169433594e-05, "teardown": 0.00010061264038085938}, "outcome": "passed", "id": "test_module_setup_teardown.py::TestClassPassing::()::test_passing", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_module_setup_teardown.py", "line": 35, "test_name": "TestClassFailing.test_failing", "duration": 0.00040459632873535156, "durations": {"setup": 0.0001227855682373047, "call": 0.0001354217529296875, "teardown": 0.00014638900756835938}, "outcome": "failed", "id": "test_module_setup_teardown.py::TestClassFailing::()::test_failing", "stdout": "TD M\nTD MO\n", "stderr": "", "error": {"humanrepr": "self = <test_module_setup_teardown.TestClassFailing object at 0x7f3a8a8b8f90>\n\n    def test_failing(self):\n>       assert False\nE       assert False\n\ntest_module_setup_teardown.py:36: AssertionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_skip.py", "line": 4, "test_name": "test_skip_function", "duration": 0.00019121170043945312, "durations": {"setup": 9.846687316894531e-05, "teardown": 9.274482727050781e-05}, "outcome": "skipped", "id": "test_skip.py::test_skip_function", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {"setup": "Skipped: Skip"}}
{"_type": "test_result", "file": "test_skip.py", "line": 10, "test_name": "TestSkipCall.test_skip_method", "duration": 0.0001850128173828125, "durations": {"setup": 9.179115295410156e-05, "teardown": 9.322166442871094e-05}, "outcome": "skipped", "id": "test_skip.py::TestSkipCall::()::test_skip_method", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {"setup": "Skipped: Skip"}}
{"_type": "test_result", "file": "test_skip.py", "line": 17, "test_name": "TestSkipClass.test_skipped_1", "duration": 0.00020432472229003906, "durations": {"setup": 0.00011539459228515625, "teardown": 8.893013000488281e-05}, "outcome": "skipped", "id": "test_skip.py::TestSkipClass::()::test_skipped_1", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {"setup": "Skipped: Skip"}}
{"_type": "test_result", "file": "test_skip.py", "line": 20, "test_name": "TestSkipClass.test_skipped_2", "duration": 0.00023365020751953125, "durations": {"setup": 0.00011086463928222656, "teardown": 0.0001227855682373047}, "outcome": "skipped", "id": "test_skip.py::TestSkipClass::()::test_skipped_2", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {"setup": "Skipped: Skip"}}
{"_type": "test_result", "file": "test_slow.py", "line": 4, "test_name": "test_slow_passing", "duration": 2.0021703243255615, "durations": {"setup": 0.00015354156494140625, "call": 2.0018436908721924, "teardown": 0.00017309188842773438}, "outcome": "passed", "id": "test_slow.py::test_slow_passing", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_std.py", "line": 4, "test_name": "test_stdout", "duration": 0.00043892860412597656, "durations": {"setup": 0.00016498565673828125, "call": 0.0001513957977294922, "teardown": 0.00012254714965820312}, "outcome": "passed", "id": "test_std.py::test_stdout", "stdout": "STDOUT\n", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_std.py", "line": 8, "test_name": "test_stderr", "duration": 0.0003924369812011719, "durations": {"setup": 0.0001385211944580078, "call": 0.00013566017150878906, "teardown": 0.000118255615234375}, "outcome": "passed", "id": "test_std.py::test_stderr", "stdout": "", "stderr": "STDERR\n", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_std.py", "line": 19, "test_name": "TestClassStdout.test_stdout", "duration": 0.0004246234893798828, "durations": {"setup": 0.0001513957977294922, "call": 0.00015020370483398438, "teardown": 0.00012302398681640625}, "outcome": "passed", "id": "test_std.py::TestClassStdout::()::test_stdout", "stdout": "STDOUT\n", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_std.py", "line": 30, "test_name": "TestClassStdoutSetup.test_stdout", "duration": 0.00042557716369628906, "durations": {"setup": 0.00016736984252929688, "call": 0.00011920928955078125, "teardown": 0.00013899803161621094}, "outcome": "passed", "id": "test_std.py::TestClassStdoutSetup::()::test_stdout", "stdout": "SETUP\n", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_std.py", "line": 41, "test_name": "TestClassStdoutAllPhases.test_stdout", "duration": 0.00043654441833496094, "durations": {"setup": 0.0001614093780517578, "call": 0.0001316070556640625, "teardown": 0.00014352798461914062}, "outcome": "passed", "id": "test_std.py::TestClassStdoutAllPhases::()::test_stdout", "stdout": "SETUP\nTEST\nTEARDOWN\n", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_std.py", "line": 52, "test_name": "TestClassFailing.test_stderr", "duration": 0.0004050731658935547, "durations": {"setup": 0.00014901161193847656, "call": 0.0001308917999267578, "teardown": 0.0001251697540283203}, "outcome": "passed", "id": "test_std.py::TestClassFailing::()::test_stderr", "stdout": "", "stderr": "STDERR\n", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_unittest.py", "line": 8, "test_name": "TestStringMethods.test_isupper", "duration": 0.000499725341796875, "durations": {"setup": 0.0001583099365234375, "call": 0.00015401840209960938, "teardown": 0.00018739700317382812}, "outcome": "failed", "id": "test_unittest.py::TestStringMethods::test_isupper", "stdout": "", "stderr": "", "error": {"humanrepr": "self = <test_unittest.TestStringMethods testMethod=test_isupper>\n\n    def test_isupper(self):\n        self.assertTrue(\"FOO\".isupper())\n>       self.assertTrue(\"Foo\".isupper())\nE       AssertionError: False is not true\n\ntest_unittest.py:10: AssertionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_unittest.py", "line": 5, "test_name": "TestStringMethods.test_upper", "duration": 0.0003757476806640625, "durations": {"setup": 0.00013113021850585938, "call": 0.0001239776611328125, "teardown": 0.00012063980102539062}, "outcome": "passed", "id": "test_unittest.py::TestStringMethods::test_upper", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "directory/test_file_2.py", "line": 1, "test_name": "test_success", "duration": 0.0005571842193603516, "durations": {"setup": 0.0002856254577636719, "call": 0.00014543533325195312, "teardown": 0.00012612342834472656}, "outcome": "passed", "id": "directory/test_file_2.py::test_success", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "directory/test_func.py", "line": 1, "test_name": "test_success", "duration": 0.00038504600524902344, "durations": {"setup": 0.00014257431030273438, "call": 0.00011277198791503906, "teardown": 0.00012969970703125}, "outcome": "passed", "id": "directory/test_func.py::test_success", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "session_end", "total_duration": 2.257065534591675, "passed": 17, "failed": 9, "error": 4, "skipped": 4}

From there, you can use the id field to relaunch the tests you want:

pytest-litf '{"nodeids": ["test_std.py::TestClassFailing::()::test_stderr"]}'
collecting ... 
{"_type": "litf_start", "litf_version": "0.0.1"}
{"_type": "session_start", "test_number": 1}
{"_type": "test_result", "file": "test_std.py", "line": 52, "test_name": "TestClassFailing.test_stderr", "duration": 0.000385284423828125, "durations": {"setup": 0.00014925003051757812, "call": 0.00012636184692382812, "teardown": 0.00010967254638671875}, "outcome": "passed", "id": "test_std.py::TestClassFailing::()::test_stderr", "stdout": "", "stderr": "STDERR\n", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "session_end", "total_duration": 0.010551691055297852, "passed": 1, "failed": 0, "error": 0, "skipped": 0}

Or whole files:

pytest-litf '{"files": ["test_unittest.py"]}'
collecting ... 
{"_type": "litf_start", "litf_version": "0.0.1"}
{"_type": "session_start", "test_number": 2}
{"_type": "test_result", "file": "test_unittest.py", "line": 8, "test_name": "TestStringMethods.test_isupper", "duration": 0.00048279762268066406, "durations": {"setup": 0.00015020370483398438, "call": 0.0001761913299560547, "teardown": 0.000156402587890625}, "outcome": "failed", "id": "test_unittest.py::TestStringMethods::test_isupper", "stdout": "", "stderr": "", "error": {"humanrepr": "self = <test_unittest.TestStringMethods testMethod=test_isupper>\n\n    def test_isupper(self):\n        self.assertTrue(\"FOO\".isupper())\n>       self.assertTrue(\"Foo\".isupper())\nE       AssertionError: False is not true\n\ntest_unittest.py:10: AssertionError"}, "logs": "", "skipped_messages": {}}
{"_type": "test_result", "file": "test_unittest.py", "line": 5, "test_name": "TestStringMethods.test_upper", "duration": 0.0003337860107421875, "durations": {"setup": 0.00011372566223144531, "call": 0.0001251697540283203, "teardown": 9.489059448242188e-05}, "outcome": "passed", "id": "test_unittest.py::TestStringMethods::test_upper", "stdout": "", "stderr": "", "error": {"humanrepr": ""}, "logs": "", "skipped_messages": {}}
{"_type": "session_end", "total_duration": 0.028723478317260742, "passed": 1, "failed": 1, "error": 0, "skipped": 0}

The protocol and plugin are open-source. I hope my effort could help reducing the maintenance cost between the test libraries and outside tools.