Pytest: Display path different from real path

[PicouAymeric]

In my pytest plugin, I generate a lot of tests on the same test function / same class / same module. Using pytest hook pytest_generate_tests
In order to be able to to run just a subset of my tests I would like to create virtual subset of my tests eventhough they are in the end running the same test function.

The idea would be to create a pytest_pycollect_makeitem hook function in my plugin to set a display prop on all my test different from nodeid

    def pytest_pycollect_makeitem(self, collector, name, obj):
        report = yield
        items = report.get_result()
        new_results = []
        for item in items:
            item.display = [display path depending on the test]
            new_results.append(item)
        return new_results

Then in Vscode testing tab I would have the display prop displayed instead of nodeid, but if we run that test it still runs the nodeid test.
In a previous version of Vscode I found a hack by changing fspath and _nodeid of my test then force session.config.args to my actual test module in pytest_collection, but this broke after a vscode upate (I kind of expected it to break someday)

Do you think it would be possible to differentiate the test path displayed and the test path executed?

[eleanorjboyd]

Hi! Thanks for your detailed explanation! Are you able to use pytest parameterization or investigate any external plugins? We pass the testids in payloads between the python extension and the python subprocess. Therefore if the payloads sent to the python extension process display names and those are different from the run_ids which we use to signal which test runs. Do you want more info on how we get the display names and run_ids to see if something could be done there?

[PicouAymeric]

I have tried for a while to find a way to change display path without having to edit the python extension but I didn't find anything that works properly. If you have some examples of plugin doing that, that would interrest me!
I have seen in the vscode_pytest code how you build the test tree. From what I understand, we display the test tree using the name parameter of each node.
Changing name parameter do not change the test that we execute, so we are probably not that far from having a display path different from real path.

However current implementation is closely linked to the folder structure: a test inside a class inside a module inside a folder. I ll try to overwrite that and build a tree only from a display property

[eleanorjboyd]

so pytest parameterized tests allows for test renaming like so:

import pytest

@pytest.mark.parametrize(
    "input,expected", 
    [(1, 2), (2, 4), (3, 6)], 
    ids=["Double 1", "Double 2", "Double 3"]
)
def test_doubling(input, expected):
    assert 2 * input == expected

but I think this is probably hacky and not right in your situation. Let me give it some more thought

[eleanorjboyd]

Looking more I actually think your original idea was correct, you should be able to add your own hooks and have that work within the scope of the extension. Pytest has specific measures to allow hooks to interact and so this should work. If you look the plugin you can see the test tree is build from the hook pytest_sessionfinish. This would mean if you change the test_case.name before the python extension gets to it and then it should be edited. I think you might need to put in a hook like "try first" and define your own pytest_sessionfinish but if that doesn't work I could try it myself again but editing the extension to specify that the pytest_sessionfinish in it should be done last.

[PicouAymeric]

Hi,
The issue is that in pytest_sessionfinish you are sending the tree to run_pytest_script's socket with an http request, which I cannot override.
I tried a with:
Executing before your hook:
@pytest.hookimpl(tryfirst=True)
Executing after your hook:
@pytest.hookimpl(trylast=True)
Executing before and after your hook (separated by yield):
@pytest.hookimpl(tryfirst=True, hookwrapper=True)
but still, your hook is always executed and always send the 'wrong' tree to run_pytest_script

If that information was given using the session for example I could find a hack to override it, but since it is transmitted outside the pytest process I don't see any way of overwritting the tree

[PicouAymeric]

Reading again your comments, I think maybe my needs weren't as clear as I thought:
I don't just want to change the name of the test (the part between [] that I can edit with ids in @parametrize or metafunc.parametrize in pytest_generate_tests(self, metafunc)) what I want is to be able to edit the whole path.
From what I understood, this path contructed in build_test_tree only depends on the file and folder hierarchy and cannot be overwritten

[eleanorjboyd]

thank you for clarifying! If you just wanted to update the display but not actual I think those are too interconnected for this to maybe be possible in the current setup. The path of the test is used during build for both the path variable and the name and there would be no way for a separate plugin to make a change to the display path without changing the actual path the test goes to. I will leave this as a feature request then and if it gets enough support we can discuss how this could be incorporated. Thanks!