TestIds not relative to rootdir & cannot be run as pytest args

[eleanorjboyd]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
  Mac ARM
  pytest version: pytest 7.2.2
• minimal example if possible

Hi! This question is a little different but tried to provide as much detail as possible to have the report be clear. Let me know if I can provide more information! I have a project structured as following:

root
└─ module
└─ tests
─└─ unittest
──└─ test_uppercase.py
──└─ pytest.init

I created the following demo which I have attached but this is the jist.

if __name__ == "__main__":
    # with ini file
    print("with ini file")
    pytest.main(["-p", "plugin_ex", "-c", "tests/pytest.ini", "-v"])
    # with ini file
    print("with ini file & test ID from previous run plugin")
    pytest.main(
        [
            "-p",
            "plugin_ex",
            "-c",
            "tests/pytest.ini",
            "-v",
            "unittests/test_param.py::test_odd_even[1]",
        ]
    )

run the above script file.

def pytest_sessionfinish(session, exitstatus):
    for test_case in session.items:
        print("test ids from plugin:", test_case.nodeid)

have the given code for my plugin_ex

get the following output

with ini file
================================================== test session starts ===================================================
platform darwin -- Python 3.10.8, pytest-7.4.0, pluggy-1.2.0 -- /Users/eleanorboyd/testingFiles/from_users/test_directory/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/eleanorboyd/testingFiles/from_users/test_directory/tests
configfile: pytest.ini
collected 6 items                                                                                                        

tests/unittests/test_add.py::test_add_two_numbers PASSED                                                           [ 16%]
tests/unittests/test_add.py::TestAdd::test_add_two_numbers PASSED                                                  [ 33%]
tests/unittests/test_param.py::test_odd_even[1] PASSED                                                             [ 50%]
tests/unittests/test_param.py::test_odd_even[2] PASSED                                                             [ 66%]
tests/unittests/test_param.py::test_odd_even[3] PASSED                                                             [ 83%]
tests/unittests/test_param.py::test_odd_even[4] PASSED                                                             [100%]test ids from plugin: unittests/test_add.py::test_add_two_numbers
test ids from plugin: unittests/test_add.py::TestAdd::test_add_two_numbers
test ids from plugin: unittests/test_param.py::test_odd_even[1]
test ids from plugin: unittests/test_param.py::test_odd_even[2]
test ids from plugin: unittests/test_param.py::test_odd_even[3]
test ids from plugin: unittests/test_param.py::test_odd_even[4]


=================================================== 6 passed in 0.01s ====================================================
with ini file & test ID from previous run plugin
================================================== test session starts ===================================================
platform darwin -- Python 3.10.8, pytest-7.4.0, pluggy-1.2.0 -- /Users/eleanorboyd/testingFiles/from_users/test_directory/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/eleanorboyd/testingFiles/from_users/test_directory/tests
configfile: pytest.ini
collected 0 items                                                                                                        

================================================= no tests ran in 0.00s ==================================================
ERROR: file or directory not found: unittests/test_param.py::test_odd_even[1]

You can see that with the init file in the first run the rootdir is listed as rootdir: /Users/eleanorboyd/testingFiles/from_users/test_directory/tests and the test ids as printed out through the plugin are test ids from plugin: unittests/test_param.py::test_odd_even[4]. But then when I try and run a given test via its testID it doesn't work and returns the error you see in the second run even though its root dir is still listed as rootdir: /Users/eleanorboyd/testingFiles/from_users/test_directory/tests

From what I got of the doc is that all testIDs would be relative to the rootDir and therefore the testID when the rootdir stays the same should also be the same.

The reason I am experiencing this bug is I want to get testIDs from a plugin and then be able to call the given test from the command line. Is this actually a bug or am I getting the testID incorrectly somewhere or misunderstanding the docs in that the testID is relative to the rootdir?

Appreciate any help!

TO REPRO:
run script.py from the following example code I uploaded
test_directory.zip

Output of pip list:
Package Version

---

exceptiongroup 1.1.2
iniconfig 2.0.0
packaging 23.1
pip 23.2
pluggy 1.2.0
pytest 7.4.0
setuptools 65.4.1
tomli 2.0.1

Thanks! Sorry for the length here

[bluetech]

Hi @eleanorjboyd,

The "node IDs" provided on the command line are in fact not relative to the rootdir, but to the "invocation dir", i.e. the working directory when pytest was invoked. This I think makes more sense than making them relative to the rootdir, because the user works in the context of the CWD, does auto-completion based on the CWD, etc. I don't think this is something we'll be changing.

The docs are currently somewhat misleading in this regard, this was also mentioned in issue #11107. I'll be sending a PR soon to rectify this.

[eleanorjboyd]

Hi! Thanks for getting back to me and answering my question! This is very useful as I continue to work. One clarifying question, so what you're saying is that the "-c" parameter does not change the "invocation directory" since "-c" changes the root right?

One issue I think I am still running into is when I add test_directory to my python path then run pytest -p plugin_ex -c tests/pytest.ini -v the print out I get is below. What pytest prints as part of the "-v" is test names that are relative to the test_directory like tests/unittests/test_add.py::test_add_two_numbers, while when I print out the test_case.nodeid from the plugin I get unittests/test_add.py::test_add_two_numbers which is relative to the config file. I think what you are saying is the tests/unittests/test_add.py::test_add_two_numbers should be right but I am not sure how I then get this value when I hook in via a plugin since the plugin has a different node id when I print it. Thanks!

platform darwin -- Python 3.7.9, pytest-7.2.2, pluggy-1.0.0 -- /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
cachedir: .pytest_cache
rootdir: /Users/eleanorboyd/testingFiles/from_users/test_directory/tests, configfile: pytest.ini
collected 6 items                                                                                        

tests/unittests/test_add.py::test_add_two_numbers PASSED                                           [ 16%]
tests/unittests/test_add.py::TestAdd::test_add_two_numbers PASSED                                  [ 33%]
tests/unittests/test_param.py::test_odd_even[1] PASSED                                             [ 50%]
tests/unittests/test_param.py::test_odd_even[2] PASSED                                             [ 66%]
tests/unittests/test_param.py::test_odd_even[3] PASSED                                             [ 83%]
tests/unittests/test_param.py::test_odd_even[4] PASSED                                             [100%]cwd /Users/eleanorboyd/testingFiles/from_users/test_directory
 test ids as known to the plugin: unittests/test_add.py::test_add_two_numbers
 test ids as known to the plugin: unittests/test_add.py::TestAdd::test_add_two_numbers
 test ids as known to the plugin: unittests/test_param.py::test_odd_even[1]
 test ids as known to the plugin: unittests/test_param.py::test_odd_even[2]
 test ids as known to the plugin: unittests/test_param.py::test_odd_even[3]
 test ids as known to the plugin: unittests/test_param.py::test_odd_even[4]

To give another example of what I am saying, running pytest from the same directory should always result in the same testids (since the invocation directory does not change). But when I run pytest -p plugin_ex -c tests/pytest.ini vs pytest -p plugin_ex from test_directory my plugin finds different testids.

eleanorboyd@Eleanors-MacBook-Pro test_directory % pytest -p plugin_ex -c tests/pytest.ini
========================================== test session starts ===========================================
platform darwin -- Python 3.7.9, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/eleanorboyd/testingFiles/from_users/test_directory/tests, configfile: pytest.ini
collected 6 items                                                                                        

tests/unittests/test_add.py ..                                                                     [ 33%]
tests/unittests/test_param.py ....                                                                 [100%]cwd /Users/eleanorboyd/testingFiles/from_users/test_directory
 test ids as known to the plugin: unittests/test_add.py::test_add_two_numbers
 test ids as known to the plugin: unittests/test_add.py::TestAdd::test_add_two_numbers
 test ids as known to the plugin: unittests/test_param.py::test_odd_even[1]
 test ids as known to the plugin: unittests/test_param.py::test_odd_even[2]
 test ids as known to the plugin: unittests/test_param.py::test_odd_even[3]
 test ids as known to the plugin: unittests/test_param.py::test_odd_even[4]


=========================================== 6 passed in 0.02s ============================================
eleanorboyd@Eleanors-MacBook-Pro test_directory % pytest -p plugin_ex                    
========================================== test session starts ===========================================
platform darwin -- Python 3.7.9, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/eleanorboyd/testingFiles/from_users/test_directory
collected 6 items                                                                                        

tests/unittests/test_add.py ..                                                                     [ 33%]
tests/unittests/test_param.py ....                                                                 [100%]cwd /Users/eleanorboyd/testingFiles/from_users/test_directory
 test ids as known to the plugin: tests/unittests/test_add.py::test_add_two_numbers
 test ids as known to the plugin: tests/unittests/test_add.py::TestAdd::test_add_two_numbers
 test ids as known to the plugin: tests/unittests/test_param.py::test_odd_even[1]
 test ids as known to the plugin: tests/unittests/test_param.py::test_odd_even[2]
 test ids as known to the plugin: tests/unittests/test_param.py::test_odd_even[3]
 test ids as known to the plugin: tests/unittests/test_param.py::test_odd_even[4]

sorry this got lengthy but hopefully this explanation makes sense. Thanks!

[eleanorjboyd]

Hello again! I think I have figured it out, below is my solution.

Since the nodeid is relative I calculate the difference between the root and the invocation path so my test_ids always have the relativity to the invocation location.

calculate if there is a difference between root and invocation paths

invocation_dir = early_config.invocation_params.dir
    root = early_config.rootpath
    if invocation_dir != root:
        try:
            global RELATIVE_INVOCATION_PATH
            RELATIVE_INVOCATION_PATH = root.relative_to(invocation_dir)
        except:
            pass

update testids to always be relative to the invocation path

def get_workspace_node_id(testid: str):
    id = testid
    global RELATIVE_INVOCATION_PATH
    if RELATIVE_INVOCATION_PATH:
        id = str(pathlib.Path(RELATIVE_INVOCATION_PATH) / testid)
    return id

I think the biggest thing that confused me was running pytest -p plugin_ex -c tests/pytest.ini tests/unittests/test_add.py::test_add_two_numbers works but pytest -p plugin_ex -c tests/pytest.ini unittests/test_add.py::test_add_two_numbers doesn't. This means that when calling specific tests the cwd/root directory that pytest runs on is not where the testids are computing against instead the tests are looked for at the invocation path. I previously didn't understand this since I thought the testids would continue to be calculated against the rootdir. What I noticed is that the output printed on -v is always relative to the invocation directory. You have this function that converts it before printing. Not sure if I missed something that explained this but wanted to update on my findings. Thanks!

    def cwd_relative_nodeid(self, nodeid: str) -> str:
        # nodeid's are relative to the rootpath, compute relative to cwd.
        if self.invocation_params.dir != self.rootpath:
            fullpath = self.rootpath / nodeid
            nodeid = bestrelpath(self.invocation_params.dir, fullpath)=
        return nodeid

from

pytest/src/_pytest/config/__init__.py

Line 1117 in 32f4808

def cwd_relative_nodeid(self, nodeid: str) -> str:

Thank you for your help and for maintaining pytest! I appreciate it!

[bluetech]

One clarifying question, so what you're saying is that the "-c" parameter does not change the "invocation directory" since "-c" changes the root right?

-c does not change the invocation directory, the invocation directory is always the CWD when pytest is invoked.

-c doesn't in itself change the rootdir, it changes the config file path, and what happens is that if --rootdir is not used then the directory of the config file is used as the rootdir.

One issue I think I am still running into is when I add test_directory to my python path then run pytest -p plugin_ex -c tests/pytest.ini -v the print out I get is below. What pytest prints as part of the "-v" is test names that are relative to the test_directory like tests/unittests/test_add.py::test_add_two_numbers,

One important thing to note is that what is printed by -v is not actually the nodeids as they are, instead the printing code (the internal terminal plugin in pytest) massages the paths to be relative to the invocation dir (AKA "startpath"). Searching for bestrelpath in terminal.py should show the places where this happens approximately.

while when I print out the test_case.nodeid from the plugin I get unittests/test_add.py::test_add_two_numbers which is relative to the config file.

Just to be precise, nodeids are relative to the rootdir. (Well most of the time... see #11245).

Since the nodeid is relative I calculate the difference between the root and the invocation path so my test_ids always have the relativity to the invocation location.

Makes sense, if you want the path relative the invocation location from the nodeid, you can do this.

update testids to always be relative to the invocation path

What is get_workspace_node_id in this case?

What I noticed is that the output printed on -v is always relative to the invocation directory. You have this function that converts it before printing. Not sure if I missed something that explained this but wanted to update on my findings.

Right :)

---

BTW, I am not sure exactly what you are doing with the nodeids, but you might want to avoid parsing the path of the nodeid entirely, and instead get the path from the Node itself, i.e. use item.path in your plugin.

[eleanorjboyd]

appreciate your response here! Glad it all got straightened out and thanks for the extra clarification. I ended up moving forward with the node path for an absolute reference path and that seems to be working well. Thanks