Handling of symlinks in pytest discovery

[rjra100]

Type: Bug

Behaviour

I'm finding it difficult to get VSCode to correctly discover pytest tests in a workspace whose path involves a symlink. The output of <python> run_adapter.py discover pytest varies depending exactly how it's called (and the extension reacts differently to these variations), but the Output pane doesn't accurately reflect what inputs were passed to run_adapter.py, making it tricky to work out exactly what's going on.

Ultimately, I'd like to be able to create my workspace in an arbitrary location (i.e. where my IT department provided me with storage - something like /data/store_123456/workspace) but reference it via a symlink from my home directory (so Open Folder /home/<me>), and have Pytest discovery work without my needing to muck about with its inputs.

As a secondary issue, the Output pane should accurately reflect how the test adapter is actually invoked.

Steps to reproduce:

1. Make a directory, put a dummy pytest in it:

mkdir -p ~/real_location/test
echo -e "def test_something():\n    pass" > ~/real_location/test/test_example.py

2. Make a symlink to it:

ln -s ~/real_location~/my_workspace 

3. Open the symlink folder in VSCode (Open Folder, navigate to ~/my_workspace)
4. Activate the Python extension (open the file); open the testing pane and enable pytest testing.

Case 1 : default settings (does not work)

5. Examine the testing pane.

• Expected behaviour: the test shows up
• Actual behaviour: the test does not appear (a real_location workspace root node appears, nothing underneath)

Output pane

2024-08-14 09:06:51.493 [info] Discover tests for workspace name: my_workspace - uri: /home/ralexander72/my_workspace
2024-08-14 09:06:51.525 [warning] could not find a pixi interpreter for the interpreter at /bin/python3
2024-08-14 09:06:51.562 [warning] could not find a pixi interpreter for the interpreter at /bin/python3
2024-08-14 09:06:51.567 [info] > /bin/python3 ~/.vscode-server/extensions/ms-python.python-2024.12.3/python_files/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear
2024-08-14 09:06:51.567 [info] cwd: .

Case 2 : explicit directory (does not work, differently)

6. Set python.testing.pytestArgs to ${workspaceFolder}
7. Examine the testing pane.

• Expected behaviour: the test shows up and can be run
• Actual behaviour:
  • my_workspace node shows up in test explorer, containing the test (if run in the same window, the existing real_location node remains)
  • Attempting to run the test (click the run or debug button next to it), nothing happens (Test Results pane shows "Finished running tests!"; the test does not run).

Output pane

2024-08-14 09:09:13.173 [info] Discover tests for workspace name: my_workspace - uri: /home/ralexander72/my_workspace
2024-08-14 09:09:13.173 [warning] could not find a pixi interpreter for the interpreter at /bin/python3
2024-08-14 09:09:13.226 [warning] could not find a pixi interpreter for the interpreter at /bin/python3
2024-08-14 09:09:13.229 [info] > /bin/python3 ~/.vscode-server/extensions/ms-python.python-2024.12.3/python_files/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .
2024-08-14 09:09:13.229 [info] cwd: .

Case 3 : explicit directory and rootdir (does work)

8. Set python.testing.pytestArgs to ["--rootdir=.", "${workspaceFolder}"]
9. Examine the testing pane.

• Expected and actual behaviour: the test shows up and can be run. Phew.

Output pane

2024-08-14 09:13:37.175 [info] Discover tests for workspace name: my_workspace - uri: /home/ralexander72/my_workspace
2024-08-14 09:13:37.176 [warning] could not find a pixi interpreter for the interpreter at /bin/python3
2024-08-14 09:13:37.184 [warning] could not find a pixi interpreter for the interpreter at /bin/python3
2024-08-14 09:13:37.188 [info] > /bin/python3 ~/.vscode-server/extensions/ms-python.python-2024.12.3/python_files/testing_tools/run_adapter.py discover pytest -- -s --cache-clear --rootdir=. .
2024-08-14 09:13:37.188 [info] cwd: 

I gather the wittering about pixi is harmless noise (though it'd be nice if that could be shut up). There is no further output displayed: no error in any case; things just don't work quite right.

Note that despite passing ${workspaceFolder} (i.e. a full path ~/my_workspace), the output pane consistently substitutes the workspace directory with ., making it impossible to distinguish --rootdir=. from --rootdir=${workspaceFolder}, for example. Note that for Pytest discovery, this difference is significant.

Analysis

What I think's happening depends on exactly what arguments are really passed to run_adapter.py discover pytest. Adding pretty formatting to the output for readability:

python <...>/run_adapter.py discover pytest --pretty -- --rootdir ~/my_workspace -s --cache-clear

[
    {
        "parents": [
            {
                "id": "./test",
                "kind": "folder",
                "name": "test",
                "parentid": ".",
                "relpath": "./test"
            },
            {
                "id": "./test/test_example.py",
                "kind": "file",
                "name": "test_example.py",
                "parentid": "./test",
                "relpath": "./test/test_example.py"
            }
        ],
        "root": "/home/ralexander72/real_location",
        "rootid": ".",
        "tests": [
            {
                "id": "./test/test_example.py::test_something",
                "markers": [],
                "name": "test_something",
                "parentid": "./test/test_example.py",
                "source": "./../real_location/test/test_example.py:1"
            }
        ]
    }
]

Note: "root": "/home/ralexander72/real_location"; "source": "./../real_location/test/test_example.py:1"

I suspect this is what's actually being invoked by default (--rootdir . in the output pane notwithstanding), and that the extension sees the file locations as somewhere outside of the workspace (/home/ralexander72/my_workspace) and ignores the tests.

python <...>/run_adapter.py discover pytest --pretty -- --rootdir ~/my_workspace -s --cache-clear ~/my_workspace

[
    {
        "parents": [
            {
                "id": "./test",
                "kind": "folder",
                "name": "test",
                "parentid": ".",
                "relpath": "./test"
            },
            {
                "id": "./test/test_example.py",
                "kind": "file",
                "name": "test_example.py",
                "parentid": "./test",
                "relpath": "./test/test_example.py"
            }
        ],
        "root": "/home/ralexander72/my_workspace",
        "rootid": ".",
        "tests": [
            {
                "id": "./test/test_example.py::test_something",
                "markers": [],
                "name": "test_something",
                "parentid": "./test/test_example.py",
                "source": "./../real_location/test/test_example.py:1"
            }
        ]
    }
]

Note: "root": "/home/ralexander72/my_workspace"; "source": "./../real_location/test/test_example.py:1"

I suspect this is what's actually being invoked with "python.testing.pytestArgs" : [ "${workspaceFolder}" ]. Now the root path is sane, and the tests are therefore considered part of the workspace and show up. However, the source paths are "relative" paths from the symlinked path to the "real" one - opening the test via the "Go To Test" button shows the file path as /home/ralexander72/real_location/test/test_example.py, and the test won't run.

python <...>/run_adapter.py discover pytest --pretty -- --rootdir . -s --cache-clear ~/my_workspace

[
    {
        "parents": [
            {
                "id": "./test",
                "kind": "folder",
                "name": "test",
                "parentid": ".",
                "relpath": "./test"
            },
            {
                "id": "./test/test_example.py",
                "kind": "file",
                "name": "test_example.py",
                "parentid": "./test",
                "relpath": "./test/test_example.py"
            }
        ],
        "root": "/home/ralexander72/my_workspace",
        "rootid": ".",
        "tests": [
            {
                "id": "./test/test_example.py::test_something",
                "markers": [],
                "name": "test_something",
                "parentid": "./test/test_example.py",
                "source": "./test/test_example.py:1"
            }
        ]
    }
]

Note: "root": "/home/ralexander72/my_workspace"; "source": "./test/test_example.py:1"

These arguments are now as passed explicitly on the command line ("python.testing.pytestArgs" : [ "--rootdir=.", "${workspaceFolder}" ] - though the output pane still shows --rootdir=. .) Now the root and source paths both look sensible, and configured this way, the test both shows up and works.

Version info

Extension version: 2024.12.2, 2024.12.3 (seems to have updated overnight!) VS Code version: Code 1.92.1 (eaa41d57266683296de7d118f574d0c2652e1fc4, 2024-08-07T20:16:39.455Z) OS version: Windows_NT x64 10.0.19045 Modes: Remote OS version: Linux x64 5.15.153.1-microsoft-standard-WSL2

• Python version: version-independent, but tried 3.10.12+Pytest 8.2, 3.12.4+Pytest 8.1.1
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): None
• Value of the python.languageServer setting: Default

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "" or "${workspaceFolder}; see analysis above
• pytestEnabled: true

Installed Extensions

Extension Name	Extension Id	Version
Auto Snippet	Gruntfuggly.auto-snippet	0.0.11
Black Formatter	ms-python.black-formatter	2024.2.0
C/C++	ms-vscode.cpptools	1.21.6
clangd	llvm-vs-code-extensions.vscode-clangd	0.1.29
CMake	twxs.cmake	0.0.17
CMake Test Explorer	fredericbonnet.cmake-test-adapter	0.17.4
CMake Tools	ms-vscode.cmake-tools	1.18.44
CodeLLDB	vadimcn.vscode-lldb	1.10.0
Command Variable	rioj7.command-variable	1.63.0
GitLens — Git supercharged	eamodio.gitlens	15.2.3
JavaScript Debugger	ms-vscode.js-debug	1.92.0
Live Preview	ms-vscode.live-server	0.4.14
markdownlint	DavidAnson.vscode-markdownlint	0.55.0
Pylance	ms-python.vscode-pylance	2024.8.1
Python	ms-python.python	2024.12.2
Python Debugger	ms-python.debugpy	2024.10.0
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.9
Test Adapter Converter	ms-vscode.test-adapter-converter	0.1.9
Test Explorer UI	hbenl.vscode-test-explorer	2.21.1
Time Travel Debug for C/C++	undo.udb	2.0.4

[eleanorjboyd]

Hello! I think this issue is fixed by opting into this experiment. Give it a try and let me know! Add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"] then try again. You can confirm you have the rewrite enabled by setting log levels to trace (via command palette) then check for Experiment 'pythonTestAdapter' is active in your python logs.

Let me know if the rewrite fixes your issue. Thanks!

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[rjra100]

Apologies, I missed this - you're quite right, the experimental python test adapter does work much better. Will switch to that :)