Unittest id parsing fails when using testscenarios

[bigjools]

Environment data

• VS Code version: 1.60.2
• Extension version (available under the Extensions sidebar): v2021.9.1246542782
• OS and version: Ubuntu 20.04
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.10
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
• Relevant/affected Python packages and their versions:
• Relevant/affected Python-related VS Code extensions and their versions:
• Value of the python.languageServer setting: pylance

[NOTE: If you suspect that your issue is related to the Microsoft Python Language Server (python.languageServer: 'Microsoft'), please download our new language server Pylance from the VS Code marketplace to see if that fixes your issue]

Expected behaviour

Clicking "Run Test" on a test should run it.

Actual behaviour

An error is shown in the Python log and the test is not run (see log below).

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

1. Configure tests to use unittest

    "python.testing.unittestEnabled": true,
    "python.testing.unittestArgs": [
        "-s",
        "my.tests.module"
    ],

2. After discovery, click "Run Test"

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~/src/helios/.tox/py3/bin/python3 ~/.vscode/extensions/ms-python.python-2021.9.1246542782/pythonFiles/visualstudio_py_testlauncher.py --us=helios.tests --up=test*.py --uvInt=1 --result-port=44367 -thelios.tests.tasks.test_tasks.TestUnlinkCVEsTask.test_ignores_unknown_CVEs --testFile=/home/juledwar/src/helios/helios.tests/helios/tests/tasks/test_tasks.py
cwd: ~/src/helios
Error 2021-10-06 13:48:56: [object Object] [object Object]

[karthiknadig]

@bigjools Can you share your project layout? or a minimal repro?

[bigjools]

I can't share my private repo but I'll describe the layout here and see if I can reproduce with another public one I have on Monday.

The tree looks like this:

src-root/
|
+ project/
    |
    + tests/
    |       |
    |       + testmodule1/ ... etc
    |
    + codemodule/  ... etc

And I have unittestArgs as -s project.tests and my workspace root is at src-root

[karthiknadig]

@bigjools -s is used for start directory path not modules. Can you try with -s ./project? the start directory should be ./project because using might be importing from codemodule.

[bigjools]

Huh interesting! Test discovery works with that pattern but not running them. Using a shell path works fine for both.

Ok I guess you can turn this ticket into a "print a sensible error message".

Thank you!

[bigjools]

Actually I spoke too soon.

If I use -s "project/tests", the load_tests function that lives in tests/__init__.py doesn't run, which is bad because that contains a loader for testscenarios, so you end up with missing/invalid tests.

If I use -s . then it finds all the tests just fine, however every test it tries to run has the same error I originally reported. ([object Object] etc.)

Running the tests from the CLI with python -m unittest discover -s . works just fine (or even -s project).

[bigjools]

Here's an example also of what the sidebar is doing. (This is a public project of mine so please feel free to test with it https://github.com/juledwar/soufi)
The tests that use testscenarios in the loader create multiple tests based on one test function, but each test name has a (foo) appended to it. The sidebar shows weird line breaks in the middle of that bracketed part.

[bigjools]

@karthiknadig Hi, let me know if I gave you enough info on this? Thanks.

[bigjools]

@karthiknadig Hi again, please let me know if you need any more info.

[karthiknadig]

@bigjools Sorry about the delayed reply. Yes, I have enough info for this.

[bigjools]

I thought I'd give an update on this:

Good news: The latest vscode seems to detect the tests correctly!
Bad news: The test runner craps out completely with this error text:

> ./.tox/py3/bin/python3 ~/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/visualstudio_py_testlauncher.py --us=soufi --up=test*.py --uvInt=1 --result-port=42757 -ttests.finders.test_python_finder.TestPythonFinder.test_get_source_info_raises_when_response_fails(pypi) --testFile=./soufi/tests/finders/test_python_finder.py
cwd: .
[ERROR 2022-2-7 9:56:26.649]: [object Object] [object Object]
Python interpreter path: ~/src/soufi/.tox/py3/bin/python3

Additionally, I cannot recreate in a public repo that I have but in my private repo I get an additional error about two tests having the same ID:

[ERROR 2022-2-7 9:49:28.894]: Multiple (2) raw data nodes had the same id: ./helios/tests/api/
test_release.py::TestReleasePatchStatusChangeDuringCoronaSyncScenarios::test_cannot_change_status_during_sync(rc)

Also it just caught my eye that the log datestamp is saying February instead of March.

[bigjools]

I'm not sure if anything has been changing lately, but thought I'd mention I am still completely unable to run any tests in any projects that use testscenarios, and there seems to be a new problem where the python.envFile is not evaluated before test discovery, so some imports are failing (and hence discovery fails) because they depend on env vars.

[eleanorjboyd]

Hello! We have just finished our testing rewrite and are beginning the roll out to users. If you are able, it would be very helpful to know if your issue still exists on the rewrite! To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!

[bigjools]

Hi, thanks for the details, much appreciated.

I am sadly unable to get even this far now as I think a new bug has been introduced in recent releases. I have a 3rd party library which has the same name as one of my sub-modules in my main code and while my CLI test runner has no problem with this, the test discovery in vscode sees it as a circular import as I think it tries to import itself rather than the 3rd party module.

I am not sure what to do, is this a new bug or did I misconfigure? Previously it has at least discovered all my tests, even if the testscenarios-based ones were incorrect.

Many thanks.

[eleanorjboyd]

Hi thanks for reaching out! Could you include a brief layout of your project and your logs from the python output channel? Also for your logs can you set your log level to trace via the "Developer: set log level" command in the command palette? With these I can better help figure out what might be going wrong. Thanks

[bigjools]

I've included a redacted log below (the env vars were possibly revealing things I don't want in a public forum).

The project is called helios. The pertinent parts of the project layout are roughly:
helios.spinach.<files here including an __init__.py>

And then there's a 3rd party library called spinach. The helios.spinach.__init__.py file attempts to import:
from spinach import job as spinach_job_mod which triggers the issue.

It's obvious at least to me that test discovery is not considering the rest of the venv (which lives in .hatch and is set up as the working venv in vscode) when trying to import project modules.

2023-10-31 09:17:13.201 [debug] Testing: Manually triggered test refresh
2023-10-31 09:17:13.201 [debug] Testing: Clearing all discovered tests
2023-10-31 09:17:13.201 [debug] Testing: Forcing test data refresh
2023-10-31 09:17:13.201 [debug] Testing: Refreshing all test data
2023-10-31 09:17:13.201 [debug] Cached data exists KeyPrefix=Cache_Method_Output_b._selectIfLaunchedViaActivatedEnv-Args=true
2023-10-31 09:17:13.201 [debug] Cached data exists KeyPrefix=Cache_Method_Output_b._selectIfLaunchedViaActivatedEnv-Args=true
2023-10-31 09:17:13.202 [debug] Found cached env for /home/juledwar/src/helios/.hatch/bin/python
2023-10-31 09:17:13.202 [debug] Found cached env for /home/juledwar/src/helios/.hatch/bin/python
2023-10-31 09:17:13.202 [debug] Resolved /home/juledwar/src/helios/.hatch/bin/python from cache: {"name":".hatch","location":"/home/juledwar/src/helios/.hatch","kind":"virt-venv","executable":{"filename":"/home/juledwar/src/helios/.hatch/bin/python","sysPrefix":"/home/juledwar/src/helios/.hatch","ctime":1687828618618,"mtime":1687828618618},"searchLocation":{"$mid":1,"path":"/home/juledwar/src/helios/.hatch","scheme":"file"},"display":"Python 3.11.6 ('.hatch': venv)","version":{"major":3,"minor":11,"micro":6,"release":{"level":"final","serial":0},"sysVersion":"3.11.6 (main, Oct 23 2023, 22:48:54) [GCC 11.4.0]"},"arch":3,"distro":{"org":""},"source":[],"type":"Virtual","id":"/home/juledwar/src/helios/.hatch/bin/python","detailedDisplayName":"Python 3.11.6 ('.hatch': venv)"}
2023-10-31 09:17:13.202 [debug] Resolved /home/juledwar/src/helios/.hatch/bin/python from cache: {"name":".hatch","location":"/home/juledwar/src/helios/.hatch","kind":"virt-venv","executable":{"filename":"/home/juledwar/src/helios/.hatch/bin/python","sysPrefix":"/home/juledwar/src/helios/.hatch","ctime":1687828618618,"mtime":1687828618618},"searchLocation":{"$mid":1,"path":"/home/juledwar/src/helios/.hatch","scheme":"file"},"display":"Python 3.11.6 ('.hatch': venv)","version":{"major":3,"minor":11,"micro":6,"release":{"level":"final","serial":0},"sysVersion":"3.11.6 (main, Oct 23 2023, 22:48:54) [GCC 11.4.0]"},"arch":3,"distro":{"org":""},"source":[],"type":"Virtual","id":"/home/juledwar/src/helios/.hatch/bin/python","detailedDisplayName":"Python 3.11.6 ('.hatch': venv)"}
2023-10-31 09:17:13.203 [info] Discover tests for workspace name: helios - uri: /home/juledwar/src/helios
2023-10-31 09:17:13.203 [debug] Cached data exists KeyPrefix=Cache_Method_Output_b._selectIfLaunchedViaActivatedEnv-Args=true
2023-10-31 09:17:13.203 [info] Discover tests for workspace name: helios @ ad19601 - uri: /home/juledwar/src/helios
2023-10-31 09:17:13.203 [debug] Found cached env for /home/juledwar/src/helios/.hatch/bin/python
2023-10-31 09:17:13.203 [debug] Resolved /home/juledwar/src/helios/.hatch/bin/python from cache: {"name":".hatch","location":"/home/juledwar/src/helios/.hatch","kind":"virt-venv","executable":{"filename":"/home/juledwar/src/helios/.hatch/bin/python","sysPrefix":"/home/juledwar/src/helios/.hatch","ctime":1687828618618,"mtime":1687828618618},"searchLocation":{"$mid":1,"path":"/home/juledwar/src/helios/.hatch","scheme":"file"},"display":"Python 3.11.6 ('.hatch': venv)","version":{"major":3,"minor":11,"micro":6,"release":{"level":"final","serial":0},"sysVersion":"3.11.6 (main, Oct 23 2023, 22:48:54) [GCC 11.4.0]"},"arch":3,"distro":{"org":""},"source":[],"type":"Virtual","id":"/home/juledwar/src/helios/.hatch/bin/python","detailedDisplayName":"Python 3.11.6 ('.hatch': venv)"}
2023-10-31 09:17:13.209 [debug] Dump environment variables {
***REDACTED***
2023-10-31 09:17:13.223 [debug] Activation Commands received source /home/juledwar/src/helios/.hatch/bin/activate for shell bash
2023-10-31 09:17:13.223 [debug] Has Custom Env Vars
2023-10-31 09:17:13.224 [debug] Activating Environment to capture Environment variables, . /home/juledwar/src/helios/.hatch/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python /home/juledwar/.vscode/extensions/ms-python.python-2023.18.0/pythonFiles/printEnvVariables.py
2023-10-31 09:17:13.224 [debug] Shell path 'bash', base path 'bash'
2023-10-31 09:17:13.224 [debug] Shell path identified as shell 'bash'
2023-10-31 09:17:13.224 [info] > . ~/src/helios/.hatch/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode/extensions/ms-python.python-2023.18.0/pythonFiles/printEnvVariables.py
2023-10-31 09:17:13.224 [info] shell: bash
2023-10-31 09:17:13.224 [debug] Shell Exec: . /home/juledwar/src/helios/.hatch/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python /home/juledwar/.vscode/extensions/ms-python.python-2023.18.0/pythonFiles/printEnvVariables.py with options: {
    "env": {
***REDACTED***
   }
    "shell": "bash",
    "timeout": 30000,
    "maxBuffer": 1000000,
    "throwOnStdErr": false,
    "encoding": "utf8"
}
2023-10-31 09:17:13.310 [info] > ~/src/helios/.hatch/bin/python ~/.vscode/extensions/ms-python.python-2023.18.0/pythonFiles/testing_tools/unittest_discovery.py ./helios test*.py
2023-10-31 09:17:13.310 [info] cwd: ~/src/helios
2023-10-31 09:17:14.391 [error] Error discovering unittest tests:
 Failed to import test module: spinach
Traceback (most recent call last):
  File "/usr/lib/python3.11/unittest/loader.py", line 452, in _find_test_path
    package = self._get_module_from_name(name)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/loader.py", line 362, in _get_module_from_name
    __import__(name)
  File "/home/juledwar/src/helios/helios/spinach/__init__.py", line 7, in <module>
    from spinach import job as spinach_job_mod
ImportError: cannot import name 'job' from partially initialized module 'spinach' (most likely due to a circular import) (/home/juledwar/src/helios/helios/spinach/__init__.py)

[eleanorjboyd]

@karrtikr, thoughts on the environment setup issue here?

[karthiknadig]

Should we also split this Issue into a separate one to track the sys.path problem?

Yes, a separate issue would be better to track that.

[bigjools]

Should we also split this Issue into a separate one to track the sys.path problem?

Yes, a separate issue would be better to track that.

I'm trying to report using vscode since I can't open here directly any more, and it's doing nothing when I hit the "Create on Github" button. :(

Do you have admin powers to duplicate this one?

[karthiknadig]

created this issue: #23239

[bigjools]

Thanks @karthiknadig . Was that testscenarios pointer enough to help? Sorry I can't do much more at the moment I am very short on free cycles.

[eleanorjboyd]

Hello! We have put in a fix for this issue and it should now be working! If you could try it on the newest version of the python extension nightly pre-release that would be great so we know if it works!

Happy coding!

[eleanorjboyd]

Hi! I started looking into the python path ordering issue and am not able to repro it on my side- I tried with both mac and windows and am seeing the path ordering stay the same minus the additions we make to the path for the plugin etc.

@karthiknadig any ideas here why it could be happening if I am not able to repro?

@bigjools are you able to send a minimal repro where you experience the path re-order issue as this could help me dig in. If you send the whole workspace folder as a zip I can also look through your settings.

[karthiknadig]

We can discuss the path ordering here: #23239

I am suspecting this has to do with activation experiment

[bigjools]

Hello! We have put in a fix for this issue and it should now be working! If you could try it on the newest version of the python extension nightly pre-release that would be great so we know if it works!

Happy coding!

How do I get the nightly pre-release? I tried the "pre-release" option in the extension but it's not fixed there.

[eleanorjboyd]

We are actually having some delays getting the newest changes out due to some other release issues. I will reach out when the changes are on the most recent pre-release and then you should be able to see them on your side. Thanks!

[bigjools]

@bigjools are you able to send a minimal repro where you experience the path re-order issue as this could help me dig in. If you send the whole workspace folder as a zip I can also look through your settings.

@eleanorjboyd Currently it only happens to me on a very large, very private repo, so I will have to see if I can make a smaller example somehow. It only shows up as a problem when I have modules that are the same name as 3rd party ones.

[eleanorjboyd]

Karthik also has some guesses on the path ordering so no worries on the minimal repro- I can reach out later if I need it later. Thanks!

[eleanorjboyd]

Hi! This should now be out on insiders- sorry for the delay!

[bigjools]

@eleanorjboyd I'm not familiar with the different types of release, is that the same as the pre-release setting in the extension? It has not updated beyond v2024.5.11021008 so far.

[karthiknadig]

You need version extension 2024.5.11141012, for that you need VS Code Insiders.

[connor4312]

Are there verification steps for this issue?

[eleanorjboyd]

17641-testscenarios.zip

1. load the above repo
2. go to test explorer
3. make sure the explorer looks like the above screenshot

[bigjools]

Confirming this is fixed, thank you very much, this is going to make a huge difference.