unittest_discovery evaluates __init__ methods

[davidjayb]

Type: Bug

Behaviour

When unittest_discovery runs all class init methods are evaluated. I'm not sure how this can be remediated but it can result in difficult to troubleshoot situations.

I ran into this issue while looking into getting Selenium tests running in my Django application. The fix was to use a @classmethod setup method instead which is better practice but figuring out the problem was difficult. It was launching Firefox instances without a clear reason as to where it was spawning from:

$ pstree -p <pid>
-+= 00001 root /sbin/launchd
 \-+- <pid> <user> /Applications/Firefox.app/Contents/MacOS/firefox-bin --marionette --headless --remote-debugging-port 55769 --remote-allow-hosts localhost -foreground -no-remote -profile <profile>

Expected vs. Actual

I am not sure what the solution to this could be. If you can avoid evaluating the __init__ method that would solve the problem. If that isn't possible then the documentation should be updated to warn about this behaviour in test discovery:

https://code.visualstudio.com/docs/python/testing#_test-discovery

Steps to reproduce:

I'll present the reproduction steps with the assumption that a Django application has been installed.

1. Create a settings.json at <project_root>/.vscode/settings.json

{
    "python.testing.unittestArgs": [
        "-v",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

2. Install the selenium and geckodriver package

pip install selenium

# download and install the geckodriver from https://github.com/mozilla/geckodriver/releases

3. Create a test class that has an init method with side effects:

from selenium import webdriver

from django.contrib.staticfiles.testing import StaticLiveServerTestCase

class SeleniumLiveServerTestCase(StaticLiveServerTestCase):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        options = webdriver.FirefoxOptions()
        options.add_argument("--headless")
        self.driver = webdriver.Firefox(
            options=options,
        )

4. Save the file and observe Firefox being launched in headless mode.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.8.13
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Pyenv
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~/.pyenv/versions/3.8.13/envs/<env>/bin/python ~/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/testing_tools/unittest_discovery.py . test_*.py
cwd: .

User Settings

venvPath: "~/.pyenv/versions/3.8.13/envs/<env>/bin/python"

languageServer: "Pylance"

formatting
• provider: "black"

testing
• unittestArgs: "[
        "-v",
        "-p",
        "test_*.py"
    ]"
• unittestEnabled: true

Extension version: 2023.6.0
VS Code version: Code 1.77.3 (Universal) (704ed70d4fd1c6bd6342c436f1ede30d1cff4710, 2023-04-12T09:19:37.325Z)
OS version: Darwin arm64 22.3.0
Modes:
Sandboxed: Yes

[eleanorjboyd]

Hello! Unittest does all the actual evaluation and running not our extension. In general, we suggest you keep code out of the init file. Which part of the docs do you think needs to be updated?

[davidjayb]

Hello @eleanorjboyd thank you for reviewing, it is good to know that this is due to the Python unittest framework:
https://github.com/microsoft/vscode-python/blob/main/pythonFiles/testing_tools/unittest_discovery.py
https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover

I was thinking a note under the test discovery section could be helpful:
https://code.visualstudio.com/docs/python/testing#_test-discovery

Perhaps something similar to this "tip" but suggesting the use of setup methods instead of overriding the init method:

Tip: Sometimes tests placed in subfolders aren't discovered because such test files cannot be imported. To make them importable, create an empty file named init.py in that folder.