Discover PyTest tests only when necessary

[logstar]

Environment data

• VS Code version: 1.22.2 (1.22.2) 3aeede733d9a3098f7b4bdc1f66b63b0f48c1ef9
• Extension version (available under the Extensions sidebar): 2018.3.1
• OS and version: OSX 10.13.4
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.4
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Relevant/affected Python packages and their versions: n/a

Actual behavior

Test discovery automatically runs after saving edits of a test file.

Although this behavior sounds natural, it causes a spike of energy consumption on every save of an edited test file. One might just want to save it without intending to run test immediately afterwards, e.g. for linting. The fan of my computer starts howling after a few saves.

In OSX activity monitor, energy impact changes from ~1 to ~50 after saving an edited test file, and the energy impact stays around 50 for about 3 seconds. I have 22 tests in the file I saved and 245 tests in total.

Related to issue #1474 and #1132.

Expected behavior

Only run test discovery after giving a command to run or clicking Run Test or Debug Test.

Steps to reproduce:

1. Edit a test file.
2. Save the edited test file.
3. Go to step 1 for a few more times until the fan starts howling. This takes 5 times on my computer.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

============================= test session starts ==============================
platform darwin -- Python 3.6.4, pytest-3.4.2, py-1.5.2, pluggy-0.6.0 -- /XXXXXXXX
cachedir: .pytest_cache
Matplotlib: 2.2.2
Freetype: 2.8.1
rootdir: /XXXXXX, inifile: setup.cfg
plugins: xdist-1.22.2, mpl-0.9, forked-0.2, cov-2.5.1
collecting ... collected 245 items

<<Collected tests>>

========================= no tests ran in 2.89 seconds =========================

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

<<Empty>>

Question

Any pointer on how test discovery works in vscode-python?

[brettcannon]

@logstar the code resides in https://github.com/Microsoft/vscode-python/tree/master/src/client/unittests/pytest/services where you can find the commands we are using and how we parse the output. We could probably stand to be smarter about how we refresh the test discovery mechanism.

[logstar]

@brettcannon Thank you for your attention to this issue and the pointer to the test discovery service.

After exploring the repository for a while, I think I will not be able to help on this issue.

Just for learning purpose, I was wondering if you could briefly explain how a test discovery service is triggered after saving a test file.

[DonJayamanne]

how a test discovery service is triggered after saving a test file.

We add a callback for when a document gets saved onDidSaveTextDocument

The handler onDocumentSaved is what triggers the test discovery.

[logstar]

@DonJayamanne Thank you for your explanation!

Just paste here some additional references for anyone who is also interested:

VS code api: https://code.visualstudio.com/docs/extensionAPI/vscode-api . The event onDidSaveTextDocument is mentioned in workspace -> EVENTS.

Overview of VS code extension api: https://code.visualstudio.com/docs/extensionAPI/vscode-api

[brettcannon]

Providing a feature to disable automatic test discovery seems reasonable.

[brettcannon]

This was implemented in the latest release: https://github.com/Microsoft/vscode-python/blob/master/CHANGELOG.md#enhancements