Test discovery fails with pytest-libfaketime

[giallu]

Environment data

VS Code Version: 1.54.2
Commit: fd6f3bce6709b121a895d042d343d71f317d74e7
Date: 2021-03-11T00:56:09.783Z (4 days ago)
Electron: 11.3.0
Chrome: 87.0.4280.141
Node.js: 12.18.3
V8: 8.7.220.31-electron.0
OS: Darwin x64 19.6.0

• Extension version (available under the Extensions sidebar): v2021.2.636928669

• Python version (& distribution if applicable, e.g. Anaconda): 3.9.2

• Type of virtual environment used: pipenv

• Value of the python.languageServer setting: Jedi

Expected behaviour

Tests are correctly discoverd

Actual behaviour

Test discovery fails with when pytest-libfaketime is installed

Steps to reproduce:

I created a small repo to easily reproduce the issue:
https://github.com/giallu/vscodetest

1. clone the repo and create the virtual env with pipenv install
2. Discover tests using pytest and root set to . or animals
3. open the project in vscode: tests discovery and test running works
4. install pytest-libfaketime with pipenv install pytest-libfaketime
5. test discovery fails like this:

Test Discovery failed: 
SyntaxError: Unexpected end of JSON input

Logs

I am attaching here the Python Test Log window content; I tried in sequence: discover tests, run tests (install pytest-libfaketime), discover tests, run tests.

Interestingly, tests are a success before and after the library installation, it's only the discovery phase that is failing.

python /Users/giallu/.vscode/extensions/ms-python.python-2021.2.636928669/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/giallu/vscodetest -s --cache-clear .
============================= test session starts ==============================
platform darwin -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
django: settings: vscodetest.settings (from ini)
rootdir: /Users/giallu/vscodetest, configfile: pytest.ini
plugins: django-4.1.0
collected 1 item

animal/tests.py .                                                        [100%]

- generated xml file: /var/folders/j8/_m5rss0s475_92j2207sdhxh0000gn/T/tmp-23058OVmOvk3C0G3U.xml -
============================== 1 passed in 0.52s ===============================
python /Users/giallu/.vscode/extensions/ms-python.python-2021.2.636928669/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/giallu/vscodetest -s --cache-clear .
Test Discovery failed: 
SyntaxError: Unexpected end of JSON input
python /Users/giallu/.vscode/extensions/ms-python.python-2021.2.636928669/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/giallu/vscodetest -s .
Test Discovery failed: 
SyntaxError: Unexpected end of JSON input
re-exec with libfaketime dependencies
============================= test session starts ==============================
platform darwin -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
django: settings: vscodetest.settings (from env)
rootdir: /Users/giallu/vscodetest, configfile: pytest.ini
plugins: libfaketime-0.1.2, django-4.1.0
collected 1 item

animal/tests.py .                                                        [100%]

- generated xml file: /var/folders/j8/_m5rss0s475_92j2207sdhxh0000gn/T/tmp-230583iuiT9nNMlnD.xml -
============================== 1 passed in 0.55s ===============================

[kimadeline]

Hi @giallu, thank you for reaching out!

What is being printed when you run the discovery command directly in your terminal? python /Users/giallu/.vscode/extensions/ms-python.python-2021.2.636928669/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/giallu/vscodetest -s --cache-clear .

[giallu]

The output is empty when the library is installed

(vscodetest) ➜  vscodetest git:(master) ✗ python /Users/giallu/.vscode/extensions/ms-python.python-2021.2.636928669/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/giallu/vscodetest -s --cache-clear .
(vscodetest) ➜  vscodetest git:(master) ✗ pipenv uninstall pytest-libfaketime
Uninstalling pytest-libfaketime...
Found existing installation: pytest-libfaketime 0.1.2
Uninstalling pytest-libfaketime-0.1.2:
  Successfully uninstalled pytest-libfaketime-0.1.2

Removing pytest-libfaketime from Pipfile...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (536b9f)!
(vscodetest) ➜  vscodetest git:(master) ✗ python /Users/giallu/.vscode/extensions/ms-python.python-2021.2.636928669/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/giallu/vscodetest -s --cache-clear .
[{"rootid": ".", "root": "/Users/giallu/vscodetest", "parents": [{"id": "./animal", "kind": "folder", "name": "animal", "parentid": ".", "relpath": "./animal"}, {"id": "./animal/tests.py", "kind": "file", "name": "tests.py", "parentid": "./animal", "relpath": "./animal/tests.py"}, {"id": "./animal/tests.py::AnimalTestCase", "kind": "suite", "name": "AnimalTestCase", "parentid": "./animal/tests.py"}], "tests": [{"id": "./animal/tests.py::AnimalTestCase::test_animals_can_speak", "name": "test_animals_can_speak", "source": "./animal/tests.py:10", "markers": [], "parentid": "./animal/tests.py::AnimalTestCase"}]}]

[giallu]

Is there anything I need to do to drop the "needinfo" label? If @kimadeline is waiting for that before looking at this again, it could be a long wait 😅

[kimadeline]

No there is nothing for you to do, don't worry I haven't forgotten about you 🙂

[kimadeline]

Hi @giallu, thank you for your patience!

pytest-libfaketime writes in stdout, you can see it if you execute the discovery command without the adapter script from the extension:

(15669-pytest-libfaketime) python -m pytest --rootdir /path/to/my/folder/15669-pytest-libfaketime -s --cache-clear animal
re-exec with libfaketime dependencies 
====================================================== test session starts =======================================================
platform darwin -- Python 3.9.1, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
django: settings: vscodetest.settings (from env)
rootdir: /path/to/my/folder/15669-pytest-libfaketime, configfile: pytest.ini
plugins: libfaketime-0.1.2, django-4.1.0
collected 1 item                                                                                                                 

animal/tests.py .

======================================================= 1 passed in 0.39s ========================================================

The adapter relies on the output of this command to discover the tests, as such the re-exec with libfaketime dependencies line is causing the issue.

We have plans to address that with #11729, feel free to upvote it to help us bump up its priority! In the meantime, the only workaround I can suggest is to find a plugin that doesn't write to stdout.

Closing in favor of #11729.

[giallu]

Thank you. In the meanwhile I did exactly what you suggested and changed to https://github.com/adamchainz/time-machine that does not seem to suffer for the same issue.