Unknown virtual env/environment (venv) is being queried for by Jupyter extension

[brettcannon]

Discussed in microsoft/vscode-python#20431

^{Originally posted by drorata December 25, 2022}
When opening a py file I see in the logs:

Experiment 'pythonaacf' is active
Experiment 'pythonSurveyNotificationcf' is active
LSP Notebooks experiment is enabled
LSP Notebooks interactive window support is enabled
Python interpreter path: ~/.poetry_envs/foo-cv-BPVTkgqj-py3.11/bin/python
> poetry env list --full-path
cwd: .
> ~/.pyenv/versions/3.10.4/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> ~/.pyenv/versions/3.11.1/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> conda info --json
> ~/.pyenv/versions/3.7.13/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> ~/.pyenv/versions/3.9.13/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> . ~/.poetry_envs/foo-cv-BPVTkgqj-py3.11/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/printEnvVariables.py
> ~/.poetry_envs/username-cv-lqqRfM4F-py3.11/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> ~/.poetry_envs/fast-api-sql-example-hJw5i7sx-py3.11/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> ~/.poetry_envs/foo-cv-BPVTkgqj-py3.11/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> ~/.poetry_envs/kumsitz-lib-iVmOjJvt-py3.10/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> ~/.poetry_envs/pyspark-playground-Dg6oNfQx-py3.10/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> ~/.poetry_envs/stocks-playground-bIdRsd_o-py3.10/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> /usr/bin/python3 ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> /opt/homebrew/bin/python3 ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> /opt/homebrew/bin/python3.11 ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
> ~/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
[ERROR 2022-11-23 8:14:35.526]: [Error: Command failed: /Users/username/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python /Users/username/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py /Users/username/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
/bin/sh: /Users/username/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python: No such file or directory

	at ChildProcess.exithandler (node:child_process:408:12)
	at ChildProcess.emit (node:events:526:28)
	at maybeClose (node:internal/child_process:1092:16)
	at Socket.<anonymous> (node:internal/child_process:451:11)
	at Socket.emit (node:events:526:28)
	at Pipe.<anonymous> (node:net:687:12)] {
  killed: false,
  code: 127,
  signal: null,
  cmd: '/Users/username/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python /Users/username/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py /Users/username/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py'
}
Starting Pylance language server.
Extension search path: /Users/username/.vscode/extensions
> ~/.poetry_envs/foo-cv-BPVTkgqj-py3.11/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/linter.py -p flake8 /opt/homebrew/bin/flake8 --max-line-length=120 ./foo_cv/__init__.py
cwd: .
##########Linting Output - flake8##########


> ~/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
[ERROR 2022-11-23 8:14:37.540]: [Error: Command failed: /Users/username/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python /Users/username/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py /Users/username/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py
/bin/sh: /Users/username/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python: No such file or directory

	at ChildProcess.exithandler (node:child_process:408:12)
	at ChildProcess.emit (node:events:526:28)
	at maybeClose (node:internal/child_process:1092:16)
	at ChildProcess._handle.onexit (node:internal/child_process:302:5)] {
  killed: false,
  code: 127,
  signal: null,
  cmd: '/Users/username/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python /Users/username/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/get_output_via_markers.py /Users/username/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/interpreterInfo.py'
}
> /opt/homebrew/bin/black --diff --quiet ./foo_cv/__init__.py.fcaa6f612d7121d453663cbc59080266.tmp
cwd: .
Extension search path: /Users/username/.vscode/extensions
> ~/.poetry_envs/foo-cv-BPVTkgqj-py3.11/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/linter.py -p flake8 /opt/homebrew/bin/flake8 --max-line-length=120 ./foo_cv/__init__.py
cwd: .
##########Linting Output - flake8##########


> ~/.poetry_envs/foo-cv-BPVTkgqj-py3.11/bin/python ~/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
cwd: .

I totally don't understand where and why vscode uses the environment /Users/username/.local/share/virtualenvs/test-project-GV0hGqPR? It might be related to the bug reported here microsoft/pylance-release#3784.

[karrtikr]

@drorata Can you please provide the debug logs? Set python.logging.level to debug in User settings, reload window, then perform the steps to reproduce before providing the logs.

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

VSCode looks for environments under /Users/username/.local/share/virtualenvs, so can you demonstrate that the directory contents of /Users/username/.local/share/virtualenvs/test-project-GV0hGqPR/bin does not contain the python file?

Please delete the directory /Users/username/.local/share/virtualenvs/test-project-GV0hGqPR and error should go away.

[drorata]

Here are the logs using the debug level.

The directory /Users/username/.local/share is empty for me.

[karrtikr]

Some other extension is asking the Python extension about this environment /Users/username/.local/share/virtualenvs/test-project-GV0hGqPR/bin/python, I highly suspect that extension is Jupyter. Do you have Jupyter installed, if so can you disable it and see if you still notice this path in the logs?

[drorata]

@karrtikr Spot on! Disabling the Jupyter extension of vscode eliminates the reference to the aforementioned virtualenv. Now the question is how to keep the extension enabled and avoid this problem?

[karrtikr]

That's a question for the Jupyter team, so I'm transferring your issue to Jupyter. I think Jupyter has a clear cache command, perhaps that should help clear list of stored kernels.

[DonJayamanne]

Please could you enable logging as follows:

• Open settings & go into Jupyter->Logging
• Change the value to verbose
• Reload VS Code,
• Attempt to repro the issue & then copy the output from the Jupyter output panel.

[drorata]

Thanks for the suggestion @DonJayamanne. I inspected the logs and I made me do the following:

In the logs, I found something like Updating cache with Local kernels startUsingLocalKernelSpec which made me look for the old virtual environment in the Jupyter's directory. I found reference to this old environment in ~/Library/Jupyter/kernels/test-project/kernel.json. Next I deleted the whole directory ~/Library/Jupyter/kernels/test-project/.

Otherwise, I scanned all the locations indicated by jupyter --paths and found no further mentioning of the virtual environment test-project-GV0hGqPR.

After that, I restarted vscode and opened a notebook. The logs still contain a reference to the environment. You can find the new logs in this gist.

[DonJayamanne]

@drorata
Please could you try the following:

• Add a setting into your user settings file as follows
  "jupyter.development": true,
• Reload VS Code
• Look for the command Jupyter (Dev): Clear Cache
• Try that a few times and reload vscode
• Check the logs again, and if the issue persists please could you share the logs again

[drorata]

I fail to do what you ask. When trying to adjust the setting it looks like so:

Note, it is is "pale".

When looking then, after reload for some "cache" related command, all I see are:

[DonJayamanne]

Sorry, you are right that will not work.
Could you instead please try the following:

• Add the following setting into your settings file

"notebook.kernelPicker.type": "mru",

• Reload VC Code
• Go into the kernle picker, and select Select Another Kernel -> Local Jupyter Kernel
• On the top right hand corner of the quick pick click the Refresh icon
• Wait for everything to get updated,
• Reload VS Code and then let me know if that still appears in the logs
• If it still appears there, pelase upload the new logs (after refreshing and re-loading vscode)

[drorata]

Step 1: I added "notebook.kernelPicker.type": "mru", to my user's settings.json and then reloaded (=quit and open again).

Step 2: I opened a notebook and clicked the Select Another Kernel. In there I didn't have the option Local Jupyter Kernel; the closest is Local Kernel Specs...

I clicked this option.

Step 3: In there, the offending kernel was present:

It was still there also after refreshing the list.

Step 4: Here are the new logs.

[DonJayamanne]

Thanks again for the logs and for your patience,
please could you install VS Code insiders and the latest pre-release version of the Jupyter extension.
And try that once again.
I believe this issue should be resolved in the latest pre-release version (which currently only runs in vscode insiders).

[drorata]

I installed the insider version, installed the Jupyter extension. Closed and opened the editor, open the Jupyter output and opened a notebook. The offending virtual environment was not to be found in the logs!

[DonJayamanne]

The offending virtual environment was not to be found in the logs!

Thanks for getting back, this confirms the bug has been fixed and will be shipped with the next release.