Activated environments not resolving PYTHONPATH=./ to a fully qualified value

[DonJayamanne]

Assume you have a .env file with the following contents

PYTHONPATH=./

When you debug a Python file, then the env variable is resolved and the PYTHONPATH value is fully resolved to the full path of the workspace. I.e. Workspace path is now in the current Python path (sys.path)

However when Jupyter extension gets the activated env variables the above value is not resolved, as a result the workspace directory is not added to the PATH or PYTHONPATH value in the env variables.

Looks like Python ext only resolves this for debugging of python code and not when returning activated variables.

[karrtikr]

We had a discussion about this and it was mentioned that we should not resolve environment variables specified and simply send them as is, it should be up-to each tool on how to resolve it. Even the special treatment of PYTHONPATH that we currently do, was noted that it probably shouldn't have been done:

vscode-python/src/client/common/variables/environmentVariablesProvider.ts

Lines 117 to 119 in d17ebce

	if (this.process.env.PYTHONPATH) {
	this.envVarsService.appendPythonPath(mergedVars!, this.process.env.PYTHONPATH);
	}

However I do notice an inconsistency here from user's perspective, Pylance somehow resolves . in .env as workspace directory as so does debugpy.

When you debug a Python file, then the env variable is resolved and the PYTHONPATH value is fully resolved to the full path of the workspace. I.e. Workspace path is now in the current Python path (sys.path)

@int19h Can you help with how this happens, does debugpy resolves this or is workspace path added in sys.path due to some other reason?

@karthiknadig Any idea how Pylance is able to resolve it?

[karthiknadig]

@debonte How does pylance resolve relative paths in environment variables? especially if PYTHONPATH=./ does it expand the .?

[int19h]

debugpy doesn't know anything about workspaces. However, if you have a launch configuration with "program":"/foo/bar/baz.py" and no "cwd", it will assume that "cwd" should be "/foo/bar". Thus having . in PYTHONPATH works in practice in this scenario.

If the path is not absolute or you use "module", it will not do anything special, so all relative paths will be resolved depending on what the current directory is for the terminal running the debuggee.

[debonte]

How does pylance resolve relative paths in environment variables? especially if PYTHONPATH=./ does it expand the .?

We only support resolving a few environment variables in paths (ex. HOME, USERNAME, VIRTUAL_ENV), but yes we do support relative paths there.

The PYTHONPATH environment variable is handled differently. We don't support it in all paths. We just check if it's set (via the python._envPYTHONPATH config setting) and if so, add it to our extraPaths. Although we support relative paths in PYTHONPATH (ex. ./foo), it appears that ./ gets incorrectly resolved to \\ rather than the workspace root. This is also true if you directly set "python.analysis.extraPaths": ["./"] though. I'm not sure we care about that scenario since I don't think it makes sense to add the workspace root to extraPaths.

Adding @heejaechang since, coincidentally, he was also looking at our code that handles the PYTHONPATH env var today.

[karrtikr]

Thank you both, as each tool/extension can have it's own cwd or preference to resolve relative paths such as ./, Python extension cannot always resolve the ./ to workspace root.

@DonJayamanne Based on this I think Jupyter extension should probably set the cwd when executing using the variables and paths can be resolved depending on what the current directory is, similar to how debugger does it. Closing this.

[heejaechang]

we get python search path from running python interpreter and ./ will be resolved to whatever python interpreter think current working folder is at the time, and in vscode/pylance, that's where our server starts which is .vscode\extensions\ms-python.vscode-pylance-xxxx\dist

but to help user to have similar impact, our import resolver will always try to resolve import at current folder if no module gets resolved with the name.

ex) /lib/lib1.py and /lib/nested/lib2.py and /lib/nested/test.py

in /lib/nested/test.py, one can refer lib2 as import lib.nested.lib2 or import lib2, this is to support a case where user runs test.py directly at /lib/nested and python will put /lib/nested in search path. our normal workspace set up with vscode can't support this since we will put only / as root. so we do the heuristic to support those cases.

[DonJayamanne]

@karrtikr
See here

When you debug a Python file, then the env variable is resolved and the PYTHONPATH value is fully resolved to the full path of the workspace. I.e. Workspace path is now in the current Python path (sys.path)

The Jupyter extension does not know what is in PYTHONPATH (in .env file), hence we cannot add the path value.
Basically, from an end user perspective, when running Python code using the debugger (run with debugging or run without debugging), then Python extension will honour the value in PYTHONPATH found in the .env file.

However when Jupyter tries to get the same env varaibles for running python code outside the pytyhon extension, then pytohn extension does not honour the PYTHONPATH in .env file

Re-Opening this, as the Jupyter extension cannot add the values without knowing what is in it.
But the whole point of the API is for the python extension to return the same env variables that will be used by python extension when running python code.
but there is a descrepancy, hence this issue.