Automatically set pythonpath correctly for default project setups

[WolfgangFahl]

see vs-code Issue 203607 which pointed here

the current project directory "." should be part of the PYTHONPATH automatically and the necessary settings should be hidden and the user not to be forced to investigate the inner workings of vscode and the extension

see see https://stackoverflow.com/questions/53653083/how-to-correctly-set-pythonpath-for-visual-studio-code
some 1/4 million views so far approx 150 upvotes but no implementation in vscode so far.

Please improve to make python developers happy campers and not get ModuleNotFoundError: on trying to run the first piece of code or test - it is too frustrating when knowing that quite a few IDEs will do this out of the box with no such errors.

After this issue was closed last month i have now added it to the top of the list at Microsoft does not listen!

[brettcannon]

Can you be more specific about what issue you are having (i.e. what are you doing w/ the extension that is causing you to request this)? When we execute Python we purposefully don't manipulate it behind the scenes so it is more predictable for users.

[WolfgangFahl]

Take any of the python based github projects with a standard project layout:

• <some module>
• tests

check out the github project. Open it it with VSCode and try to run any of the code examples or tests.
You 'll get an error message immediately. No other IDE i have tried but VSCode behaves this way.All other IDE that i know will set the PYTHONPATH immediately and correctly to "." in respect to the root directory of the project. It is really that simple!

When we execute Python we purposefully don't manipulate it behind the scenes so it is more predictable for users.

Ok i'll admit that the failure behavior is predictable. I have never have seen this as an argument in the "is it a bug or is it a feature discussion" before :-)

[WolfgangFahl]

Personally it took me hours to figure out how the settings of VsCode might be set and i actually created a script for this.
Take the https://pypi.org/project/pyLodStorage/ example

git clone https://github.com/WolfgangFahl/pyLoDStorage

will check out this project. To run the TestSamples code from the command line you can do

# make sure all libraries are available (optional)
pip install . 
# set PYTHONPATH
export PYTHONPATH="."
# run a single unit test from the command line
python -m unittest tests.test_samples.TestSamples
Starting test test_json_file_operations, debug=False ...
test test_json_file_operations, debug=False took   0.5 s
.Starting test test_json_serialization, debug=False ...
test test_json_serialization, debug=False took   0.3 s
.Starting test test_load_from_yaml_url, debug=False ...
test test_load_from_yaml_url, debug=False took   0.2 s
.Starting test test_royals, debug=False ...
test test_royals, debug=False took   0.0 s
.Starting test test_samples_yaml_round_trip, debug=False ...
test test_samples_yaml_round_trip, debug=False took   0.6 s
.Starting test test_yaml_file_operations, debug=False ...
test test_yaml_file_operations, debug=False took   0.5 s
.
----------------------------------------------------------------------
Ran 6 tests in 2.146s

OK

With VSCode:

and then

Output:

But that is just an example that should not bring us away from the problem

[WolfgangFahl]

I tried a script as recommended by ChatGPT-4 - see below - which would even further distract us from the feature request stated here. The expectation is that things should work out of the box immediately with not fiddling or user interaction necessary. Dead simple setting the PYTHONPATH to the project directory per default.

The script gives a glimpse of the complexity that needs to be hidden using the Information Hiding principle. I'd appreciate if this 50 year old principle would be applied in this case to make the world a better place.

https://github.com/WolfgangFahl/pyLoDStorage/blob/master/scripts/vsenv

#!/bin/bash

# Stop on first error
set -e

# Define your project root directory here
PROJECT_ROOT="$(pwd)"
VENV_DIR="$PROJECT_ROOT/venv"

echo "Setting up Python virtual environment..."
# Check if the virtual environment directory exists
if [ ! -d "$VENV_DIR" ]; then
    # Create a virtual environment
    python3 -m venv "$VENV_DIR"
fi

# Activate the virtual environment
source "$VENV_DIR/bin/activate"

echo "Installing the project and its dependencies..."
# Install the project in editable mode along with its dependencies
pip install -e .

echo "Setting up .env file..."
# Create or overwrite the .env file
echo "PYTHONPATH=$PROJECT_ROOT" > "$PROJECT_ROOT/.env"

echo "Setting up VS Code configuration files..."
# Create .vscode directory if it doesn't exist
mkdir -p "$PROJECT_ROOT/.vscode"

# Create or overwrite settings.json
cat > "$PROJECT_ROOT/.vscode/settings.json" << EOF
{
    "python.pythonPath": "$VENV_DIR/bin/python",
    "python.envFile": "\${workspaceFolder}/.env",
    "python.testing.unittestEnabled": true,
    "python.testing.pytestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "test_*.py"
    ]
}
EOF

# Create or overwrite launch.json
cat > "$PROJECT_ROOT/.vscode/launch.json" << EOF
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "\${file}",
            "console": "integratedTerminal",
            "cwd": "\${workspaceFolder}",
            "env": {
                "PYTHONPATH": "\${workspaceFolder}\${pathSeparator}\${env:PYTHONPATH}"
            }
        },
        {
            "name": "Python: Debug Unittests",
            "type": "python",
            "request": "launch",
            "console": "integratedTerminal",
            "cwd": "\${workspaceFolder}",
            "env": {
                "PYTHONPATH": "\${workspaceFolder}\${pathSeparator}\${env:PYTHONPATH}"
            },
            "purpose": ["debug-test"],
            "justMyCode": true
        }
    ]
}
EOF

echo "Setup complete. Please restart VS Code."

[WolfgangFahl]

Beging able to debug the code immediately as in:

would be the goal.

[brettcannon]

Take any of the python based github projects with a standard project layout:

• <some module>
• tests

check out the github project. Open it it with VSCode and try to run any of the code examples or tests. You 'll get an error message immediately.

As someone who uses this layout I can assure you this is not typically a problem, otherwise I would have personally made sure it got fixed.

Did you want to focus on your testing issue at #22824 (comment) ? If so, we would need you to provide your .vscode/settings.json, what environment you selected (screenshot helps), and a bigger screenshot of the Output panel to see the selected channel. You can also follow the instructions at https://aka.ms/pvsc-bug to file a bug report w/ a lot of this info automatically included.

I will say, though, I got your tests to work. After creating a GitHub codespace and installing the stable version of our extension I ran our Create Environment command and created a virtual environment that including the "test" extra. I ran our test configuration and got the following .vscode/settings.json:

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "test*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

Test discovery initially failed, but that was due to requests not being listed as a dependency in pyproject.toml (I opened brettcannon/pyLoDStorage#1 to fix that):

2024-02-05 18:11:56.102 [error] Unittest test discovery error for workspace:  /workspaces/pyLoDStorage 
 Failed to import test module: test_queries
Traceback (most recent call last):
  File "/home/codespace/.python/current/lib/python3.10/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/codespace/.python/current/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/workspaces/pyLoDStorage/tests/test_queries.py", line 22, in <module>
    from lodstorage.querymain import QueryMain
  File "/workspaces/pyLoDStorage/./lodstorage/querymain.py", line 23, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

But after I installed requests I was able to get the tests discovered and have them run:

As you can see from the settings.json I used, there was no need to change PYTHONPATH. As such, I'm not sure what about your setup is causing you such issues w/ our extension.

[WolfgangFahl]

@brettcannon - thank you for looking into this. And sorry for having provided an example that has an incomplete pyproject.toml. The project's details are IMHO irrelevant for this feature request and even this broken project shows what the problem is because the fact that the pyproject.toml is broken should not influence the capability to run tests. The tests will happily show that there is a problem themselves.

My Feature request ist about not having to update to the lastest version of the Extension and not have to run "Create Environment" at all. In the past few month i had multiple attempts with multiple people mostly in the MacOS context where VsCode would not run python tests out of the box. My Request is about making this possible as soon as a project directory is openend and it's feasible to deduct the project structure from the existence of a "tests" directory and the python unit tests within.

[brettcannon]

the fact that the pyproject.toml is broken should not influence the capability to run tests.

If you can't install the dependencies then the tests can't run because the imports fail. And test discovery also fails since both unittest and pytest import your code to discover tests. This led to the test explorer telling me that discovery failed and I looked at the failure which showed the import failure.

My Feature request ist about not having to update to the laste version of the Extension

We don't maintain old versions of the extension, so this isn't feasible if you rely on some new feature or bug fix.

not have to run "Create Environment" at all

I used that command out of convenience, not a requirement. I could have created the virtual environment and installed the dependencies manually if I wanted to.

My Request is about making this possible as soon as a project directory is openend and it's feasible to deduct the project structure from the existence of a "tests" directory and the python unit tests within.

Had you not said, "The project's details are IMHO irrelevant", I would have assumed you were asking us to try and minimize questions during test configuration and/or prompting the user to create a virtual environment to get dependencies installed before trying to discover tests. But discovering or running tests in the face of import errors is not possible w/o circumnavigating the test tools by doing code analysis and we already decided long ago that goes against our general policy of being a tool integrator (and can lead to false results if test creation is dynamic).

As such, I'm going to close this feature request. If I have misunderstood what you're asking for then please let me know and we can either clarify this issue or start a new feature request.

[WolfgangFahl]

You did misinterpret the request indeed. It is very disappointing that you came to the conclusion to close this request given that 250 thousand times other people were interested in exactly the same question. All other vendors implement by the need of people here at Microsoft the ticket is closed quickly.

Your assumptions on what the basic need here is IMHO much too technical. I can try to run any python test i want using the python command line no matter what the config is and I'll get any complaint but "module for test" not found if i set the PYTHONPATH correctly. I donot need test discovery because i can navigate to the file to test my self. I can usually click a run/debug button and the IDE will happily try to run/debug the test. Just VsCode won't do this out of the box without fiddling around and clicking before starting. I did not have any different experience yet. If i don't want to install new versions of extensions that simply means that if i download the latest version of VsCode and configure python as my environment that should be it and things should work out of the box. My experience still is it's not like that. If i am wrong let me know which version of VsCode is behaving as expected and I'll happily try it out and report here. Then its valid to close the ticket i opened because my experience is consistent with what is promised by the software (or expected by users).

[theloglizard]

Thank you for raising this, Wolfgang. As you note, you are far from alone.

I agree the basic question of managing the PYTHONPATH is too complex for something so fundamental. I'm running vscode to access a remote machine, and it's driving me crazy trying to figure out how to get the PYTHONPATH set.

The response reads to me as "it's not broken, just complicated let me work through the complexity with you". But that's not your point, at least as I'm reading it. I'm guessing the complexity comes from vscode needing to be all things to all people.

[brettcannon]

It is very disappointing that you came to the conclusion to close this request given that 250 thousand times other people were interested in exactly the same question.

I'm going to assume that number comes from Stack Overflow as we are not seeing equivalent asks (and Stack Overflow specifically asked us to not send users there for support).

You assumptions on what the basic need here is IMHO much too technical.

Then please explain exactly what you're after. You said "the current project directory "." should be part of the PYTHONPATH automatically". We are not going to do that as I said. VS Code views everything as a workspace, so effectively . is your workspace and so there's nothing to add from our perspective. And as I said we purposefully don't manipulate your Python environment w/o you asking, so we won't implicitly set PYTHONPATH behind your back.

Now if you want to change your ask to be something like, "I want an easier way to set PYTHONPATH and have everything that runs Python use it" then that's fine and we can either make that clarification here or you can open a new feature request. If this is specifically about testing then please be specific. I will also say that people typically set PYTHONPATH via a .env file or terminal.integrated.env.*, so we will ask why that isn't sufficient.

If that's not you're asking for then I still don't understand what you want. I got your example repo working w/o any PYTHONPATH manipulations, so I'm not seeing a specific issue there. The vast majority of our users "can usually click a run/debug button and the IDE will happily try to run/debug the test" w/o issue, so I'm not sure what you're trying to infer w/ that comment.

managing the PYTHONPATH is too complex for something so fundamental.

... to you. I'm not trying to be flippant, but if this was a constant ask we would have come up w/ a solution by now, so I don't think this is as fundamental to our users as you may think (as I said, most users use a .env file in general to manipulate environment variables). That doesn't mean we aren't willing to work on it, but I want to be clear we are not ignoring users on this as it simply does not come up as often as e.g. Django testing support. I'm also trying to not take the tone in this issue as suggesting we are ignoring users, but it's starting to feel that way (and that's after spending more effort on this issue than any other this week).

I'm guessing the complexity comes from vscode needing to be all things to all people.

That's a big part of it. The amount of project variance is huge, so trying to make everyone happy is extremely hard. Tack on the fact that everyone thinks their approach is "normal" and it adds on a bit of stress and exhaustion to our little team.

[theloglizard]

OK, and also not trying to be flippant, but is it possible your user base are simply the ones who did not select away from the tool because they could figure out how to set the PYTHONPATH (or hack it, as I have done) and there might be a fraction of the cited 250K visits who simply gave up? While I understand the squeaky wheel gets the grease, could there not also be a survivorship bias in the cited silence? You can tell me they are not the users you either are supporting or wish to support, and so be it. But silence is not necessarily satisfaction or approval.

The title of the oft-referenced stackoverflow article is:
"How to correctly set PYTHONPATH for Visual Studio Code"

and its large reference count suggests people are looking for, at least, help or better examples and possibly an accommodation or feature. That is how I stumbled on this thread, and for me it was the second Google recommendation after the MS document here: https://code.visualstudio.com/docs/python/environments#:~:text=To%20do%20so%2C%20open%20the,Settings%2C%20with%20the%20appropriate%20path.
which really isn't enough to figure out from what's there how to set PYTHONPATH. Ironically, if I search Bing, it sends me to the Stackoverflow page.

And so to your wonderfully specific point, perhaps people would be pleased with "an easier way to set PYTHONPATH". I certainly would. No idea how hard that would be or how much other neglected users (e.g. Django) would have to suffer as an opportunity cost. For example, there are some helpful ways to navigate kernels and environments which are pretty spiffy and don't require a lot of fiddling around with dotfiles. While maybe not addressing Wolfgang's particular issue (which seems perhaps specialized and not covered by the "easier way to set PYTHONPATH"), might some similar helper like you have for the kernels but for PYTHONPATH not be in order as opposed to the "everything's a workspace with a .env file". I guess there's also a lot of room between "sufficient" workspace and .env management (apparently the standard) and convenient. And at what cost.

And you are right, there are probably (at least) two issues here to disentangle.

I suspect, as a primarily Unix coder, I am used to my environment variables following me around with a sort of global scope, and I am unused to being constrained by my particular "workspace" . I run an editor, cd wherever, run whatever, and the three are not particularly bound. I can imagine there are disadvantages to that kind of recklessness, as well, and that's not really the style of an IDE which is managing a number of problems.

I probably belong in another thread, and there is no action for you from my perspective, as I have a hack. Apologies for the further confusion I have caused and thanks for your consideration.

[brettcannon]

OK, and also not trying to be flippant, but is it possible your user base are simply the ones who did not select away from the tool because they could figure out how to set the PYTHONPATH (or hack it, as I have done) and there might be a fraction of the cited 250K visits who simply gave up?

Possibly and I'm sure that's true for all things we don't support in a way that makes people happy. But we have enough work to do to not need to go to 3rd-party sites like Stack Overflow looking for more to do. 😅

perhaps people would be pleased with "an easier way to set PYTHONPATH".

Since I don't want to hijack this issue from @WolfgangFahl if that's not what he's asking for, did you want to open a feature request?

I suspect, as a primarily Unix coder, I am used to my environment variables following me around with a sort of global scope, and I am unused to being constrained by my particular "workspace" . I run an editor, cd wherever, run whatever, and the three are not particularly bound. I can imagine there are disadvantages to that kind of recklessness, as well, and that's not really the style of an IDE which is managing a number of problems.

Are you launching VS Code via code or as a desktop app? If it's the former then the overall process should be inheriting the environment and thus transparently be picked up when we run processes on your behalf.

Apologies for the further confusion I have caused and thanks for your consideration.

No worries! We are all just trying to figure out what's best for Python developers (within all of our time constraints).

[theloglizard]

Since you asked:

Are you launching VS Code via code or as a desktop app? If it's the former then the overall process should be inheriting the environment and thus transparently be picked up when we run processes on your behalf.

Oh, no -- I think it's much worse than that. I launch it on a Mac desktop and tunnel through to a Cloud linux instance and develop there. The miracle is that I can do it at all.
Thanks

[jimp-lxc]

I got here from the stackoverflow page. Please create an easy way to set PYTHONPATH, as suggested by others. This is the one issue stopping me from moving to vscode.

[tace]

It's VERY difficult to set Python working in VSC!

[brettcannon]

It's VERY difficult to set Python working in VSC!

I assume this is in regards to you needing to set PYTHONPATH? Otherwise this is too vague to take as constructive feedback.

[WolfgangFahl]

@brettcannon
You wrote: Otherwise this is too vague to take as constructive feedback.

I already offered to try things out again to check whether the current version of VsCode behaves as expected on MacOS - which version is the one you consider this issue may be closed?

[tace]

It's VERY difficult to set Python working in VSC!

I assume this is in regards to you needing to set PYTHONPATH? Otherwise this is too vague to take as constructive feedback.

Sorry, yes, setting pythonpath for any random python code I open from any repo. Many times we have repos where main langiage is something else and just some folders has some python code and tests. I never yet figured out how to set the pythonpath so that I could immediately run unit tests with pytest etc. whatever plugins. And python interepreter is found fine i.e. my python installation. And it's also many times unclear weather this pythonpath setting means to pointing where is the python interpreter or is it that I want certain folders to be included into pythonpath so that python will can import those.

[brettcannon]

You wrote: Otherwise this is too vague to take as constructive feedback

That comment wasn't directed at you, @WolfgangFahl .

I already offered to try things out again to check whether the current version of VsCode behaves as expected on MacOS - which version is the one you consider this issue may be closed?

I specified why I closed this issue in #22824 (comment) and that hasn't changed, so there's nothing to test.

setting pythonpath for any random python code I open from any repo. Many times we have repos where main langiage is something else and just some folders has some python code and tests. I never yet figured out how to set the pythonpath so that I could immediately run unit tests with pytest etc. whatever plugins. And python interepreter is found fine i.e. my python installation. And it's also many times unclear weather this pythonpath setting means to pointing where is the python interpreter or is it that I want certain folders to be included into pythonpath so that python will can import those.

People typically use a .env file (I covered that in #22824 (comment) ).

If people want a different mechanism to explicitly set PYTHONPATH than what is provided, please open a new feature request as the original ask of this issue was much more specific than that and it's led to a lot of cross-talk.