Project's folder is not added to sys.path when creating/loading a project with Spyder 5.3.2

[DanielReyesLastiri]

Issue Report Checklist

• Searched the issues page for similar reports
• Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• Could not reproduce inside jupyter qtconsole (if console-related)
• Tried basic troubleshooting (if a bug/error)
  • Restarted Spyder
  • Reset preferences with spyder --reset
  • Reinstalled the latest version of Anaconda
  • Tried the other applicable steps from the Troubleshooting Guide
• Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

Using Spyder 5 in a miniconda environment in Windows 10, Spyder does not add the folder to sys.path when loading or creating a project. Therefore, I cannot import functions from other files as expected.
Uninstalling miniconda, manually removing leftover folders (like ~.spyder-py3), and starting from a fresh miniconda installation does not help.

What steps reproduce the problem?

1. Fresh install of miniconda latest.
2. Create and activate Python 3.9 environment: conda create -n myenv python=3.9, conda activate myenv
3. Install numpy, scipy, matplotlib, and spyder in myenv: conda install package (no version specified)
4. Create a new project or load an existing one.
5. Can't import packages from folders in project. Check sys.path. Folder is not there.

What is the expected output? What do you see instead?

sys.path should include the project folder. It works in Ubuntu 20, but not in two different machines I've tried with Windows 10.

The command window returns an error:
zmq.error.ZMQError: not a socket

Versions

• Spyder version: 5.2.2
• Python version: 3.9.12 64-bit
• Qt version: 5.15.12
• PyQt version: PyQt5 5.15.7
• Operating System name/version:

Dependencies

# Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  2.0.0 (OK)
cookiecutter >=1.6.0          :  1.7.3 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.6.0;<8.0.0        :  7.31.1 (OK)
jedi >=0.17.2;<0.19.0         :  0.18.1 (OK)
jellyfish >=0.7               :  0.9.0 (OK)
jsonschema >=3.2.0            :  4.4.0 (OK)
keyring >=17.0.0              :  23.4.0 (OK)
nbconvert >=4.0               :  6.4.4 (OK)
numpydoc >=0.6.0              :  1.4.0 (OK)
paramiko >=2.4.0              :  2.8.1 (OK)
parso >=0.7.0;<0.9.0          :  0.8.3 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.9.0 (OK)
pygments >=2.0                :  2.11.2 (OK)
pylint >=2.5.0                :  2.14.5 (OK)
pyls_spyder >=0.4.0           :  0.4.0 (OK)
pylsp >=1.3.2;<1.4.0          :  1.3.3 (OK)
pylsp_black >=1.0.0           :  1.0.0 (OK)
qdarkstyle =3.0.2             :  3.0.2 (OK)
qstylizer >=0.1.10            :  0.1.10 (OK)
qtawesome >=1.0.2             :  1.0.3 (OK)
qtconsole >=5.2.1;<5.3.0      :  5.2.2 (OK)
qtpy >=1.5.0                  :  2.0.1 (OK)
rtree >=0.9.7                 :  0.9.7 (OK)
setuptools >=49.6.0           :  63.4.1 (OK)
sphinx >=0.6.6                :  5.0.2 (OK)
spyder_kernels >=2.2.1;<2.3.0 :  2.2.1 (OK)
textdistance >=4.2.0          :  4.2.1 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  2.1.6 (OK)
zmq >=17                      :  23.2.0 (OK)

# Optional:
cython >=0.21                 :  None (NOK)
matplotlib >=2.0.0            :  3.5.2 (OK)
numpy >=1.7                   :  1.21.5 (OK)
pandas >=1.1.1                :  None (NOK)
scipy >=0.17.0                :  1.7.3 (OK)
sympy >=0.7.3                 :  None (NOK)

[DanielReyesLastiri]

It seems to be an issue of backslash (used in Windows directories). When I enter the PYTHONPATH manager, the directory of my project folder has forward slashes. When I add manually my project folder and synchronize, the directory has backslashes, and it is then added to sys.path.
(I though Windows had solved that issue a while back, but I wouldn't be surprised if MS broke it again).

[ccordoba12]

Hey @DanielReyesLastiri, thanks for reporting. I think this error is already fixed in our latest version (5.3.2).

Since it's still not available in Anaconda, you have two options:

1. Use our Windows installer, which you can find here.

2. Create a new environment with conda-forge packages. For that, please close Spyder, open the Anaconda Prompt and run the following commands there:

    conda create -n spyder-cf -c conda-forge spyder
    conda activate spyder-cf
    spyder
   

Please do that and let us know if it works for you.

[DanielReyesLastiri]

Thanks for the quick reply. Spyder 5.3.2 via the conda-forge channel partly solves the problem. The directory is still not added to sys.path, but it is now added with backslashes to the PYTHONPATH manager. For a strange reason, I just need to add a second folder to the PYTHONPATH, and then both get imported to sys.path (the intended project folder already listed, and the second added manually).

[ccordoba12]

Ok, thanks for checking that @DanielReyesLastiri. I can reproduce this problem by doing the following:

• Resetting Spyder preferences.
• Start Spyder
• Opening a project.
• Checking sys.path in a console and noticing that the project's path was not added to sys.path.

@mrclary, please take care of this one because it looks related to the changes you did in PR #17511.

[mrclary]

The apparent cause for this issue seems to be that upon opening a new project, the signal to update the paths is received by the existing IPython Console, but then the signal is sent to restart the console and upon restart the SpyderKernelSpec only retrieves the spyder_pythonpath in the configuration, which is up-to-date but does not carry the project path.

This issue is present in the 5.3.1 artifact, which predates both #17512 (updating PYTHONPATH) and #17408 (updating bootstrap), so I'm not sure when it was introduced. Nevertheless, I think we have a few options:

1. In the projects plugin we can emit the sig_pythonpath_changed after restarting consoles. If sig_pythonpath_changed is emitted after starting consoles in all situations, then SPY_PYTHONPATH could be eliminated from spydercustomize.py altogether.
2. We could combine the project paths with spyder_pythonpath when assigning SPY_PYTHONPATH in SpyderKernelSpec. SpyderKernelSpec does not have access to the main window's get_spyder_pythonpath_dict, which performs this sythesis, so SPY_PYTHONPATH may need to be updated in get_new_kernel in the ipythonconsole main_widget.

@spyder-ide/core-developers, I favor the first option; does anybody have a different suggestion?

[ccordoba12]

In the projects plugin we can emit the sig_pythonpath_changed after restarting consoles

I don't think this would work because we need for the kernel to be up in order to call any comm handler on it. So, If we simply emit sig_pythonpath_changed after calling IPythonConsole.restart, it wouldn't have any effect (that's what I concluded when working on PR #19068).

If sig_pythonpath_changed is emitted after starting consoles in all situations, then SPY_PYTHONPATH could be eliminated from spydercustomize.py altogether

I don't think that's a good idea for something that has been in place for years and just to solve a bug. Besides, there's still the problem of issue #18942, which we need to address somehow for 5.4.0 and for which we could still need SPY_PYTHONPATH.

We could combine the project paths with spyder_pythonpath when assigning SPY_PYTHONPATH in SpyderKernelSpec

I think this is how things were working before, isn't it?

SpyderKernelSpec does not have access to the main window's get_spyder_pythonpath_dict

Wasn't Projects emitting a signal before, which has captured by the main window and added the project's path to spyder_pythonpath? I know that was convoluted, but it was working without issues.

In conclusion, I'm in favor of option 2. for safety, but I'm really not happy with all these regressions when things were working perfectly fine before.

[ccordoba12]

One more thing: I think option 2. could be implemented in a simpler fashion if we make Projects write its current path to a config option and then we retrieve it in SpyderKernelSpec when building the contents of SPY_PYTHONPATH.

Of course, this is a workaround until we have a proper PPM plugin that can handle the requested changes to PYTHONPATH or sys.path from different plugins, but we can't do better for now.

[CAM-Gerlach]

In conclusion, I'm in favor of option 2. for safety, but I'm really not happy with all these regressions when things were working perfectly fine before.

I'm not sure what specifically you're referring to by "all these regressions", given at least the proximate issue in #18942 was an outstanding bug that predates @mrclary 's changes, and the user's broader issue was the fully intended, expected and agreed result of the change. Could you clarify?

[mrclary]

In the projects plugin we can emit the sig_pythonpath_changed after restarting consoles

I don't think this would work because we need for the kernel to be up in order to call any comm handler on it. So, If we simply emit sig_pythonpath_changed after calling IPythonConsole.restart, it wouldn't have any effect (that's what I concluded when working on PR #19068).

Hmm... That's interesting. I'd like to hear more about that. When I tested emitting the signal after console restart, it seemed to work just fine. However, I can understand how that could depend on how long it takes the console to startup and therefore could be unstable.

If sig_pythonpath_changed is emitted after starting consoles in all situations, then SPY_PYTHONPATH could be eliminated from spydercustomize.py altogether

I don't think that's a good idea for something that has been in place for years and just to solve a bug.

I didn't mean to suggest that we would eliminate SPY_PYTHONPATH for this bug fix; only that if (in the future) we emitted a python path change signal after all console creations, then it would not be needed. If we chose option 1, it would just be moving toward that end.

Besides, there's still the problem of issue #18942, which we need to address somehow for 5.4.0 and for which we could still need SPY_PYTHONPATH.

Resolving #18942 does not require SPY_PYTHONPATH; that is just the current mechanism for injecting paths into the console at startup. Sending that stuff to the kernel via comm is how it is updated while running. #18942 just requires adding PYTHONPATH to the list of paths sent to the console, whatever mechanism that happens to be.

We could combine the project paths with spyder_pythonpath when assigning SPY_PYTHONPATH in SpyderKernelSpec

Yes, that is the essence of option 2.

I think this is how things were working before, isn't it?

I assume that must have been the case at some point in the past, I just don't know when that was. I checked as far back as 5.3.1, before changes to bootstrap and PYTHONPATH handling, and this issue was there. I know that we have two path "configuration" mechanisms floating around: spyder_pythonpath in the modern configuration scheme; and path/not_active_path files. I suspect that this may be complicating the issue.

I'm not trying to shift blame away from myself; if any of my work has caused this problem, I apologize. But at this point, I just don't know when this issue was actually created.

SpyderKernelSpec does not have access to the main window's get_spyder_pythonpath_dict

Wasn't Projects emitting a signal before, which has captured by the main window and added the project's path to spyder_pythonpath? I know that was convoluted, but it was working without issues.

I'm not familiar with what signals were captured by main window from projects in the past, but right now I don't believe this is the case. As far as I can tell, on opening a project, Projects emits its own sig_pythonpath_changed which is connected directly to main.pythonpath_changed method. From here all the correct paths are assembled from path, not_active_path, and project path, and sent to the consoles; but project path has never been added to spyder_pythonpath (modern config) that I know of.

[ccordoba12]

I'm not sure what specifically you're referring to by "all these regressions", given at least the proximate issue in #18942 was an outstanding bug that predates @mrclary 's changes

If I understand things correctly, our usage of PYTHONPATH was inconsistent in several ways and that's what PR #17512 tried to address. However, I'd hardly say that was an "outstanding bug" because I don't remember users complaining about it. In other words, it was quite ugly that PYTHONPATH was such a mess in Spyder, but that's a power user feature that not many people know or use frequently (otherwise, our mess would have been pointed out loudly by users, like so many other inconsistencies in Spyder).

And regressions are issue #18942 and this one, which have been reported by users because we broke how things have been working for them for a very long time.

and the user's broader issue was the fully intended, expected and agreed result of the change

Sure, it was agreed by the team, but I doubt we were aware that we will break PYTHONPATH and sys.path in the ways we did. I certainly wasn't and I don't like it. Sorry if I sound harsh about it, but I think this is the dilemma and burden of any maintainer: why do we have to spend time fixing these regressions if things weren't broken before and practically no one was asking to change them?

[ccordoba12]

Hmm... That's interesting. I'd like to hear more about that. When I tested emitting the signal after console restart, it seemed to work just fine. However, I can understand how that could depend on how long it takes the console to startup and therefore could be unstable.

Yeah, on Windows the first console takes a long time to start. So we need to send comm requests only when the prompt is ready so we can be sure they are not missed.

I didn't mean to suggest that we would eliminate SPY_PYTHONPATH for this bug fix; only that if (in the future) we emitted a python path change signal after all console creations, then it would not be needed

As you mentioned, we have two configuration mechanisms for kernels: using env vars, which is necessary to set up the kernel before it's created; and sending comms, which you can use when the kernel is running. So we can keep using SPY_PYTHONPATH to set up PYTHONPATH at kernel startup with the current paths in PPM. That has been working just fine for many years and I don't see any need to change it. For the second case you mention, i.e. when paths are changed in PPM, we're already using a comm handler.

#18942 just requires adding PYTHONPATH to the list of paths sent to the console, whatever mechanism that happens to be.

Sure, but the idea is not make more potentially breaking changes now. In addition, I wouldn't like to run too many things when the prompt is ready because it could lead to delays when creating the first console (specially if there's a way to send them through env vars, which is already in place).

As far as I can tell, on opening a project, Projects emits its own sig_pythonpath_changed which is connected directly to main.pythonpath_changed method. From here all the correct paths are assembled from path, not_active_path, and project path, and sent to the consoles; but project path has never been added to spyder_pythonpath (modern config) that I know of.

Ok, so why don't we add the project's path to spyder_pythonpath in MainWindow? That should be another way to solve this, right?

[mrclary]

So we can keep using SPY_PYTHONPATH to set up PYTHONPATH at kernel startup with the current paths in PPM. That has been working just fine for many years and I don't see any need to change it.

That's fine. I thought that you had suggested that we should get rid of it some day. But perhaps I'm mistaken. (#18308 (comment))

[mrclary]

Ok, so why don't we add the project's path to spyder_pythonpath in MainWindow? That should be another way to solve this, right?

Yes, I think so; and I think that is the root of the problem.

It appears that main.save_python_path intends to update spyder_pythonpath with active paths from path, not_active_path, and project path, but this never gets called when projects is updated. The reason is that main.pythonpath_changed (only called when projects are opened/closed) updates main.project_path attribute, calls main.get_spyder_pythonpath_dict (which will now have updated project path) and sends this to main.update_python_path. However, main.update_python_path calls main.get_spyder_pythonpath_dict, expecting it to be an "old" path list (but, oops, it's already been updated) to compare to this "new" path list, and only calls main.save_python_path if they are different. Thus, main.pythonpath_changed unwittingly ensures that main.save_python_path is not called and spyder_pythonpath will never have project paths.

I'll make a new proposal shortly that ensures spyder_pythonpath is properly updated with the project path.

[ccordoba12]

That's fine. I thought that you had suggested that we should get rid of it some day

Hehe, no no, I thought you were suggesting to remove it. I'm very sorry for the confusion and also for venting my frustration here, but we have hundreds of bugs to fix, so I get angry when we introduce more that also break long-standing functionality (without intention, of course, but we still need to pay attention to them).

This also shows that we need more tests to catch regressions like this one in the future.

I'll make a new proposal shortly that ensures spyder_pythonpath is properly updated with the project path.

Great! Sounds like a good plan.

[DanielReyesLastiri]

Thanks for the follow-up everyone!