Spyder can't start pylsp on pip installations

[arian-f]

Problem Description

Spyder doesn't manage to start pylsp, even though it's installed.

in server_python_$PID.log:

C:\Program Files\Python310\pythonw.exe: No module named pylsp

(there's also a corresponding GUI message after 5 or so start attempts)

however, pylsp seems to be installed correctly in a python interpreter, I can import pylsp, plus when I copy'n'paste the command that should start the python language server, that works.

Therefore I suspect that the PYTHONPATH is set improperly inside spyder when attempting to start pylsp, without the pylsp directory

This is an installation on windows, for that matter. Pythonpath is:

sys.path
Out[2]: 
['C:\\Program Files\\Python310\\python310.zip',
 'C:\\Program Files\\Python310\\DLLs',
 'C:\\Program Files\\Python310\\lib',
 'C:\\Program Files\\Python310',
 '',
'%APPDATA%\\Python\\Python310\\site-packages',
'%APPDATA%\\Python\\Python310\\site-packages\\win32',
'%APPDATA%\\Python\\Python310\\site-packages\\win32\\lib',
'%APPDATA%\\Python\\Python310\\site-packages\\Pythonwin',
 'C:\\Program Files\\Python310\\lib\\site-packages',
'%APPDATA%\\Python\\Python310\\site-packages\\IPython\\extensions',
 '%USERPROFILE'%\\.ipython']

and pylsp is installed in %APPDATA%\\Python\\Python310\\site-packages

What steps reproduce the problem?

1. install python3.10.4
2. pip install spyder
3. launch spyder (in debug mode to get the error log, but happens without debug mode as well)

Versions

• Spyder version: 5.3.0 None
• Python version: 3.10.4 64-bit
• Qt version: 5.15.2
• PyQt5 version: 5.15.6
• Operating System: Windows 10

Dependencies

# Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  2.0.0 (OK)
cookiecutter >=1.6.0          :  1.7.3 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.31.1;<8.0.0       :  7.32.0 (OK)
jedi >=0.17.2;<0.19.0         :  0.18.1 (OK)
jellyfish >=0.7               :  0.9.0 (OK)
jsonschema >=3.2.0            :  4.4.0 (OK)
keyring >=17.0.0              :  23.5.0 (OK)
nbconvert >=4.0               :  6.5.0 (OK)
numpydoc >=0.6.0              :  1.2.1 (OK)
paramiko >=2.4.0              :  2.10.3 (OK)
parso >=0.7.0;<0.9.0          :  0.8.3 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.9.0 (OK)
pygments >=2.0                :  2.11.2 (OK)
pylint >=2.5.0                :  2.13.5 (OK)
pyls_spyder >=0.4.0           :  0.4.0 (OK)
pylsp >=1.4.1;<1.5.0          :  1.4.1 (OK)
pylsp_black >=1.2.0           :  1.2.0 (OK)
qdarkstyle >=3.0.2;<3.1.0     :  3.0.3 (OK)
qstylizer >=0.1.10            :  0.2.1 (OK)
qtawesome >=1.0.2             :  1.1.1 (OK)
qtconsole >=5.3.0;<5.4.0      :  5.3.0 (OK)
qtpy >=2.0.1                  :  2.0.1 (OK)
rtree >=0.9.7                 :  1.0.0 (OK)
setuptools >=49.6.0           :  58.1.0 (OK)
sphinx >=0.6.6                :  4.5.0 (OK)
spyder_kernels >=2.3.0;<2.4.0 :  2.3.0 (OK)
textdistance >=4.2.0          :  4.2.2 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  2.1.7 (OK)
zmq >=17                      :  22.3.0 (OK)

# Optional:
cython >=0.21                 :  None (NOK)
matplotlib >=3.0.0            :  3.5.1 (OK)
numpy >=1.7                   :  1.22.3 (OK)
pandas >=1.1.1                :  1.4.2 (OK)
scipy >=0.17.0                :  1.8.0 (OK)
sympy >=0.7.3                 :  None (NOK)

[dalthviz]

Hi @arian-f thank you for the feedback! Could you reset you preferences running with the reset flag (spyder --reset)? Let us know if that helps

[arian-f]

still reproduce it with a new spyder config directory (%userprofile%.spyder-py3) - that should be equivalent, right?

[dalthviz]

Just to be sure maybe try then to run Spyder using the safe-mode flag (running with that will use a clean config directory without removing yours). For that you can run spyder --safe-mode. Let us know!

[arian-f]

spyder --safe-mode shows the same thing.

[dalthviz]

Did you install Python with admin privileges? Could you start Spyder from a cmd with admin privileges and check if the pylsp starts?

[arian-f]

I installed python itself as admin, but packages as unprivileged user -> packages that are not part of python itself are in %appdata%\Python\Python310\site-packages. When I run it with admin priviliges, it's still reproducing the error. However, I don't see how that would make a difference: it would be able to modify system files, but it should be able to read everything anyway.

However I have the feeling it would be more productive to inspect why it can't find the installed package instead of general debugging attempts. Easiest would be to somehow gain a python shell instead of the pylsp start. Could you give me a pointer on how to achieve that?

This is my work laptop -> I'm going off for easter now. Might look into this specifically over the weekend, but don't be surprised if I only manage to get back on Tuesday.

[dalthviz]

Well since seems like you are able to launch manually the pylsp you could try to use the Advanced configuration (Tools > Preferences > Completion and linting > Advanced) to see if Spyder is able to connect to the manually started pylsp server instance.

Since you have your interpreter and packages in different places most probably as you say is something regarding the sys.path. Checking a little bit I think Spyder handles that here:

spyder/spyder/plugins/completion/providers/languageserver/client.py

Lines 266 to 319 in 270f141

	def start_server(self):
	"""Start server."""
	# This is not necessary if we're trying to connect to an
	# external server
	if self.external_server or self.stdio:
	return
	logger.info('Starting server: {0}'.format(' '.join(self.server_args)))
	# Create server process
	self.server = QProcess(self)
	env = self.server.processEnvironment()
	# Use local PyLS instead of site-packages one.
	if (DEV or running_under_pytest()) and not running_in_ci():
	sys_path = self._clean_sys_path()
	env.insert('PYTHONPATH', os.pathsep.join(sys_path)[:])
	# Adjustments for the Python language server.
	if self.language == 'python':
	# Set the PyLS current working to an empty dir inside
	# our config one. This avoids the server to pick up user
	# files such as random.py or string.py instead of the
	# standard library modules named the same.
	cwd = osp.join(get_conf_path(), 'lsp_paths', 'cwd')
	if not osp.exists(cwd):
	os.makedirs(cwd)
	# On Windows, some modules (notably Matplotlib)
	# cause exceptions if they cannot get the user home.
	# So, we need to pass the USERPROFILE env variable to
	# the PyLS.
	if os.name == "nt" and "USERPROFILE" in os.environ:
	env.insert("USERPROFILE", os.environ["USERPROFILE"])
	else:
	# There's no need to define a cwd for other servers.
	cwd = None
	# Most LSP servers spawn other processes, which may require
	# some environment variables.
	for var in os.environ:
	env.insert(var, os.environ[var])
	logger.info('Server process env variables: {0}'.format(env.keys()))
	# Setup server
	self.server.setProcessEnvironment(env)
	self.server.errorOccurred.connect(self.handle_process_errors)
	self.server.setWorkingDirectory(cwd)
	self.server.setProcessChannelMode(QProcess.MergedChannels)
	if self.server_log_file is not None:
	self.server.setStandardOutputFile(self.server_log_file)
	# Start server
	self.server.start(self.server_args[0], self.server_args[1:])

Maybe you can check if unindenting there the line 308 (logger.info('Server process env variables: {0}'.format(env.keys()))) and running again in debug mode helps us get more info regarding how your env variables look for the pylsp process being launched.

[ccordoba12]

@arian-f, are you setting your PYTHONPATH manually outside Spyder? Because if you do, then that should be the cause of this problem.

[arian-f]

@ccordoba12 PYTHONPATH is not set.

The problem might be in python, however it can be solved in spyder: start_server starts a process with a clean environment, whereas start_transport does not. The clean environment causes python not to add site.USER_SITE to sys.path even though site.ENABLE_USER_SITE is True. From site's documentation on windows, site.USER_SITE is %APPDATA%\Python\PythonXY\site-package - this hints that %APPDATA% must be set.

It's definitely confusing that site.USER_SITE points to the correct directory even if it's not added. That's why I think that the bug might be in python itself.

Indeed, adding %APPDATA% to the server environment fixes things.
Thus the fix in spyder is literally one line. I'd add it with a pull request, which branch should I base it on? 5.x?

The same issue might occur in unix and macos. I could try to check it on linux, but I don't have access to macos.

details

I started a python shell instead of the server via:

$ diff -u Python39/site-packages/spyder/plugins/completion/providers/languageserver/client.py Python310/site-packages/spyder/plugins/completion/providers/languageserver/client.py
--- Python39/site-packages/spyder/plugins/completion/providers/languageserver/client.py 2022-04-19 10:16:16.536655900 +0200
+++ Python310/site-packages/spyder/plugins/completion/providers/languageserver/client.py        2022-04-19 12:07:10.120589200 +0200
@@ -316,7 +316,11 @@
             self.server.setStandardOutputFile(self.server_log_file)

         # Start server
-        self.server.start(self.server_args[0], self.server_args[1:])
+        # self.server.start(self.server_args[0], self.server_args[1:])
+        import win32api
+        exe = win32api.GetShortPathName(self.transport_args[0])
+        self.server.start('c:/windows/system32/cmd.exe', ['/c', f'start {exe}'])

and indead, sys.path doesn't have the site.USER_SITE: (sorry for the 8.3 names here, I didn't manage to start a program with spaces)

...
COMSPEC: c:\windows\system32\cmd.exe
PATH: %APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;%APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;%APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;%APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;%APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\libnvvp;C:\Program Files\Python39\Scripts\;C:\Program Files\Python39\;C:\Program Files\Python310\Scripts\;C:\Program Files\Python310\;C:\Program Files\ImageMagick-7.1.0-Q16;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\PuTTY\;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\Program Files\NVIDIA Corporation\Nsight Compute 2022.1.1\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Users\asanusi.FLISOMAG\AppData\Local\Microsoft\WindowsApps;C:\Users\asanusi.FLISOMAG\.dotnet\tools;C:\Program Files\CMake\bin;C:\Program Files (x86)\Nmap
PATHEXT: .COM;.EXE;.BAT;.CMD;.VBS;.JS;.WS;.MSC
PROMPT: $P$G
SYSTEMROOT: C:\WINDOWS
USERPROFILE: C:\Users\User

>>> for p in sys.path: print(p)
C:\PROGRA~1\PYTHON~1\python310.zip
C:\PROGRA~1\PYTHON~1\DLLs
C:\PROGRA~1\PYTHON~1\lib
C:\PROGRA~1\PYTHON~1
C:\PROGRA~1\PYTHON~1\lib\site-packages

>>> sys.executable
'C:\\PROGRA~1\\PYTHON~1\\python.exe'

>>> site.USER_SITE
'%APPDATA%\\Python\\Python310\\site-packages'

[ccordoba12]

Ok, thanks for investigating this further @arian-f! So, this problem would be solved if we add APPDATA to the env vars of the process that starts the server?

[arian-f]

exactly, e.g. this diff makes it work:

$ diff -u Python39/site-packages/spyder/plugins/completion/providers/languageserver/client.py Python310/site-packages/spyder/plugins/completion/providers/languageserver/client.py
--- Python39/site-packages/spyder/plugins/completion/providers/languageserver/client.py 2022-04-19 12:33:22.461020400 +0200
+++ Python310/site-packages/spyder/plugins/completion/providers/languageserver/client.py        2022-04-19 13:12:20.711573600 +0200
@@ -297,6 +297,7 @@
             # the PyLS.
             if os.name == "nt" and "USERPROFILE" in os.environ:
                 env.insert("USERPROFILE", os.environ["USERPROFILE"])
+            if os.name == "nt" and "APPDATA" in os.environ:
+                env.insert("APPDATA", os.environ["APPDATA"])
         else:
             # There's no need to define a cwd for other servers.
             cwd = None

[ccordoba12]

Ok, thanks for the confirmation @arian-f! I'm going to do this only for non-Anaconda installations because packages in USER_SITE pollute (and usually break) all conda environments.

That's why we prefer to start the server with the minimal amount of env vars, i.e. so that it can't be broken by packages installed outside the environment in which it's present.