Python doesn't load libraries when ran from vim

[urbster1]

Having a similar issue to #1728

Packages
python 3.8.2-1
vim 8.2.0592-1

Using the mingw64 shell, I get this error when trying to use https://github.com/chiel92/vim-autoformat. Modules load just fine from the prompt, but running in vim I get the following:

line   59:
Traceback (most recent call last):
  File "/mingw64/lib/python3.8/subprocess.py", line 64, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mingw64/lib/python3.8/subprocess.py", line 69, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'```

[StarWolf3000]

Did you even read my comment on #1728?

[urbster1]

I need to use the mingw64 prompt. Is there no solution for that?

…

On Mon, Apr 27, 2020, 11:55 PM StarWolf3000 ***@***.***> wrote: Did you even read my comment on #1728 <#1728>? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1930 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLMCWODAXRPIUVGVEATW3DROZHSBANCNFSM4MSMMAZQ> .

[StarWolf3000]

It is not supported and most likely never will. Why do you need to use Python in vim in mingw64 shell?There's nothing that justifies it. You can use it just fine in msys2 shell. Well yeah, you need to open a second instance of mintty (just by running msys2.exe), running in msys2 mode, but that's okay.

Vim does not load the Python installed under /usr, when run from a mingw shell, and this results in missing modules it needs to work properly.

@lazka

[lazka]

I guess we have to force the vim python support to use the msys one.

I don't know much about vim, but the README from your plugin suggests that putting let g:python3_host_prog=/usr/bin/python3 into .vimrc might help.

[urbster1]

I tried this but it's showing this error:

Error detected while processing .vimrc:
line    2:
E15: Invalid expression: /usr/bin/python3

I think this only works for neovim, not vanilla vim. I also tried with quotes around the path, but still receive the same error using Autoformat as before.

[Jokia]

I installed the plug youcompleteme for vim, it worked well in python 3.7.
But when I update python to 3.8 and vim to 8.2 592-1, the ycm doesn't work. I got the message:
YouCompleteMe unavailable: No module named '_socket'
So I tried to input the command py3 import socket in vim,
It responsed as the following message:
Traceback (most recent call last): File "<string>", line 1, in <module> File "/mingw64/lib/python3.8/socket.py", line 49, in <module> import _socket ModuleNotFoundError: No module named '_socket'
How to make ycm work in vim 8.2.0592-1 ?

[anythingapplied]

@StarWolf3000 The reason for doing this is to run python based vim plugins such as YouCompleteMe (I'm was running into the same issue @Jokia is with this same plugin).

@Jokia I was running into the same issue with the same plugin. I was able to resolve the error message by adding the following line to my .bashrc:
export PYTHONPATH=$PYTHONPATH:/usr/lib/python3.8/lib-dynload/

Though there is a chance I might be mixing and matching msys/python with mingw64/mingw-w64-x86_64-python in a weird way since I have them both installed. I may be using one's executable to point to the other's libraries?

I created a small program:

import os, sys, _socket
print(sys.executable)
print(sys.version)
print(_socket.__file__)

[~]$ /mingw64/bin/python socket_location.py
C:/msys64/mingw64/bin/python.exe
3.8.2 (default, Apr  9 2020, 13:17:39)  [GCC 9.3.0 64 bit (AMD64)]
C:/msys64/mingw64/lib/python3.8/lib-dynload/_socket-cpython-38.dll
[~]$ /bin/python socket_location.py
/bin/python.exe
3.8.3 (default, May 18 2020, 08:39:23)
[GCC 9.3.0]
/usr/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-msys.dll

(note the mingw version says 3.8.2 up there, the same version that pacman says for my mingw version)

But inside of vim when I run the same program using the vim command :python3 import os, sys, _socket; print(sys.executable); print(sys.version); print(_socket.__file__);, I get:

/mingw64/bin/python3.exe
3.8.3 (default, May 18 2020, 08:39:23)
[GCC 9.3.0]
/usr/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-msys.dll

It is saying it is the mingw64 python3 executable (version 3.8.2) but its reporting my msys python version number (version 3.8.3). Not even sure how that is possible, but I guess that is what you get when my paths are pointing to some components of the other version of python.

Note when I tried export PYTHONPATH=$PYTHONPATH:/mingw64/lib/python3.8/lib-dynload/ (which was the first thing I tried) it did NOT fix the issue. Maybe I'm just not giving it the right path style for mingw program.

[Jokia]

Thank for your reply. @anythingapplied

I have tried update vim and python to the newest version, and add export PYTHONPATH=$PYTHONPATH:/mingw64/lib/python3.9/lib-dynload/ to .bashrc, when I open vim from the MINGW64 commandline, the pluging ycm still does'nt work, but if I open vim from the MSYS commandline, it works.

Now, I'm using the vim from MSYS for programming, and it works well.

mingw64/mingw-w64-x86_64-python and msys python are diffrent: conan-io/conan#2638 (comment)

[puremourning]

How does the msys2 /usr/bin/vim determine the python sys path?

A tribal way to reproduce/test this is to install msys2 python and msys2 vim, along with mingw64 python (there is no mingw64 vim afaict):

1. open msys2.exe and pacman -S vim python (or whatever)
2. open mingw64.exe and packman -S mingw64/python (or whatever)

Then, tell vim to load the msys2 python dll and confirm that "works" in both envs:

• open msys2.exe and

• vim --clean

• :set pythonthreedll=msys2-python3.10.dll

• :echo has('python3')

• :py3 pass

• :py3 import sys; print( sys.path )

• :py3 import socket

• open mingw64.exe and

• vim --clean (note, this is /usr/bin/vim i.e. an MSYS2 app, not mingw)

• :set pythonthreedll=msys2-python3.10.dll (again msys2)

• :py3 import sys; print( sys.path )

• :py3 import socket

You will see in the mingw64 env:

1. sys.path is something like /mingw64/lib/python3.10 (should be /usr/lib/python3.10)
2. import socket fails (because python mingw doesn't have unix sockets)

This seems kind of funky. I'm trying to understand how the default sys path is worked out in the mingw64 environment, even when loading msys binaries.

I understand the comment about "don't mix mingw64 and msys in the same env", but that can't be a universal rule, as /usr/bin/ls wouldn't be allowed in a mingw64 env (this being as I understand it, the entire purpose of msys + mingw).

To answer the banal question about why you would "want" to use libpython within Vim in a mingw environment, many Vim plugins, mappings, scripts etc. want to operate within your build envrionment. YCM is an example, so is Vimspector. SO is :make

[puremourning]

Incidentally, if you open mingw64.exe and do /usr/bin/python and import sys; print( sys.path ), you get the "correct" (msys) paths.

It seems like things (other than /usr/bin/python) loading msys-python3.10.dll are somehow not getting the right sys path, but I'm getting into very hairy "hand wavy" areas of my understanding now.

[RivenSkaye]

This seems to be an issue with either allowing dynamically loading a Python3 interpreter if needed, or with the configure and compilation commands.
This line in vim/src/Makefile lists a --with-python3-command flag, which could be set to /usr/bin/python3 which I think would solve the problem.
The biggest issue here is having both msys/python and $MSYSTEM's python installed, which causes msys/vim to load the current environment's rather than the msys-specific one.

This problem can be circumvented by prefixing the PATH with the msys2's bin folders before opening vim, which will make it load everything from there if it's available. It's a hackjob workaround, but seems to work just fine on my end without any errors.

Errorstack from loading the offending Python-based plugin (UltiSnips in my case):

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/__init__.py", line 6, in <module>
    from UltiSnips.snippet_manager import UltiSnips_Manager
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/snippet_manager.py", line 13, in <module>
    from UltiSnips import vim_helper
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/vim_helper.py", line 13, in <module>
    from UltiSnips.snippet.source.file.common import normalize_file_path
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/snippet/source/__init__.py", line 8, in <module>
    from UltiSnips.snippet.source.file.snipmate import SnipMateFileSource
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/snippet/source/file/snipmate.py", line 10, in <module>
    from UltiSnips.snippet.definition import SnipMateSnippetDefinition
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/snippet/definition/__init__.py", line 3, in <module>
    from UltiSnips.snippet.definition.ulti_snips import UltiSnipsSnippetDefinition
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/snippet/definition/ulti_snips.py", line 6, in <module>
    from UltiSnips.snippet.definition.base import SnippetDefinition
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/snippet/definition/base.py", line 16, in <module>
    from UltiSnips.text_objects import SnippetInstance
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/text_objects/__init__.py", line 9, in <module>
    from UltiSnips.text_objects.shell_code import ShellCode
  File "/home/Martin/.vim/plugged/ultisnips/pythonx/UltiSnips/text_objects/shell_code.py", line 8, in <module>
    from subprocess import Popen, PIPE
  File "/mingw64/lib/python3.11/subprocess.py", line 104, in <module>
    from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'

Looking at the relevant code in /mingw64/lib/python3.11/subprocess/py, it seems to be probing the environment to see if fork_exec is available, which should be everywhere that doen't set sys.platform to emscripten or wasi and that doesn't have an import available for msvcrt. Is something in vim actively preventing that import by chance?