Ubuntu 16.04: ycmd crashing because of install.py picking wrong Python version

[ulidtko]

Honestly, I got a bit lost in all those Python3-related discussions and PRs that you have here.

Here's a problem I had, its error messages and how to solve it.

On Ubuntu 16.04 xenial, vim is Py3-only (-python +python3). That means, no support for Python 2.7 plugins.

On my system, where I compile a lot of people's softwares, there are both python3.5-dev and python2.7-dev installed.

In this situation, the YCM build process (./install.py --all) selects py2.7 by default. With the vim support remark above, one can immediately see that this selection is incorrect. Nevertheless, the build completes without error; just doesn't work at all afterwards.

Ycm client log says 'Connection refused':

requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=42553): Max retries exceeded with url: /ready (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f5108c00710>: Failed to establish a new connection: [Errno 111] Connection refused',))

ycmd error log says:

2017-07-24 18:07:31,483 - ERROR - ycm_core library compiled for Python 2 but loaded in Python 3.
Traceback (most recent call last):
  File "/home/ulidtko/.config/dotfiles/HOME/vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/server_utils.py", line 96, in CompatibleWithCurrentCore
    ycm_core = ImportCore()
  File "/home/ulidtko/.config/dotfiles/HOME/vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/server_utils.py", line 88, in ImportCore
    import ycm_core as ycm_core
ImportError: dynamic module does not define module export function (PyInit_ycm_core)

A-ha! "compiled for Python 2 but loaded in Python 3", great!

---

How do we compile for Python 3 then? Turns out, not so easy with the ./install.py concoction.

Re-run the build, now with --build-dir BUILD argument. This creates a directory called BUILD. Now, command ccmake BUILD, and meet the CMake manual build configuration tool (may need to apt install cmake-curses-gui, whatever).

It presents a USE_PYTHON2=ON option. Nice, flip it! Hit c to configure. Whoopsies:

 CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
   Could NOT find PythonLibs: Found unsuitable version "2.7.12", but required
   is at least "3.3" (found
   /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so)
 Call Stack (most recent call first):
   /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:386 (_FPHSA_FAILURE_MESSAGE)
   /usr/share/cmake-3.5/Modules/FindPythonLibs.cmake:265 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
   CMakeLists.txt:237 (find_package)

Doesn't work, damn it!

Okay, it looks like we'll have to resort to typing in library paths manually... In ccmake, hit t (or just edit the BUILD/CMakeCache.txt directly; I guess vimers are not into fancy GUIs), find and set these variables:

PYTHON_INCLUDE_DIR=/usr/include/python3.5m
PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.5m.so

Hit c, g, (cd BUILD; ninja) or (cd BUILD; make) if you don't have Ninja... aaaand, done.

:YcmRestartServer and you're ready to roll!

---

P.S. Not an actual bug report. Any similarities, explicit or implied, are purely coincidental.

Many thanks for the awesome plugin!

[vheon]

With the vim support remark above, one can immediately see that this selection is incorrect.

This is actually wrong.

YCM does not require that the client and the server run on the same Python version, i.e. one can run the client with Python3 while the server run under Python2 and vice versa. The important thing is that when we start the server we use the same Python binary that was used during the building of the server. To do that in the build.py script of ycmd (which is called by the install.py script of YCM we actually save the Python that was used to build the server https://github.com/Valloric/ycmd/blob/master/build.py#L517-L520 so when we have to start the server from YCM we know which Python to pick. Of course that is overwritable using the g:ycm_server_python_interpreter option, so if you builded the server with Python2 but your vim only support Python3 and for some reason (like your case) we still pick the wrong Python interpreter (we could investigate that) you can still point YCM to the right interpreter with such option; there is no need to recompile the server from scratch unless you actually want to build the server with Python3 which actually is easier than you think because python3 install.py should be more than enough.

[puremourning]

As @vheon correctly points out, this should all just work these days.

Could you let us know what you have set in g:ycm_server_python_interpreter and what is the contents of the file PYTHON_USED_DURING_BUILDING within YouCompleteMe/third_party/ycmd directory ? And any information on any sort of virtualenv/pyenv or whatever that was set when building but not set when launching Vim ?

How do we compile for Python 3 then? Turns out, not so easy with the ./install.py concoction.

It's pretty easy: run python3 ./install.py (if that's really what you need). But sure, use the manual build instructions if it works for you.

I regularly run a python3 ycmd with a python2 Vim. And vice versa.

[ulidtko]

@puremourning nice, it didn't occur to me to try python3 install.py.

I didn't set g:ycm_server_python_interpreter; cat third_party/ycmd/PYTHON_USED_DURING_BUILDING gives me 'no such file or directory'. virtualenv and similar are not used neither when launching vim, nor during ycm build.

Any ideas on why USE_PYTHON2=OFF doesn't quite work as intented?.. I'm afraid that could be an issue with CMake's FindPythonLibs or something. The call site at ycmd/cpp/CMakeLists.txt looks pretty good, but cmake configure still chokes on 2.7. That seems weird.

[puremourning]

third_party/ycmd/PYTHON_USED_DURING_BUILDING gives me 'no such file or directory'.

Well that's suspicious. To confirm, after running install.py, and it exits successfully, there is no file in <path to YouCompleteMe>/third_party/ycmd/PYTHON_USED_DURING_BUILDING ?

The USE_PYTHON2 internal macro is used by install.py (also within our CI environment) so i'm pretty sure that it works as expected.

[ulidtko]

Well that's suspicious. To confirm, after running install.py, and it exits successfully, there is no file in <path to YouCompleteMe>/third_party/ycmd/PYTHON_USED_DURING_BUILDING ?

Agreed, and somehow I'm unable to repro that anymore.

Regarding the CI environment -- I'm pretty sure that it doesn't get to have the problem of several Python versions being available (in headers+libs form), hence it doesn't trigger the Found unsuitable version "2.7.12", but required is at least "3.3" error with USE_PYTHON2=OFF.

[micbou]

Regarding the CI environment -- I'm pretty sure that it doesn't get to have the problem of several Python versions being available (in headers+libs form), hence it doesn't trigger the Found unsuitable version "2.7.12", but required is at least "3.3" error with USE_PYTHON2=OFF.

We are using pyenv on Travis so they are at least two Python: one from system and another from pyenv. We don't get the error simply because we are using the build.py script (through run_tests.py) on our CI environments. This script sets the PYTHON_LIBRARY, PYTHON_INCLUDE_DIR, and USE_PYTHON2 CMake variables according to the Python used to run it.

Any ideas on why USE_PYTHON2=OFF doesn't quite work as intented?.. I'm afraid that could be an issue with CMake's FindPythonLibs or something. The call site at ycmd/cpp/CMakeLists.txt looks pretty good, but cmake configure still chokes on 2.7. That seems weird.

From my experience, the FindPythonLibs module from CMake is quite buggy. That's why we set all the variables mentioned above in the build.py script. See PR ycm-core/ycmd#481.

[Kentoseth]

So I am dealing with the issue of trying to get YCM to run using only python3 on my machine.

python3 --version
Python 3.4.3

(hint, there is no python command that can be run)

Going to the YCM folder to run python3 install.py or python3 ./install.py results in the following error:

Searching Python 3.4 libraries...
Found Python library: /usr/lib/python3.4/config-3.4m-x86_64-linux-gnu/libpython3.4.so
Found Python headers folder: /usr/include/python3.4m
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
  Could NOT find PythonLibs (missing: PYTHON_INCLUDE_DIRS) (found suitable
  version "3.4.3", minimum required is "3.3")
Call Stack (most recent call first):
  /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-2.8/Modules/FindPythonLibs.cmake:208 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:237 (find_package)


-- Configuring incomplete, errors occurred!
See also "/tmp/ycm_build_u7ixck1i/CMakeFiles/CMakeOutput.log".
ERROR: the build failed.

Checking the stderr-log shows:

- ERROR - ycm_core library not detected; you need to compile it by running the build.py script. See the documentation for more details.
Traceback (most recent call last):
  File "/home/ubuntu/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/server_utils.py", line 96, in CompatibleWithCurrentCore
    ycm_core = ImportCore()
  File "/home/ubuntu/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/server_utils.py", line 88, in ImportCore
    import ycm_core as ycm_core
ImportError: No module named 'ycm_core'

Steps to reproduce the issue:

1) Ubuntu 14.04 (recommended to attempt in 14.04 container that only has py3-support)
2) sudo apt remove vim vim.tiny
3) sudo apt install libncurses5-dev python3-dev libsm-dev libice-dev
4) ./configure --with-features=huge --enable-multibyte --enable-python3interp=yes --with-python3-config-dir=/usr/lib/python3.4/config-3.4m-x86_64-linux-gnu --prefix=/usr/local
5) make VIMRUNTIMEDIR=/usr/local/share/vim/vim80
6) sudo make install
7) my .vimrc file has the following (relevant) content w.r.t YCM:
    let g:ycm_server_python_interpreter = '/usr/bin/python3'
    let g:ycm_autoclose_preview_window_after_insertion=1
    let g:ycm_add_preview_to_completeopt=1

If this should be its own issue, please tell me and I will create one.

Edit:
running ./install.py results in:

/usr/bin/env: python: No such file or directory

[micbou]

@Kentoseth Did you install the python3-dev package?

sudo apt-get install python3-dev

[Kentoseth]

So I self-diagnosed this issue after finding this: link

The problem is cmake 2.8 . Follow these instructions:

sudo apt-get remove cmake 
sudo apt-get update 
sudo apt-get upgrade 
sudo apt-get install cmake3

then run: python3 install.py

and voila, it works!

[Kentoseth]

@micbou see my answer above.