ImportError: $PREFIX/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type #3580

mova · 2020-01-25T23:46:14Z

Issue Prelude

Please complete these steps and check these boxes (by putting an x inside
the brackets) before filing your issue:

Issue Details

What did you do?
I'm trying to get YCM to run with an old machine without root privileges.
As both python and vim are too old to be used, i need to set both of them up under a prefix in my home.
$PREFIX/bin is added to the path and $PREFIX/lib to the LD_LIBRARY_PATH.

Include steps to reproduce here.
Python

wget https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz
tar xvfvz Python-3.8.1.tgz
pushd Python-3.8.1
#pymode needs to use shared libraries 
 ./configure --prefix=$PREFIX --enable-shared
make
make install
 popd

Vim

git clone --depth 1 https://github.com/vim/vim.git vim-tmp
pushd vim-tmp
pythonver=python3.8
./configure --prefix=$PREFIX --with-features=huge \
            --enable-terminal \
            --enable-multibyte \
            --enable-python3interp=yes \
            --with-python3-command=$pythonver \
            --with-python3-config-dir=$(${pythonver}-config --configdir) \
            --enable-cscope \
            --enable-fail-if-missing
make
make install

recompiling ycm

cd ~/.vim/plugged/YouCompleteMe
python3.8 install.py --clang-completer

What actually happened?
YCM crashes on startup.

Include description of the observed behaviour, including actual output,
screenshots, etc.
:messages yields:

Traceback (most recent call last):
  File "<string>", line 61, in <module>
  File "/home/$USER/.vim/plugged/YouCompleteMe/python/ycm/base.py", line 25, in <module>
    from ycm import vimsupport
  File "/home/$USER/.vim/plugged/YouCompleteMe/python/ycm/vimsupport.py", line 31, in <module>
    from ycmd.utils import ( ByteOffsetToCodepointOffset,
  File "/home/$USER/.vim/plugged/YouCompleteMe/third_party/ycmd/ycmd/utils.py", line 32, in <module>
    import socket
  File "$PREFIX/lib/python3.8/socket.py", line 49, in <module>
    import _socket
ImportError: $PREFIX/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type

This was not an issue when compiling python without the --enable-shared flag.

Diagnostic data

Output of `vim --version`

vim --version             
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Jan 25 2020 23:41:58)
Included patches: 1-148
Compiled by user@machine
Huge version without GUI.  Features included (+) or not (-):
+acl               -farsi             -mouse_sysmouse    -tag_old_static
+arabic            +file_in_path      +mouse_urxvt       -tag_any_white
+autocmd           +find_in_path      +mouse_xterm       -tcl
+autochdir         +float             +multi_byte        +termguicolors
-autoservername    +folding           +multi_lang        +terminal
-balloon_eval      -footer            -mzscheme          +terminfo
+balloon_eval_term +fork()            +netbeans_intg     +termresponse
-browse            +gettext           +num64             +textobjects
++builtin_terms    -hangul_input      +packages          +textprop
+byte_offset       +iconv             +path_extra        +timers
+channel           +insert_expand     -perl              +title
+cindent           +job               +persistent_undo   -toolbar
-clientserver      +jumplist          +popupwin          +user_commands
-clipboard         +keymap            +postscript        +vartabs
+cmdline_compl     +lambda            +printer           +vertsplit
+cmdline_hist      +langmap           +profile           +virtualedit
+cmdline_info      +libcall           -python            +visual
+comments          +linebreak         +python3           +visualextra
+conceal           +lispindent        +quickfix          +viminfo
+cryptv            +listcmds          +reltime           +vreplace
+cscope            +localmap          +rightleft         +wildignore
+cursorbind        -lua               -ruby              +wildmenu
+cursorshape       +menu              +scrollbind        +windows
+dialog_con        +mksession         +signs             +writebackup
+diff              +modify_fname      +smartindent       -X11
+digraphs          +mouse             -sound             -xfontset
-dnd               -mouseshape        +spell             -xim
-ebcdic            +mouse_dec         +startuptime       -xpm
+emacs_tags        -mouse_gpm         +statusline        -xsmp
+eval              -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+ex_extra          +mouse_netterm     +syntax            -xterm_save
+extra_search      +mouse_sgr         +tag_binary        
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "$PREFIX/share/vim"
Compilation: gcc -std=gnu99 -c -I. -Iproto -DHAVE_CONFIG_H     -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       
Linking: gcc -std=gnu99   -L/usr/local/lib -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -ldl     -L$PREFIX/lib/python3.8/config-3.8-x86_64-linux-gnu -lpython3.8 -lcrypt -lpthread -ldl -lutil -lm -lm

Output of `YcmDebugInfo`

E492: Not an editor command: YcmDebugInfo

Contents of YCM, ycmd and completion engine logfiles

OS version, distribution, etc.

CentOs7

Output of build/install commands

─ python3.8 install.py --clang-completer
Searching Python 3.8 libraries...
Found Python library: $PREFIX/lib/libpython3.8.so
Found Python headers folder: $PREFIX/include/python3.8
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found PythonLibs: $PREFIX/lib/libpython3.8.so (found suitable version "3.8.1", minimum required is "3.5")
-- Using libclang archive: /home/$USER/.vim/plugged/YouCompleteMe/third_party/ycmd/cpp/../clang_archives/libclang-9.0.0-x86_64-unknown-linux-gnu.tar.bz2
-- Using libclang to provide semantic completion for C/C++/ObjC
-- Using external libclang: /tmp/ycm_build_88g2hoyc/lib/libclang.so.9
-- NOT using clang-tidy for static analysis.
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/ycm_build_88g2hoyc
Scanning dependencies of target BoostParts
[  2%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/filesystem/src/windows_file_codecvt.cpp.o
[  6%] [  6%] [  8%] [ 10%] [ 12%] [ 14%] [ 16%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/filesystem/src/path_traits.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/filesystem/src/codecvt_error_category.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/filesystem/src/portability.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/filesystem/src/operations.cpp.o
[ 18%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/filesystem/src/utf8_codecvt_facet.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/filesystem/src/unique_path.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/filesystem/src/path.cpp.o
[ 20%] [ 24%] [ 24%] [ 26%] [ 28%] [ 30%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/fileiter.cpp.o
[ 32%] [ 34%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/regex.cpp.o
[ 36%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/instances.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/usinstances.cpp.o
[ 38%] [ 40%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/wide_posix_api.cpp.o
[ 44%] [ 44%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/wc_regex_traits.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/regex_debug.cpp.o
[ 46%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/cregex.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/posix_api.cpp.o
[ 48%] [ 51%] Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/regex_raw_buffer.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/c_regex_traits.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/icu.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/cpp_regex_traits.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/w32_regex_traits.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/static_mutex.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/winstances.cpp.o
Building CXX object BoostParts/CMakeFiles/BoostParts.dir/libs/regex/src/regex_traits_defaults.cpp.o
Linking CXX static library libBoostParts.a
[ 51%] Built target BoostParts
Scanning dependencies of target ycm_core
[ 53%] [ 55%] [ 57%] Building CXX object ycm/CMakeFiles/ycm_core.dir/Utils.cpp.o
[ 59%] [ 61%] [ 63%] [ 65%] [ 67%] [ 71%] [ 71%] [ 73%] Building CXX object ycm/CMakeFiles/ycm_core.dir/Candidate.cpp.o
[ 75%] Building CXX object ycm/CMakeFiles/ycm_core.dir/CodePointRepository.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/CharacterRepository.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/Word.cpp.o
[ 77%] Building CXX object ycm/CMakeFiles/ycm_core.dir/versioning.cpp.o
[ 81%] [ 81%] [ 83%] Building CXX object ycm/CMakeFiles/ycm_core.dir/CandidateRepository.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/IdentifierUtils.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/IdentifierCompleter.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/Result.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/Character.cpp.o
[ 85%] [ 87%] [ 89%] [ 91%] [ 93%] [ 95%] Building CXX object ycm/CMakeFiles/ycm_core.dir/ycm_core.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/CompletionData.cpp.o
[ 97%] Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/Range.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/Documentation.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/TranslationUnit.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/CompilationDatabase.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/ClangCompleter.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/TranslationUnitStore.cpp.o
[100%] Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/ClangUtils.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/ClangCompleter/ClangHelpers.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/PythonSupport.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/CodePoint.cpp.o
Building CXX object ycm/CMakeFiles/ycm_core.dir/IdentifierDatabase.cpp.o



Linking CXX shared library /home/$USER/.vim/plugged/YouCompleteMe/third_party/ycmd/ycm_core.so
[100%] Built target ycm_core
-- The C compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Found PythonLibs: $PREFIX/lib/libpython3.8.so (found version "3.8.1")
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/regex_build_c1nom85x
Scanning dependencies of target _regex
[ 50%] [100%] Building C object CMakeFiles/_regex.dir/regex_3/_regex_unicode.c.o
Building C object CMakeFiles/_regex.dir/regex_3/_regex.c.o
Linking C shared library /home/$USER/.vim/plugged/YouCompleteMe/third_party/ycmd/third_party/cregex/regex_3/_regex.so
[100%] Built target _regex

The text was updated successfully, but these errors were encountered:

puremourning · 2020-01-25T23:52:48Z

what happens if you do:

python3.8
>>> import socket

mova · 2020-01-25T23:55:56Z

python3.8 
Python 3.8.1 (default, Jan 26 2020, 00:14:07) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> dir(socket)
['AF_ALG', 'AF_APPLETALK', 'AF_ASH .....

puremourning · 2020-01-25T23:56:38Z

how about this:

vim --clean
:py3 import socket

mova · 2020-01-25T23:57:37Z

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "$PREFIX/lib/python3.8/socket.py", line 49, in <module>
    import _socket
ImportError: $PREFIX/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type

puremourning · 2020-01-25T23:57:57Z

In that case, the issue is with your vim/python configuration, not YCM.

puremourning · 2020-01-26T00:00:24Z

For the record, I would recommend using something like pyenv (or at least its python build thing) to build your custom python. I build Vim against pyenv pythons regularly without issue.

moon548834 · 2020-02-22T08:42:32Z

@mova Hi! Did you solve this issue? I meet the same problem. After struggling with it for two days, I solved this problem. As @puremourning said, this issue is with your vim configuration(not python in fact). You just need to

export LDFLAGS="-rdynamic"

before your compile the vim, hopefully this can help others who meet the same issue:-)

puremourning · 2020-02-22T09:29:18Z

Odd, I've never had to do that, but thanks for sharing!

mova · 2020-02-24T09:31:38Z

@moon548834 Thanks a lot. I switched to neovim, where the setup worked out of the box.

puremourning · 2020-04-23T12:58:19Z

For the record, another user was able to resolve a similar issue by ensuring -rdynamic is used when linking Vim. The explanation is that while libpython is statically linked into the Vim executable, when it loads dynamic python modules (such as the cyython socket module), the symbols exported by lib python need to be visible to the loaded dyanamic python module. The -rdynamic flag tells the linker to make these symbols visible to dynamically loaded libraries from the statically linked vim exe.

The other solution is to use --with-python3interp=dynamic, but in my experience this has its own woes.

puremourning closed this as completed Jan 25, 2020

bstaletic mentioned this issue Mar 12, 2020

ImportError: [path-to]/.pyenv/versions/3.8.2/lib/python3.8/lib-dynload/_socket.cpython-38-arm-linux-gnueabihf.so: undefined symbol: PyFloat_Type ycm-core/ycmd#1408

Closed

ycm-core locked as resolved and limited conversation to collaborators Apr 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ImportError: $PREFIX/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type #3580

ImportError: $PREFIX/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type #3580

mova commented Jan 25, 2020 •

edited

puremourning commented Jan 25, 2020

mova commented Jan 25, 2020

puremourning commented Jan 25, 2020

mova commented Jan 25, 2020 •

edited

puremourning commented Jan 25, 2020

puremourning commented Jan 26, 2020

moon548834 commented Feb 22, 2020

puremourning commented Feb 22, 2020

mova commented Feb 24, 2020

puremourning commented Apr 23, 2020

ImportError: $PREFIX/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type #3580

ImportError: $PREFIX/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type #3580

Comments

mova commented Jan 25, 2020 • edited

Issue Prelude

Issue Details

Diagnostic data

Output of vim --version

Output of YcmDebugInfo

Contents of YCM, ycmd and completion engine logfiles

OS version, distribution, etc.

Output of build/install commands

puremourning commented Jan 25, 2020

mova commented Jan 25, 2020

puremourning commented Jan 25, 2020

mova commented Jan 25, 2020 • edited

puremourning commented Jan 25, 2020

puremourning commented Jan 26, 2020

moon548834 commented Feb 22, 2020

puremourning commented Feb 22, 2020

mova commented Feb 24, 2020

puremourning commented Apr 23, 2020

mova commented Jan 25, 2020 •

edited

Output of `vim --version`

Output of `YcmDebugInfo`

mova commented Jan 25, 2020 •

edited