SystemError: dynamic module not initialized properly

[yshahin]

I am trying to run ViZDoom on OSX, everything is built but running the python example file with vizdoom.so symlinked into the folder

Using homebrew package manager and pyenv for python management

cmake command

cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_PYTHON=ON -DBUILD_JAVA=OFF \
  -DFMOD_LIBRARY="/Developer/FMOD Programmers API/api/lib/libfmodex.dylib" \
  -DFMOD_INCLUDE_DIR="/Developer/FMOD Programmers API/api/inc/" \
  -DNO_ASM=true

cmake output

-- Could NOT find GME (missing:  GME_LIBRARIES GME_INCLUDE_DIR) 
-- Using system zlib
-- Using system jpeg library
-- Using system bzip2 library
-- Using internal gme library
-- /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include
-- /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include
-- /Users/yshahin/Developer/Work/doom/ViZDoom/vizdoom_src/lzma/C
-- FMOD include files found at /Developer/FMOD Programmers API/api/inc
-- FMOD library found at /Developer/FMOD Programmers API/api/lib/libfmodex.dylib
-- Could not find clock_gettime. Timing statistics will not be available.
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   thread
--   system
--   date_time
--   chrono
--   atomic
-- Fluid synth libs: /usr/local/lib/libfluidsynth.dylib
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   filesystem
--   thread
--   system
--   date_time
--   chrono
--   regex
--   iostreams
--   atomic
-- Found PythonInterp: /Users/yshahin/.pyenv/shims/python2.7 (found suitable version "2.7.10", minimum required is "2.7") 
-- Found PythonInterp: /Users/yshahin/.pyenv/shims/python2.7 (found version "2.7.10") 
-- Found components for NumPy
-- NUMPY_ROOT_DIR    = /usr/local
-- NUMPY_INCLUDES    = /Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/site-packages/numpy/core/include
-- NUMPY_LIBRARIES   = 
-- NUMPY_API_VERSION = 1.9.3
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   python
-- Configuring done
CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   vizdoomshared

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/yshahin/Developer/Work/doom/ViZDoom

python error output

Traceback (most recent call last):
  File "/Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/site.py", line 548, in <module>
    main()
  File "/Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/site.py", line 266, in addusersitepackages
    user_site = getusersitepackages()
  File "/Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/site.py", line 241, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/site.py", line 231, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/sysconfig.py", line 520, in get_config_var
    return get_config_vars().get(name)
  File "/Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/sysconfig.py", line 453, in get_config_vars
    import re
  File "/Users/yshahin/.pyenv/versions/2.7.10/lib/python2.7/re.py", line 108, in <module>
    import _locale
SystemError: dynamic module not initialized properly

[wjaskowski]

I confirm that (OSX). We have not much experience on OSX, so I have no solution at the moment. C++ examples do work on OSX so this seems like a problem with the python wrapper...

[awentzonline]

Since you're using brew-managed python you might check out the OSX/Compile section in the readme. The problem may be that CMake isn't using your brew python installation. You can see which libpython is linked with: otool -L vizdoom.so | grep python Make sure to replace the 2.x.x in the path with whatever is correct for your system.

If that doesn't work, make sure your brew-managed boost installation is also using your brew python. If it isn't, try reinstalling boost with the flag --build-from-source

[wjaskowski]

Stupid me! The error you are experiencing also shows up when you run the examples using python3 instead of python2. Check your defaults.

[pawni]

Could someone elaborate on the wrong python linking? I am facing the same dynamic module problem and already build boost again from source.

CMake:

cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_PYTHON=ON
-- The C compiler identification is AppleClang 7.3.0.7030029
-- The CXX compiler identification is AppleClang 7.3.0.7030029
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found BZip2: /usr/lib/libbz2.dylib (found version "1.0.6") 
-- Looking for BZ2_bzCompressInit
-- Looking for BZ2_bzCompressInit - found
-- Found JPEG: /usr/local/lib/libjpeg.dylib  
-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.5") 
-- Could NOT find GME (missing:  GME_LIBRARIES GME_INCLUDE_DIR) 
-- Using system zlib
-- Using system jpeg library
-- Using system bzip2 library
-- Using internal gme library
-- Performing Test HAVE_NO_ARRAY_BOUNDS
-- Performing Test HAVE_NO_ARRAY_BOUNDS - Success
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of char
-- Check size of char - done
-- Check size of short
-- Check size of short - done
-- Check size of int
-- Check size of int - done
-- Check size of long
-- Check size of long - done
-- /usr/include
-- /usr/include
-- /Users/nickpawlowski/Documents/Projects/ViZDoom/vizdoom_src/lzma/C
-- Looking for itoa
-- Looking for itoa - not found
-- Performing Test DUMB_CAN_USE_SSE
-- Performing Test DUMB_CAN_USE_SSE - Success
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29") 
-- Found OpenAL: /System/Library/Frameworks/OpenAL.framework  
-- Could not find FMOD include files
-- Could NOT find SndFile (missing:  SNDFILE_LIBRARY SNDFILE_INCLUDE_DIR) 
-- Could NOT find MPG123 (missing:  MPG123_LIBRARIES MPG123_INCLUDE_DIR) 
-- Could NOT find FluidSynth (missing:  FLUIDSYNTH_LIBRARIES FLUIDSYNTH_INCLUDE_DIR) 
-- Looking for filelength
-- Looking for filelength - not found
-- Looking for strupr
-- Looking for strupr - not found
-- Looking for stricmp
-- Looking for stricmp - not found
-- Looking for strnicmp
-- Looking for strnicmp - not found
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - not found
-- Looking for clock_gettime
-- Looking for clock_gettime - not found
-- Could not find clock_gettime. Timing statistics will not be available.
-- Performing Test HAS_VA_COPY
-- Performing Test HAS_VA_COPY - Success
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   thread
--   system
--   date_time
--   chrono
-- Fluid synth libs: FLUIDSYNTH_LIBRARIES-NOTFOUND
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   filesystem
--   thread
--   system
--   date_time
--   chrono
--   regex
--   iostreams
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
-- Found PythonInterp: /usr/local/bin/python2.7 (found suitable version "2.7.11", minimum required is "2.7") 
-- Found PythonLibs: /usr/lib/libpython2.7.dylib (found suitable version "2.7.10", minimum required is "2.7") 
-- Found PythonInterp: /usr/local/bin/python2.7 (found version "2.7.11") 
-- Found NUMPY: /usr/local/lib/python2.7/site-packages/numpy/core/include  
-- Found components for NumPy
-- NUMPY_ROOT_DIR    = /usr/local
-- NUMPY_INCLUDES    = /usr/local/lib/python2.7/site-packages/numpy/core/include
-- NUMPY_LIBRARIES   = 
-- NUMPY_API_VERSION = 1.11.0
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   python
-- Configuring done
CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   vizdoomshared

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/nickpawlowski/Documents/Projects/ViZDoom

Python:

which python
/usr/local/bin/python

/usr/local/bin/python -> ../Cellar/python/2.7.11/bin/python

otool:

otool -L vizdoom.so 
vizdoom.so:
    /usr/local/opt/boost/lib/libboost_filesystem-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_system-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_date_time-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_chrono-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_regex-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_iostreams-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost-python/lib/libboost_python-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.10)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

[maciejjaskowski]

It would seem that CMakeCache.txt should be the answer to our issues.

So I tried to make sure that it conseqently contains packages from anaconda installation (Numpy, Boost, cmake)
What I did:

conda install boost
conda install -c meznom boost-python=1.55.0
conda install cmake
./cmake_clean
/Users/maciej/anaconda/envs/python27/bin/cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_PYTHON=ON -DBUILD_JAVA=OFF -DPYTHON_INCLUDE=/Users/maciej/anaconda/envs/python27/include -DPYTHON_LIBRARY=/Users/maciej/anaconda/envs/python27/lib/libpython2.7.dylib -DPYTHON_INCLUDE_DIR=/Users/maciej/anaconda/envs/python27/include/python2.7

and then cmake found the proper boost and numpy libraries (see below)

However, the problem persists.
See my CMakeCache.txt

CMakeCache.txt excerpt

Boost_CHRONO_LIBRARY_DEBUG:FILEPATH=/Users/maciej/anaconda/envs/python27/lib/libboost_chrono.dylib
Boost_CHRONO_LIBRARY_RELEASE:FILEPATH=/Users/maciej/anaconda/envs/python27/lib/libboost_chrono.dylib
...

[BenjaminEjlertsen]

I've got the same error message when trying to compile basic.py.
I've tried to run the otool -L vizdoom.so, which gives me this:

/usr/local/opt/boost/lib/libboost_filesystem-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/boost/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/boost/lib/libboost_system-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/boost/lib/libboost_date_time-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/boost/lib/libboost_chrono-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/boost/lib/libboost_regex-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/boost/lib/libboost_iostreams-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/boost/lib/libboost_atomic-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/boost-python/lib/libboost_python-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

Within Pycharm it says the interpreter is 2.7.1 though. Is this the what is causing the issue?

awentzonline says that i should just change 2.x.x to what ever it should be... But my question is, how??

Hope someone can help. Have spend days on this now :-)

[kaleposhobios]

The steps here solved my issue #80

Basically I had to make completely sure that everything about the build was using the same Python version. When I tried to use the CMake GUI I couldn't make that work, so I scrapped the CMake GUI completely and simply followed the solution from issue 80, and it worked.