Segmentation Fault on checking mujoco licences

[wookayin]

Hi all, thanks for open-sourcing such a great toolkit!
I would like to share and report a bug or issue in which segmentation fault occurs, which might also be happening for others.

>>> from dm_control import suite
>>> env = suite.load(domain_name="cartpole", task_name="swingup")
[1]    28081 segmentation fault (core dumped)  python

I found that it happens during verifying the mujoco license, say when we call _maybe_register_license() (and mjlib.mj_activate).

stacktrace on gdb:

#0  0x0000000000017946 in ?? ()
#1  0x00007ffff0a84fff in mjl_check () from $HOME/.mujoco/mjpro150/bin/libmujoco150.so
#2  0x00007ffff0a4c013 in mjl_activate () from $HOME/.mujoco/mjpro150/bin/libmujoco150.so
#3  0x00007ffff0a30e4d in mj_activate () from $HOME/mujoco/mjpro150/bin/libmujoco150.so
...

So the segfault happens when mj_activate from libmujoco150.so is invoked. I think it might be a problem of mujoco, not of this package; but maybe something can be wrong with auto-wrapping via ctypes. Plus, the test (python setup.py test) also fails with the same segmentation fault.

It happened on a linux 64bit machine (Ubuntu 14.04), with python 3.6 (anaconda) and 2.7.

I verified that mjkey.txt exists and the liecense worked well with mujoco C++ samples and OpenAI's mujoco-py.

[borgwang]

Same issue

[yotam]

We haven't been able to reproduce the issue so far. @borgwang are your system details the same?

[borgwang]

@yotam Exactly the same (64bit machine running Ubuntu 14.04 and anaconda python 3.6). The segfault raised when mj_activate from libmujoco150.so is invoked.

[HenryZhou7]

Ubuntu Release 14.04, anaconda python 2.7 has the same issue here.

>>> from dm_control import suite
>>> env = suite.load(domain_name='cartpole', task_name='swingup')
Segmentation fault (core dumped)

[wookayin]

I highly suspect that it might be a glibc compatibility issue. Ubuntu 14.04 and RHEL systems have very old version (say 2.19 and 2.17, respectively), and it doesn't happen in 16.04 (with glibc 2.23).

[alimuldal]

@wookayin @borgwang Are you able to run the MuJoCo example binaries?

[wookayin]

Yes, I am. The example binary somehow checks the license, but have no why it crash if it is being used inside ctype.

❯❯❯ cd $HOME/.mujoco/
❯❯❯ export LD_LIBRARY_PATH=$HOME/.mujoco/mjpro150/bin/
❯❯❯ ls mjkey.txt
mjkey.txt
❯❯❯ mjpro150/bin/test mjpro150/model/humanoid.xml xs 10

Comparison of original and saved model
 Max difference : 2.61e-06
 Field name     : body_mass

Simulation ..........
 Simulation time      : 0.35 s
 Realtime factor      : 28.21 x
 Time per step        : 0.071 ms
 Contacts per step    : 10
 Constraints per step : 32
 Degrees of freedom   : 27

[saran-t]

Just had time to look at this: I've been able to reproduce the issue with Anaconda and Ubuntu 14 LTS.

The issue seems to stem from the fact that Anaconda brings its own libstdc++ that is newer than the system's library.

Are you able to see if the crash occurs if you don't use it inside an Anaconda environment?

[wookayin]

Yes, it happens both on non-anaconda environment (plain virtualenv, all of 2.7 and 3.x) and anaconda environments.

❯❯❯ workon tf; which python
$HOME/.virtualenvs/tf/bin/python
❯❯❯ python --version
Python 3.4.3

❯❯❯ python -c 'from dm_control import suite; env = suite.load(domain_name="cartpole", task_name="swingup")'
[1]    12613 segmentation fault (core dumped)  python -c

The python binary originates from the system python3 (say /usr/bin/python3 installed with apt, pre-compiled using GCC 4.8.4 and possibly the old version of glibc/libstdc++).

[saran-t]

@wookayin I believe you suspected right: this seems most likely an issue with glibc ABI version conflict between Anaconda and the old Ubuntu system library.

As a workaround, I've managed to get the demo script to run (in python 2.7) by:

• apt-get install libxml2 libxslt1.1
• LD_PRELOAD=libstdc++.so.6:libxslt.so.1:libexslt.so.0:libxml2.so.2 python my_script.py

Obviously this is a very hacky workaround. It might result in another crash elsewhere. It might lead to a subtle problem that doesn't result in a crash. YMMV.

*PS: Don't add the LD_PRELOAD to your .bashrc. Preferably, don't export it in your shell either.

[saran-t]

Closing this issue for now since I think the workaround above should work. If not, feel free to comment and reopen.

[wookayin]

@saran-t That workaround works for Ubuntu 14.04. Would you be available to come up with one for RHEL systems? (with glibc 2.17, quite old and legacy) The counterpart libraries of libxml2 and libxslt are installed (latest).

[1]    2739 illegal hardware instruction (core dumped)  LD_PRELOAD=libstdc++.so.6:libxslt.so.1:libexslt.so.0:libxml2.so.2 ipython

It may well be that it is not due to different distributions, but due to a hardware/instruction issue.

Well, but fundamentally, why should it happen? Is it due to the mujoco binary is only compatible with recent glibc? However, openai gym's control environment just works. It would be great for the dm_control library to not suffer from this issue (without a work-around; workaround is a workaround IMHO), so can we have this issue open until it is resolved? (actually I have no permission to re-open)

Thanks.

UPDATE: Maybe, openai gym was using mujoco 1.31.

[HenryZhou7]

I wonder if the workaround will actually do for Python 3.6
I am getting the following error when executing from dm_control import suite
OSError: /u/henryzhou/.mujoco/mjpro150/bin/libmujoco150.so: undefined symbol: __glewBlitFramebuffer