Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running ign gazebo fails on Linux #4

Closed
1 task done
traversaro opened this issue Mar 28, 2021 · 12 comments · Fixed by conda-forge/libignition-common-feedstock#36
Closed
1 task done

Running ign gazebo fails on Linux #4

traversaro opened this issue Mar 28, 2021 · 12 comments · Fixed by conda-forge/libignition-common-feedstock#36

Comments

@traversaro
Copy link
Contributor

traversaro commented Mar 28, 2021

Issue:

Running ign gazebo -s fails with the following error:

(igndome) traversaro@IITICUBLAP102:~$ ign gazebo -s
Error while loading the library [/home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins/ignition-gazebo-physics-system]: /home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins: cannot read file data: Is a directory
[Err] [SystemLoader.cc:75] Failed to load system plugin [ignition-gazebo-physics-system] : couldn't load library on path [/home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins].
Error while loading the library [/home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins/ignition-gazebo-user-commands-system]: /home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins: cannot read file data: Is a directory
[Err] [SystemLoader.cc:75] Failed to load system plugin [ignition-gazebo-user-commands-system] : couldn't load library on path [/home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins].
Error while loading the library [/home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins/ignition-gazebo-scene-broadcaster-system]: /home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins: cannot read file data: Is a directory
[Err] [SystemLoader.cc:75] Failed to load system plugin [ignition-gazebo-scene-broadcaster-system] : couldn't load library on path [/home/traversaro/miniforge3/envs/igndome/lib/ign-gazebo-4/plugins].


^C(igndome) traversaro@IITICUBLAP102:~$

Environment (conda list):
$ conda list
(igndome) traversaro@IITICUBLAP102:~$ conda list
# packages in environment at /home/traversaro/miniforge3/envs/igndome:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
assimp                    5.0.1                hedfc422_5    conda-forge
boost-cpp                 1.74.0               hc6e9bd1_2    conda-forge
bullet-cpp                3.09                 hde0f152_1    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
ca-certificates           2020.12.5            ha878542_0    conda-forge
certifi                   2020.12.5        py39hf3d152e_1    conda-forge
console_bridge            1.0.1                hc9558a2_0    conda-forge
cppzmq                    4.7.1                hf7cf922_2    conda-forge
dartsim                   6.9.5                h53b96f2_4    conda-forge
dbus                      1.13.6               hfdff14a_1    conda-forge
eigen                     3.3.9                h4bd325d_1    conda-forge
expat                     2.2.10               h9c3ff4c_0    conda-forge
fcl                       0.6.1                h2cbc392_3    conda-forge
ffmpeg                    4.3.1                hca11adc_2    conda-forge
flann                     1.9.1             h44f99b7_1008    conda-forge
fontconfig                2.13.1            hba837de_1004    conda-forge
freeimage                 3.18.0               hadfbfc9_7    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
gdbm                      1.18                 h0a1914f_2    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
glib                      2.68.0               h9c3ff4c_2    conda-forge
glib-tools                2.68.0               h9c3ff4c_2    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
gst-plugins-base          1.18.4               h29181c9_0    conda-forge
gstreamer                 1.18.4               h76c114f_0    conda-forge
gts                       0.7.6                h64030ff_2    conda-forge
hdf5                      1.10.6          nompi_h7c3c948_1111    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
ilmbase                   2.5.5                h780b84a_0    conda-forge
jpeg                      9d                   h516909a_0    conda-forge
jsoncpp                   1.9.4                h4bd325d_1    conda-forge
jxrlib                    1.1                  h516909a_2    conda-forge
krb5                      1.17.2               h926e7f8_0    conda-forge
lame                      3.100             h14c3975_1001    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libblas                   3.9.0                8_openblas    conda-forge
libcblas                  3.9.0                8_openblas    conda-forge
libccd                    2.1                  he1b5a44_1    conda-forge
libclang                  11.1.0          default_ha53f305_0    conda-forge
libcurl                   7.75.0               hc4aaa36_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_18    conda-forge
libgfortran-ng            7.5.0               h14aa051_18    conda-forge
libgfortran4              7.5.0               h14aa051_18    conda-forge
libglib                   2.68.0               h3e27bee_2    conda-forge
libglu                    9.0.0             he1b5a44_1001    conda-forge
libgomp                   9.3.0               h2828fa1_18    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libignition-cmake2        2.6.2                h9c3ff4c_1    conda-forge
libignition-common3       3.11.1               h4b9f6df_0    conda-forge
libignition-fuel-tools5   5.1.1                hf08c594_6    conda-forge
libignition-gazebo4       4.6.0                h420ee80_0    conda-forge
libignition-gui4          4.3.0                h3975b97_1    conda-forge
libignition-math6         6.7.0                h9c3ff4c_1    conda-forge
libignition-msgs6         6.4.0                hc438e01_0    conda-forge
libignition-physics3      3.1.0                h9c3ff4c_4    conda-forge
libignition-plugin1       1.1.0                h58526e2_3    conda-forge
libignition-rendering4    4.7.0                hd9181ad_1    conda-forge
libignition-sensors4      4.1.0                hc438e01_0    conda-forge
libignition-tools1        1.0.0                h443fdc1_3    conda-forge
libignition-transport9    9.1.0                h3ef4182_3    conda-forge
liblapack                 3.9.0                8_openblas    conda-forge
libllvm11                 11.1.0               hf817b99_0    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libode                    0.16                 he1b5a44_0    conda-forge
libopenblas               0.3.12          pthreads_hb3c22a3_1    conda-forge
libpng                    1.6.37               hed695b0_2    conda-forge
libpq                     13.1                 hfd2b0eb_2    conda-forge
libprotobuf               3.15.6               h780b84a_0    conda-forge
libraw                    0.20.2               h10796ff_1    conda-forge
libsdformat10             10.3.0               hf765594_1    conda-forge
libsodium                 1.0.18               h516909a_1    conda-forge
libssh2                   1.9.0                ha56f1ee_6    conda-forge
libstdcxx-ng              9.3.0               h6de172a_18    conda-forge
libtiff                   4.2.0                hdc55705_0    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libwebp-base              1.2.0                h7f98852_2    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.10               h72842e0_3    conda-forge
libxslt                   1.1.33               h15afd5d_2    conda-forge
libzip                    1.7.3                he9f05b3_0    conda-forge
lxml                      4.6.3            py39h107f48f_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
mysql-common              8.0.23               ha770c72_1    conda-forge
mysql-libs                8.0.23               h935591d_1    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nettle                    3.6                  he412f7d_0    conda-forge
nspr                      4.30                 h9c3ff4c_0    conda-forge
nss                       3.63                 hb5efdd6_0    conda-forge
numpy                     1.20.2           py39hdbf815f_0    conda-forge
octomap                   1.9.5                hc9558a2_1    conda-forge
ogre                      1.12.11              h0b9f4f7_0    conda-forge
openexr                   2.5.5                hf817b99_0    conda-forge
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.4.0                hf7af979_0    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pip                       21.0.1             pyhd8ed1ab_0    conda-forge
protobuf                  3.15.6           py39he80948d_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pugixml                   1.11.4               h9c3ff4c_0    conda-forge
python                    3.9.2           hffdb5ce_0_cpython    conda-forge
python_abi                3.9                      1_cp39    conda-forge
qt                        5.12.9               hda022c4_4    conda-forge
readline                  8.0                  he28a2e2_2    conda-forge
ruby                      2.7.2                he592edb_3    conda-forge
sdl2                      2.0.12               h58526e2_1    conda-forge
setuptools                49.6.0           py39hf3d152e_3    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
sqlite                    3.35.3               h74cdb3f_0    conda-forge
swig                      4.0.2                hd3c618e_2    conda-forge
tinyxml                   2.6.2                h4bd325d_2    conda-forge
tinyxml2                  8.0.0                he1b5a44_1    conda-forge
tk                        8.6.10               hed695b0_1    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
urdfdom                   2.3.3                hc9558a2_0    conda-forge
urdfdom_headers           1.0.5                hc9558a2_2    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
x264                      1!161.3030           h7f98852_0    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.6.12               h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxaw               1.0.13            h516909a_1002    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxmu               1.1.3                h516909a_0    conda-forge
xorg-libxpm               3.5.13               h516909a_0    conda-forge
xorg-libxt                1.1.5             h516909a_1003    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_0    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge
zziplib                   0.13.69              hed695b0_1    conda-forge

Details about conda and system ( conda info ):
$ conda info
(igndome) traversaro@IITICUBLAP102:~$ conda info

     active environment : igndome
    active env location : /home/traversaro/miniforge3/envs/igndome
            shell level : 2
       user config file : /home/traversaro/.condarc
 populated config files : /home/traversaro/miniforge3/.condarc
                          /home/traversaro/.condarc
          conda version : 4.9.2
    conda-build version : 3.21.4
         python version : 3.8.8.final.0
       virtual packages : __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/traversaro/miniforge3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/traversaro/miniforge3/pkgs
                          /home/traversaro/.conda/pkgs
       envs directories : /home/traversaro/miniforge3/envs
                          /home/traversaro/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.1 CPython/3.8.8 Linux/4.19.104-microsoft-standard ubuntu/20.04.2 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
@traversaro
Copy link
Contributor Author

@diegoferigo
Copy link

Is it possible that the OS dependent logic used in Ignition (likely ign-plugin or ign-common) to form the shared library name is broken in conda? From the error, it searches for:

<env>/lib/ign-gazebo-4/plugins/ignition-gazebo-physics-system

when, on Linux, it should search instead:

<env>/lib/ign-gazebo-4/plugins/libignition-gazebo-physics-system.so

@traversaro
Copy link
Contributor Author

Is it possible that the OS dependent logic used in Ignition (likely ign-plugin or ign-common) to form the shared library name is broken in conda? From the error, it searches for:

<env>/lib/ign-gazebo-4/plugins/ignition-gazebo-physics-system

when, on Linux, it should search instead:

<env>/lib/ign-gazebo-4/plugins/libignition-gazebo-physics-system.so

Yes, it seems something like that, I wonder if the binary relocation logic in conda is doing some dark magic.

@diegoferigo
Copy link

diegoferigo commented Apr 22, 2021

I had a look to this problem. It seems that the default plugin path added in SystemLoader.cc#L58:

    std::string homePath;
    ignition::common::env(IGN_HOMEDIR, homePath);
    systemPaths.AddPluginPaths(homePath + "/.ignition/gazebo/plugins");
    systemPaths.AddPluginPaths(IGN_GAZEBO_PLUGIN_INSTALL_DIR); // <-- This line

uses the macro IGN_GAZEBO_PLUGIN_INSTALL_DIR defined in config.hh.in#L19. For some reason I haven't yet understood, a lot of \0 characters are appended, messing up the logic used in ignition common to build the candidates for the library name.

I didn't get the origin of the problem, but the following provides a quick fix:

diff --git a/src/SystemPaths.cc b/src/SystemPaths.cc
index ff9c0c2..d87c9ed 100644
--- a/src/SystemPaths.cc
+++ b/src/SystemPaths.cc
@@ -235,7 +235,9 @@ void SystemPaths::AddFilePaths(const std::string &_path)
 std::string SystemPaths::NormalizeDirectoryPath(const std::string &_path)
 {
   std::string path = _path;
-  // Use '/' because it works on Linux, OSX, and Windows
+  path.erase(std::find(path.begin(), path.end(), '\0'), path.end()); 
+
+ // Use '/' because it works on Linux, OSX, and Windows
   std::replace(path.begin(), path.end(), '\\', '/');
   // Make last character '/'
   if (!EndsWith(path, "/"))

@traversaro
Copy link
Contributor Author

Thanks @diegoferigo . This is probably due to some interaction of the text/binary relocation feature of conda (see https://docs.conda.io/projects/conda-build/en/latest/resources/make-relocatable.html) and the line systemPaths.AddPluginPaths(IGN_GAZEBO_PLUGIN_INSTALL_DIR); . This is quite a conda-specific problem, but if that patch works I think it is worth to just add it as a patch in the recipe. However, how did you tested the patch? If you just compile ignition-gazebo from source, that should work fine even without the patch.

@diegoferigo
Copy link

This is probably due to some interaction of the text/binary relocation feature of conda (see https://docs.conda.io/projects/conda-build/en/latest/resources/make-relocatable.html) and the line systemPaths.AddPluginPaths(IGN_GAZEBO_PLUGIN_INSTALL_DIR); . This is quite a conda-specific problem, but if that patch works I think it is worth to just add it as a patch in the recipe.

Ow ok I had no idea that such workaround was in place in conda, good to know! Thinking about it, it makes totally sense. Though, if this is true as it seems, also other macros could be affected.

However, how did you tested the patch?

I worked on the same environment described in #6. On top of it, I compiled ign-common from sources (all dependecies are found in the environment) and used as install prefix the environment itself. This is quite hacky since the files of the libignition-commonX get overridden, but it was the quickest solution I reached.

If you just compile ignition-gazebo from source, that should work fine even without the patch.

I didn't try it on the environment just described, but the development environment I'm using daily in the past month is conda-based. In particular, I install all the dependencies I need in a conda environment, and then I build against them a complete colcon environment with either Dome or Edifice. This setup is not affected by the problem, so I can confirm that it only occurs when using the binaries distributed through conda-forge.

@traversaro
Copy link
Contributor Author

However, how did you tested the patch?

I worked on the same environment described in #6. On top of it, I compiled ign-common from sources (all dependecies are found in the environment) and used as install prefix the environment itself. This is quite hacky since the files of the libignition-commonX get overridden, but it was the quickest solution I reached.

I see, so you continue to use the Ignition Gazebo binaries from conda? I see, that make sense.

This is probably due to some interaction of the text/binary relocation feature of conda (see https://docs.conda.io/projects/conda-build/en/latest/resources/make-relocatable.html) and the line systemPaths.AddPluginPaths(IGN_GAZEBO_PLUGIN_INSTALL_DIR); . This is quite a conda-specific problem, but if that patch works I think it is worth to just add it as a patch in the recipe.

Ow ok I had no idea that such workaround was in place in conda, good to know! Thinking about it, it makes totally sense. Though, if this is true as it seems, also other macros could be affected.

Note that are not the macros that are corrupted (I just checked the config.hh file and it seems fine), what is corrupted with trailing zero is the constant text buffer of the libignition-gazebo binary that is created when the systemPaths.AddPluginPaths(IGN_GAZEBO_PLUGIN_INSTALL_DIR) line is compiled.

@diegoferigo
Copy link

diegoferigo commented Apr 23, 2021

However, how did you tested the patch?

I worked on the same environment described in #6. On top of it, I compiled ign-common from sources (all dependecies are found in the environment) and used as install prefix the environment itself. This is quite hacky since the files of the libignition-commonX get overridden, but it was the quickest solution I reached.

I see, so you continue to use the Ignition Gazebo binaries from conda? I see, that make sense.

Actually, no :) My daily setup is and remains based on colcon using conda for the dependencies. I'd like to switch to conda binaries when deploying to HPC since it would allow me to bypass the need of using docker / singularity, but the quick workarounds I mentioned is too dirty and I need also other workarounds for other projects not listed here. I'm following the status of conda-forge distribution, but I'm not blocked thanks to the existing containerization setup I'm currently using.

This is probably due to some interaction of the text/binary relocation feature of conda (see https://docs.conda.io/projects/conda-build/en/latest/resources/make-relocatable.html) and the line systemPaths.AddPluginPaths(IGN_GAZEBO_PLUGIN_INSTALL_DIR); . This is quite a conda-specific problem, but if that patch works I think it is worth to just add it as a patch in the recipe.

Ow ok I had no idea that such workaround was in place in conda, good to know! Thinking about it, it makes totally sense. Though, if this is true as it seems, also other macros could be affected.

Note that are not the macros that are corrupted (I just checked the config.hh file and it seems fine), what is corrupted with trailing zero is the constant text buffer of the libignition-gazebo binary that is created when the systemPaths.AddPluginPaths(IGN_GAZEBO_PLUGIN_INSTALL_DIR) line is compiled.

Yep thanks for clarifying, it is what I meant.

@traversaro
Copy link
Contributor Author

I see, so you continue to use the Ignition Gazebo binaries from conda? I see, that make sense.

Actually, no :)

I meant for this specific test. Because if you do not install ignition-gazebo from conda-forge and instead you compile it from source, it will work even without this fix.

@diegoferigo
Copy link

diegoferigo commented Apr 23, 2021

Now I got your point. Yes, for those reading this issue up to here, if you install from conda-forge all the libignition-* packages excluding libignition-gazebo4 (being careful to select the correct ogre version #6), and then build from sources ign-gazebo using as install prefix the conda environment, you should get a working setup.

Disclaimer: I only tested Ubuntu, no idea about the other platforms.

Edit: I want also to add that this setup does not match what you get from the official Ubuntu binaries, since conda-forge uses upstream dart that does not have all the fixes that the Open Robotic's fork has. However, excluding particular cases, your simulation will be ok. Check ignition-forks/dart for more details.

@traversaro
Copy link
Contributor Author

Now I got your point. Yes, for those reading this issue up to here, if you install from conda-forge all the libignition-* packages excluding libignition-gazebo4 (being careful to select the correct ogre version #6), and then build from sources ign-gazebo using as install prefix the conda environment, you should get a working setup.

Disclaimer: I only tested Ubuntu, no idea about the other platforms.

Ok, so the patch #4 (comment) was never tested with a Ignition Gazebo installed via conda binaries?

@diegoferigo
Copy link

Now I got your point. Yes, for those reading this issue up to here, if you install from conda-forge all the libignition-* packages excluding libignition-gazebo4 (being careful to select the correct ogre version #6), and then build from sources ign-gazebo using as install prefix the conda environment, you should get a working setup.
Disclaimer: I only tested Ubuntu, no idea about the other platforms.

Ok, so the patch #4 (comment) was never tested with a Ignition Gazebo installed via conda binaries?

Yes yes I tested it and it works. However, I would not recommend users to override files of libignition-commonX as I described in #4 (comment). It is definitely not a good practice, it was useful for debugging though. Using my quoted solution of this comment is much cleaner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants