.deb generation fails for Xenial, works in Bionic ("dpkg-shlibdeps can't find libs building non-catkin pkg")

[jlblancoc]

Hi guys,

Not sure if this is actually a bug of the buildfarm code, bloom, or something else, but after many days trying to debug it, I think I nailed it down to something wrong in the docker containers used in the buildfarm for some distros, so I'll share my findings in the hope someone can give me a hand.

Instead of repeating here the entire situation that leads to this error, let me link to my former posts in ROS answers:

• Original question
• Update

The latest finding is that exactly the same upstream sources, released for melodic, build as a charm for Artful and Bionic; example log:

10:55:48 	dpkg-shlibdeps -Tdebian/ros-melodic-mrpt1.substvars -l/tmp/binarydeb/ros-melodic-mrpt1-1.5.7/debian/ros-melodic-mrpt1//opt/ros/melodic/lib/ debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-nav.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-hmtslam.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-hwdrivers.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-maps.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-graphs.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-kinematics.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-detectors.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-opengl.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-vision.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-tfest.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-topography.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-slam.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-graphslam.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-obs.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-gui.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/lib/libmrpt-base.so.1.5.7 debian/ros-melodic-mrpt1/opt/ros/melodic/bin/2d-slam-demo debian/ros-melodic-mrpt1/opt/ros/melodic/bin/carmen2simplemap debian/ros-melodic-mrpt1/opt/ros/melodic/bin/RawLogViewer debian/ros-melodic-mrpt1/opt/ros/melodic/bin/kinect-3d-view debian/ros-melodic-mrpt1/opt/ros/melodic/bin/ReactiveNavigationDemo debian/ros-melodic-mrpt1/opt/ros/melodic/bin/hmt-slam-gui debian/ros-melodic-mrpt1/opt/ros/melodic/bin/graph-slam debian/ros-melodic-mrpt1/opt/ros/melodic/bin/DifOdometry-Datasets debian/ros-melodic-mrpt1/opt/ros/melodic/bin/track-video-features debian/ros-melodic-mrpt1/opt/ros/melodic/bin/robotic-arm-kinematics debian/ros-melodic-mrpt1/opt/ros/melodic/bin/icp-slam-live debian/ros-melodic-mrpt1/opt/ros/melodic/bin/rawlog-edit debian/ros-melodic-mrpt1/opt/ros/melodic/bin/kf-slam debian/ros-melodic-mrpt1/opt/ros/melodic/bin/pf-localization debian/ros-melodic-mrpt1/opt/ros/melodic/bin/velodyne-view debian/ros-melodic-mrpt1/opt/ros/melodic/bin/kinect-stereo-calib debian/ros-melodic-mrpt1/opt/ros/melodic/bin/gps2rawlog debian/ros-melodic-mrpt1/opt/ros/melodic/bin/navlog-viewer debian/ros-melodic-mrpt1/opt/ros/melodic/bin/simul-landmarks debian/ros-melodic-mrpt1/opt/ros/melodic/bin/simul-beacons debian/ros-melodic-mrpt1/opt/ros/melodic/bin/grid-matching debian/ros-melodic-mrpt1/opt/ros/melodic/bin/GridmapNavSimul debian/ros-melodic-mrpt1/opt/ros/melodic/bin/icp-slam debian/ros-melodic-mrpt1/opt/ros/melodic/bin/hmt-slam debian/ros-melodic-mrpt1/opt/ros/melodic/bin/mrpt-perfdata2html debian/ros-melodic-mrpt1/opt/ros/melodic/bin/rawlog-grabber debian/ros-melodic-mrpt1/opt/ros/melodic/bin/map-partition debian/ros-melodic-mrpt1/opt/ros/melodic/bin/carmen2rawlog debian/ros-melodic-mrpt1/opt/ros/melodic/bin/observations2map debian/ros-melodic-mrpt1/opt/ros/melodic/bin/holonomic-navigator-demo debian/ros-melodic-mrpt1/opt/ros/melodic/bin/graphslam-engine debian/ros-melodic-mrpt1/opt/ros/melodic/bin/SceneViewer3D debian/ros-melodic-mrpt1/opt/ros/melodic/bin/rbpf-slam debian/ros-melodic-mrpt1/opt/ros/melodic/bin/kinect-3d-slam debian/ros-melodic-mrpt1/opt/ros/melodic/bin/ReactiveNav3D-Demo debian/ros-melodic-mrpt1/opt/ros/melodic/bin/simul-gridmap debian/ros-melodic-mrpt1/opt/ros/melodic/bin/mrpt-performance debian/ros-melodic-mrpt1/opt/ros/melodic/bin/image2gridmap debian/ros-melodic-mrpt1/opt/ros/melodic/bin/ptg-configurator debian/ros-melodic-mrpt1/opt/ros/melodic/bin/camera-calib debian/ros-melodic-mrpt1/opt/ros/melodic/bin/features-matching debian/ros-melodic-mrpt1/opt/ros/melodic/bin/ro-localization debian/ros-melodic-mrpt1/opt/ros/melodic/bin/hmtMapViewer

(this command is run without errors in this case!!)

So, in summary:

1. Building with ROS prerelease.sh: works (but this doesn't attempt to build the .deb which is the problematic part)
2. Manually building (from a native Xenial machine) the .deb using dpkg-buildpackage using exactly the same .dsc that is built inside the build farm: works (!), whereas it fails inside the buildfarm. Grep for error: in this log for example.
3. Letting the build farm to build the same upstream sources within Artful / Bionic docker images: works (!). Example log:

I hope someone can give me some ideas on how to exactly setup a docker image with the same contents as inside the build farm so we can debug what's wrong with the case (2) above.

Obviously, my motivation is that the error prevent the package to be released for kinetic, whereas it'll be released for melodic without problems.

[nuclearsandwich]

You can use this library to generate a script which will run commands on your local machine just as our Jenkins jobs do.

1. You will need docker >= 17.05, git, and the vcs your package uses installed on your host system.
   You'll need to install the master branch of ros_buildfarm locally. I recommend creating a python3 virtualenv, activating it, and then running pip install git+https://github.com/ros-infrastructure/ros_buildfarm

2. Create a temporary directory to test in.

mkdir /tmp/kinetic-mrpt1-x64
cd /tmp/kinetic-mrpt1-x64

3. With the virtualenv activated you can generate a release script.

generate_release_script.py https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml kinetic default mrpt1 ubuntu xenial amd64 > run.sh

4. Run the generated script.
   run.sh is a shell script which performs the same steps as the buildfarm sourcedeb and binarydeb jobs do. I usually recommend running it once without any modifications in order to verify that it fails the same way as on the buildfarm.

sh run.sh

5. If you can reproduce the failure, you can go back and remove --rm from the docker run invocations in order to inspect or restart the containers. When you re-run the script, delete the source and binary directories which are created. You'll get a warning if you forget to and it should be heeded.

[dirk-thomas]

There shouldn't be a need to use the latest master branch. The latest release 2.0.1 installed from e.g. Debian packages should be sufficient.

The steps to run the release job locally are also described in the docs.

[jlblancoc]

👍 thanks a lot!

Now I could reproduce the error locally... that's good enough to allow further investigation.

[jlblancoc]

Despite I could reproduce the error, it seems I can't make much progress due to my lack of docker or ros_buildfarm know-how... This is how I've tried (unsuccessfully) to debug this issue:

• I followed @nuclearsandwich steps (including removing the --rm flags). I could reproduce the same error that happens in the farm builder in the local docker container. So far, so good.

dpkg-shlibdeps: error: couldn't find library libmrpt-gui.so.1.5 needed by debian/ros-kinetic-mrpt1/opt/ros/kinetic/bin/ReactiveNavigationDemo (ELF format: 'elf64-x86-64'; RPATH: '')
...
make[1]: *** [override_dh_shlibdeps] Error 2
debian/rules:47: recipe for target 'override_dh_shlibdeps' failed
make[1]: Leaving directory '/tmp/binarydeb/ros-kinetic-mrpt1-1.5.7'
make: *** [binary] Error 2
debian/rules:22: recipe for target 'binary' failed
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
E: Building failed
Traceback (most recent call last):
  File "/tmp/ros_buildfarm/ros_buildfarm/binarydeb_job.py", line 138, in build_binarydeb
    subprocess.check_call(cmd, cwd=source_dir)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['apt-src', 'build', 'ros-kinetic-mrpt1']' returned non-zero exit status 1
# END SUBSECTION

--------------------------------------------------------------------------------------------------
`apt-src build ros-kinetic-mrpt1` failed.
This is usually because of an error building the package.
The traceback from this failure (just above) is printed for completeness, but you can ignore it.
You should look above `E: Building failed` in the build log for the actual cause of the failure.
--------------------------------------------------------------------------------------------------

• I list the stopped docker containers:

$ docker ps -a

CONTAINER ID        IMAGE                                                         COMMAND                  CREATED             STATUS                      PORTS               NAMES
b0bcd500f3c0        binarydeb_build.kinetic_ubuntu_xenial_amd64_mrpt1             "sh -c 'PYTHONPATH=/…"   7 hours ago         Exited (1) 41 minutes ago                       distracted_albattani
7556613ee99c        binarydeb_task_generation.kinetic_ubuntu_xenial_amd64_mrpt1   "sh -c 'PYTHONPATH=/…"   7 hours ago         Exited (0) 7 hours ago                          xenodochial_chatterjee
d41677786856        sourcedeb.kinetic_ubuntu_xenial_mrpt1                         "sh -c 'PYTHONPATH=/…"   7 hours ago         Exited (0) 7 hours ago                          condescending_thompson

• Apparently, this is the way to getting into the stopped container listed above:

docker commit b0bcd500f3c0 jlblancoc/test_img
docker run -ti --entrypoint=bash jlblancoc/test_img

I can browse around and I can see many files in the /tmp directory that are left from the unit tests run as part of the package generation, but the important directory /tmp/binarydeb/ exists but it's empty.

Am I doing something wrong with docker? Or is there something else to change in the run.sh to prevent it to clean up the debian build directory?
Thanks

[dirk-thomas]

I just took a brief look. The specified branch in the tracks of the release repo of mrpt1 is null which implies the default branch. But as far as I can see the default branch is for version 2. Should the devel_branch be changed to mrpt-1.5?

[jlblancoc]

The specified branch in the tracks of the release repo of mrpt1 is null which implies the default branch.
But as far as I can see the default branch is for version 2.
Should the devel_branch be changed to mrpt-1.5?

Correct! What are the implications of this mismatch? AFAIK, it would make Jenkins to try to build the incorrect branch for the "dev" builds, right? Anyway, that wouldn't affect the builds for releases, which have an explicit tag, right?

Anyway, what's the recommended way to fix it? Manual editing, gloom,...?

PS: Just found the reason I left it to "empty" (null). Bloom docs say:

Upstream Devel Branch:

Branch in upstream repository on which to search for the version.
This is used only when version is set to ':{auto}'.
[None]:

and since version was set to :{ask}....

PS2: Will continue trying to figuring out what's going on inside the docker container...

[jlblancoc]

Good news: I nailed down the error. It's related to the use of MultiArch path postfixes.
For Kinetic, using u16.04, the following command fails:

dpkg-shlibdeps \
  -Tdebian/ros-kinetic-mrpt1.substvars \
  -ldebian/ros-kinetic-mrpt1/opt/ros/kinetic/lib/ \
  debian/ros-kinetic-mrpt1/opt/ros/kinetic/lib/x86_64-linux-gnu/libmrpt-maps.so.1.5.7 \
  ...

fails, but if the search path is changed to include the multiarch part, it works: -ldebian/ros-kinetic-mrpt1/opt/ros/kinetic/lib/x86_64-linux-gnu.

However, something like the original command line seems to work for opencv3 (log).
Will checkout how opencv3 deals with multiarch at the cmake level just in case that's the key.
In case I definitively found the fix, I'll come back to close this one.

Thanks again for the help debugging this!

[jlblancoc]

For the records (this should explain why it builds in Bionic):

From debhelper 10.2.1 onwards CMAKE_INSTALL_LIBDIR is set by debhelper (to /usr/lib/. This means that a fair number of packages (that install libraries to CMAKE_INSTALL_LIBDIR) will 'just work', so all you need to do is add: depends: debhelper (>=10.2.1)
https://wiki.debian.org/Multiarch/Implementation#CMake

debhelper versions:

• u16.04: v9
• u18.04: v11

[jlblancoc]

Fixed via mrpt-ros-pkg-release/mrpt1-release@17361d7

The problem was not related with ros_buildfarm, but with different behavior of different versions of debhelper.

Thanks for the support!