Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create our own Debian package archive #54

Open
jack-oquin opened this issue Feb 2, 2018 · 37 comments
Open

Create our own Debian package archive #54

jack-oquin opened this issue Feb 2, 2018 · 37 comments

Comments

@jack-oquin
Copy link
Member

This has become more useful with the kinetic branch, because it has several significant dependencies that we can't release to the general ROS package repository.

The build time has gotten high for the segbot_v2 laptops, and there are problems running Travis CI (#53, utexas-bwi/segbot#98) that might be load-related. Some of that can be addressed by using catkin tools to whitelist or blacklist certain packages for specific purposes. But a custom binary package repository still looks like a good idea.

@jack-oquin jack-oquin changed the title Kinetic: create our own Debian package archive Create our own Debian package archive Feb 2, 2018
@jack-oquin
Copy link
Member Author

jack-oquin commented Feb 5, 2018

After reading the links above, my initial approach is to experiment with checkinstall, which looks like the minimal path to creating Debian packages from catkin or CMake source packages.

At some point, running our own custom ROS buildfarm instance may make sense, but that seems like overkill for our current needs. Similarly, bloom is a wonderful tool, but it's mainly focused on creating Debian (and other) packages for the ROS buildfarm.

@rdelfin
Copy link
Member

rdelfin commented Feb 5, 2018

Just as a sidenote, I've been using CPack (https://cmake.org/Wiki/CMake:Packaging_With_CPack) with a regular CMake project quite successfully. If properly set up, you can use a launchpad PPA as a build farm. I haven't given it a try with ROS packages though. I can give it a try on some packages I have to see how well it works.

@jack-oquin
Copy link
Member Author

@rdelfin: that sounds like a good idea. If you want a CMake package to start with, try our fork of libsegwayrmp. If that works, we can do segway_rmp next, a catkin package that depends on that library.

@rdelfin
Copy link
Member

rdelfin commented Feb 5, 2018

I'll give it a try and let you know how it goes

@jack-oquin
Copy link
Member Author

packpack looks like an interesting option, too. It users docker and is optimized for use with github, Travis CI, and PackageCloud.

@jack-oquin
Copy link
Member Author

A sketch of a way to build a package that will run in /opt/utexas-bwi/kinetic:

mkdir -p catkin_ws/src
cd catkin_ws/src
git clone https://github.com/utexas-bwi/libsegwayrmp.git
cd ..
catkin config --install --install-space /opt/utexas-bwi/kinetic
DESTDIR=/tmp catkin build

That creates an install image in /tmp/opt/utexas-bwi/kinetic that is configured to run when installed in the desired location.

@rdelfin
Copy link
Member

rdelfin commented Feb 5, 2018

I currently have a sample of what to do to get cpack working on utexas-bwi/libsegwayrmp#1. I'll be trying packpack and then move onto segway_rmp

@jack-oquin
Copy link
Member Author

That looks great, Ricardo. I'm going to merge it into a separate cpack branch for now.

jack-oquin added a commit to utexas-bwi/libsegwayrmp that referenced this issue Feb 5, 2018
jack-oquin added a commit to utexas-bwi/libsegwayrmp that referenced this issue Feb 5, 2018
jack-oquin added a commit to utexas-bwi/libsegwayrmp that referenced this issue Feb 5, 2018
@jack-oquin
Copy link
Member Author

Checkinstall and packpack are not looking that good to me, so far. Please take a look and form your own opinion.

@jack-oquin
Copy link
Member Author

The current Travis configuration is failing: https://travis-ci.org/utexas-bwi/libsegwayrmp/builds/337782989

@rdelfin
Copy link
Member

rdelfin commented Feb 6, 2018

I can't seem to replicate the build issue on my local machine. Looking into it

@rdelfin
Copy link
Member

rdelfin commented Feb 6, 2018

Regarding checkinstall I agree. Checkinstall seems like it requires more setup than necessary. packpack might work. If this works, it looks like it could be worth it to use, just because it integrates well. The associated pull request is: utexas-bwi/libsegwayrmp#2

jack-oquin added a commit to utexas-bwi/libsegwayrmp that referenced this issue Feb 6, 2018
jack-oquin added a commit to utexas-bwi/libsegwayrmp that referenced this issue Feb 6, 2018
@jack-oquin
Copy link
Member Author

You're right, the packpack approach does look good.

Maybe there's a way to symlink the CHANGELOG.rst and LICENSE files, instead of copying them. There are bloom modules that automate creating the debian directory. Once we get this working, we can look into making it easier to convert a package.

@rdelfin
Copy link
Member

rdelfin commented Feb 6, 2018

I think the license is doable, since the format is practically identical, however the Changelog looks like it's a very different format (from what I understand it only contains the current version's Changelog as well as the target distro). Maybe a script that generates one from the other?

@jack-oquin
Copy link
Member Author

Yes, bloom has tools to build the Debian-format change log from the ROS one. But, let's not worry about that for now.

jack-oquin added a commit to utexas-bwi/libsegwayrmp that referenced this issue Feb 6, 2018
@jack-oquin
Copy link
Member Author

The original Travis test no longer works on Kinetic.

@jack-oquin
Copy link
Member Author

The root cause of the Travis failure for libsegwayrmp seems to be due to copying a header file to a different location in the src tree. I suspect those directories may be read-only in the industrial_ci setup.

Unfortunately, the old libsegwayrmp Travis CI config does not work for Kinetic.

@jack-oquin
Copy link
Member Author

My first attempt to build a libsegwayrmp Debian package on my development system failed.

Docker errors:

mk-build-deps: Unable to install libsegwayrmp-build-deps at /usr/bin/mk-build-deps line 402.
mk-build-deps: Unable to install all build-dep packages
/pack//deb.mk:117: recipe for target '/build/libsegwayrmp_0.2.0-1_amd64.changes' failed
make: *** [/build/libsegwayrmp_0.2.0-1_amd64.changes] Error 1
rm /build/ls-lR.txt /build/VERSION
make: Leaving directory '/source'

Docker problems are hard to debug. 😄

@jack-oquin
Copy link
Member Author

Right now, the highest priority for packaging is agile_grasp. It takes a long time to build, and seems to be causing Travis failures due to timeout. See: utexas-bwi/bwi_common#106.

@jack-oquin
Copy link
Member Author

Another advantage to using bloom is that it knows how to use rosdep to convert system dependencies into Debian packages for the desired target platform.

@jack-oquin
Copy link
Member Author

@jack-oquin
Copy link
Member Author

jack-oquin commented Mar 3, 2018

Since agile_grasp is causing problems with build time and probably with Travis CI, it has the highest packaging priority.

  • I created a 0.8.0 release in the kinetic branch of our agile_grasp fork using catkin_prepare_release.
  • I forgot to update the maintainer and package description, so I'll probably redo that at some point.
  • Then, I used bloom-release to create a git-buildpackage release repository. That creates the debian directory with its packaging information, reformats the change history, and converts the rosdep dependencies correctly. At the end, you have to skip the parts where bloom wants to update the rosdistro files for the ROS buildfarm.

There are several ways to generate the Debian source and binary packages from the release repo. I am looking at these:

@jack-oquin
Copy link
Member Author

jack-oquin commented Mar 3, 2018

With git-pbuilder:

$ sudo apt install git-buildpackage
$ ARCH=amd64 DIST=xenial git-pbuilder create
...
W: No local /etc/mailname to copy, relying on /var/cache/pbuilder/base-xenial.cow/etc/mailname to be correct
...
Reading package lists...
Building dependency tree...
Package cowdancer is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'cowdancer' has no installation candidate
I: unmounting dev/pts filesystem
I: unmounting run/shm filesystem
I: unmounting proc filesystem
pbuilder create failed
  forking: rm -rf /var/cache/pbuilder/base-xenial.cow 

@jack-oquin
Copy link
Member Author

Apparently, cowdancer is not in the main Ubuntu Xenial repository. A suggested work-around is adding other Ubuntu components to your ~/.pbuilderrc.

$ echo 'COMPONENTS="main universe multiverse restricted"' >> ~/.pbuilderrc
$ ARCH=amd64 DIST=xenial git-pbuilder create

@jack-oquin
Copy link
Member Author

jack-oquin commented Mar 5, 2018

I tried to build, but pbuilder does not resolve the ROS package dependencies:

$ git clone https://github.com/utexas-bwi-gbp/agile_grasp-release.git
$ cd agile_grasp-release
$ checkout origin/debian/kinetic/xenial/agile_grasp
$ gbp buildpackage --git-pbuilder --git-ignore-branch --git-dist=xenial-amd64
...
The following packages have unmet dependencies:
 pbuilder-satisfydepends-dummy : Depends: ros-kinetic-catkin which is a virtual package and is not provided by any available package.

                                 Depends: ros-kinetic-cmake-modules which is a virtual package and is not provided by any available package.

                                 Depends: ros-kinetic-eigen-conversions which is a virtual package and is not provided by any available package.

                                 Depends: ros-kinetic-geometry-msgs which is a virtual package and is not provided by any available package.
... etc ...
E: pbuilder-satisfydepends failed.

Presumably, I need to login to the chroot and configure the ROS repositories.

@jack-oquin
Copy link
Member Author

jack-oquin commented Mar 5, 2018

I added the ROS repositories like this:

$ ARCH=amd64 DIST=xenial git-pbuilder login --save-after-login
...
root@iron/# sh -c 'echo "deb http://packages.ros.org/ros/ubuntu xenial main" > /etc/apt/sources.list.d/ros-latest.list'
root@iron/# apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116
root@iron/# apt update
root@iron/# exit

@jack-oquin
Copy link
Member Author

jack-oquin commented Mar 5, 2018

After that, the build runs for a long time, but eventually fails with a compile error:

$ gbp buildpackage --git-pbuilder --git-ignore-branch --git-dist=xenial-amd64
...
[ 13%] Building CXX object CMakeFiles/learning.dir/src/agile_grasp/learning.cpp.o
/usr/bin/x86_64-linux-gnu-g++   -DDISABLE_OPENNI2 -DDISABLE_PCAP -DDISABLE_PNG -DQT_CORE_LIB -DQT_GUI_LIB -DQT_NO_DEBUG -DQT_WIDGETS_LIB -DROSCONSOLE_BACKEND_LOG4CXX -DROS_PACKAGE_NAME=\"agile_grasp\" -Dlearning_EXPORTS -DvtkFiltersFlowPaths_AUTOINIT="1(vtkFiltersParallelFlowPaths)" -DvtkIOExodus_AUTOINIT="1(vtkIOParallelExodus)" -DvtkIOGeometry_AUTOINIT="1(vtkIOMPIParallel)" -DvtkIOImage_AUTOINIT="1(vtkIOMPIImage)" -DvtkIOSQL_AUTOINIT="2(vtkIOMySQL,vtkIOPostgreSQL)" -DvtkRenderingContext2D_AUTOINIT="1(vtkRenderingContextOpenGL)" -DvtkRenderingCore_AUTOINIT="4(vtkInteractionStyle,vtkRenderingFreeType,vtkRenderingFreeTypeOpenGL,vtkRenderingOpenGL)" -DvtkRenderingFreeType_AUTOINIT="2(vtkRenderingFreeTypeFontConfig,vtkRenderingMatplotlib)" -DvtkRenderingLIC_AUTOINIT="1(vtkRenderingParallelLIC)" -DvtkRenderingVolume_AUTOINIT="1(vtkRenderingVolumeOpenGL)" -I/build/ros-kinetic-agile-grasp-0.8.0/obj-x86_64-linux-gnu/devel/include -I/usr/include/opencv -I/usr/include/vtk-6.2 -I/usr/include/jsoncpp -I/usr/include/x86_64-linux-gnu -I/usr/include/hdf5/openmpi -I/usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent -I/usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -I/usr/include/libxml2 -I/usr/include/freetype2 -I/usr/include/x86_64-linux-gnu/freetype2 -I/usr/include/tcl -I/usr/include/python2.7 -I/build/ros-kinetic-agile-grasp-0.8.0/include -I/opt/ros/kinetic/include -I/opt/ros/kinetic/share/xmlrpcpp/cmake/../../../include/xmlrpcpp -I/usr/include/eigen3 -I/usr/include/pcl-1.7 -I/usr/include/ni -isystem /usr/include/x86_64-linux-gnu/qt5 -isystem /usr/include/x86_64-linux-gnu/qt5/QtWidgets -isystem /usr/include/x86_64-linux-gnu/qt5/QtGui -isystem /usr/include/x86_64-linux-gnu/qt5/QtCore -isystem /usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g++-64  -DNDEBUG -O3 -fopenmp -Wno-deprecated -Wenum-compare -fPIC   -DDISABLE_LIBUSB-1.0 -fPIC -o CMakeFiles/learning.dir/src/agile_grasp/learning.cpp.o -c /build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp
<command-line>:0:15: warning: missing whitespace after the macro name
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp: In member function 'std::vector<GraspHypothesis> Learning::classify(const std::vector<GraspHypothesis>&, const string&, const Matrix3Xd&, bool)':
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:182:9: error: 'ml' was not declared in this scope
     Ptr<ml::SVM> svm = ml::SVM::create();
         ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:182:16: error: template argument 1 is invalid
     Ptr<ml::SVM> svm = ml::SVM::create();
                ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:182:24: error: 'ml' is not a class or namespace
     Ptr<ml::SVM> svm = ml::SVM::create();
                        ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:186:9: error: 'ml' is not a class or namespace
   svm = ml::SVM::load(svm_filename.c_str());
         ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:194:48: error: base operand of '->' is not a pointer
   std::cout << " # of support vectors: " << svm->getSupportVectors().rows << "\n";
                                                ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:226:25: error: base operand of '->' is not a pointer
   float prediction = svm->predict(features);
                         ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp: In member function 'void Learning::convertData(const std::vector<Learning::Instance>&, const string&, bool, bool)':
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:298:9: error: 'ml' was not declared in this scope
     Ptr<ml::SVM> svm = ml::SVM::create();
         ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:298:16: error: template argument 1 is invalid
     Ptr<ml::SVM> svm = ml::SVM::create();
                ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:298:24: error: 'ml' is not a class or namespace
     Ptr<ml::SVM> svm = ml::SVM::create();
                        ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:302:5: error: base operand of '->' is not a pointer
  svm->setType(ml::SVM::C_SVC);
     ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:302:15: error: 'ml' is not a class or namespace
  svm->setType(ml::SVM::C_SVC);
               ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:304:8: error: base operand of '->' is not a pointer
     svm->setKernel(ml::SVM::LINEAR);
        ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:304:20: error: 'ml' is not a class or namespace
     svm->setKernel(ml::SVM::LINEAR);
                    ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:306:8: error: base operand of '->' is not a pointer
     svm->setKernel(ml::SVM::POLY);
        ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:306:20: error: 'ml' is not a class or namespace
     svm->setKernel(ml::SVM::POLY);
                    ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:307:8: error: base operand of '->' is not a pointer
     svm->setDegree(2);
        ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:311:5: error: base operand of '->' is not a pointer
  svm->train(features, ml::ROW_SAMPLE, labels);
     ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:311:23: error: 'ml' is not a class or namespace
  svm->train(features, ml::ROW_SAMPLE, labels);
                       ^
/build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/learning.cpp:312:5: error: base operand of '->' is not a pointer
  svm->save(file_name.c_str());
     ^
CMakeFiles/learning.dir/build.make:65: recipe for target 'CMakeFiles/learning.dir/src/agile_grasp/learning.cpp.o' failed
make[4]: *** [CMakeFiles/learning.dir/src/agile_grasp/learning.cpp.o] Error 1

@nickswalker: this looks vaguely familiar. Do you remember what the problem was? Some kind of opencv2 vs. opencv3 difference, maybe?

@jack-oquin
Copy link
Member Author

I suspect the code has been migrated to OpenCV3, but the package dependency still refers to libopencv-dev. On Xenial, that resolves to version 2.4.9.1+dfsg-1.5ubuntu1.

It compiles cleanly on my development system, but that may be because it already has opencv3 installed. Since we can't run Travis on this package, the problem would not have been caught, I guess.

Assuming I am right about this, we need to change the dependency to either opencv_bridge, or opencv3.

@jack-oquin
Copy link
Member Author

I hand-patched the release repository to use opencv3, then retried the package build. It got past the OpenCV failure, but got an error here:

cpp.o -c /build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/grasp_localizer.cpp
<command-line>:0:15: warning: missing whitespace after the macro name
In file included from /build/ros-kinetic-agile-grasp-0.8.0/src/agile_grasp/grasp_localizer.cpp:1:0:
/build/ros-kinetic-agile-grasp-0.8.0/include/agile_grasp/grasp_localizer.h:47:36: fatal error: agile_grasp/CloudSized.h: No such file or directory
compilation terminated.
CMakeFiles/grasp_localizer.dir/build.make:65: recipe for target 'CMakeFiles/grasp_localizer.dir/src/agile_grasp/grasp_localizer.cpp.o' failed
make[4]: *** [CMakeFiles/grasp_localizer.dir/src/agile_grasp/grasp_localizer.cpp.o] Error 1

That looks like a completely different problem, so the previous fix looks good. I'll commit that update to our agile_grasp fork.

@jack-oquin
Copy link
Member Author

jack-oquin commented Mar 6, 2018

CloudSized.h is a generated message header, so there's a missing add_dependencies().

I patched that, and the build completed with lots of warnings, but no errors.

jack-oquin added a commit to utexas-bwi/agile_grasp that referenced this issue Mar 6, 2018
jack-oquin added a commit to utexas-bwi/agile_grasp that referenced this issue Mar 6, 2018
jack-oquin added a commit to utexas-bwi/agile_grasp that referenced this issue Mar 6, 2018
@jack-oquin
Copy link
Member Author

jack-oquin commented Mar 8, 2018

I am having trouble figuring out where the generated .deb files go.

The generated files go in the parent of the build directory.

@jack-oquin
Copy link
Member Author

If I run fakeroot debian/rules binary in the origin/debian/kinetic/xenial/agile_grasp branch, the parent directory contains: ros-kinetic-agile-grasp_0.8.0-0xenial_amd64.deb and some other files.

I can install that .deb in my devel system:

$ sudo dpkg -i ../ros-kinetic-agile-grasp_0.8.0-0xenial_amd64.deb 
[sudo] password for joq: 
Selecting previously unselected package ros-kinetic-agile-grasp.
(Reading database ... 430206 files and directories currently installed.)
Preparing to unpack .../ros-kinetic-agile-grasp_0.8.0-0xenial_amd64.deb ...
Unpacking ros-kinetic-agile-grasp (0.8.0-0xenial) ...
Setting up ros-kinetic-agile-grasp (0.8.0-0xenial) ...

After that, I can create a CATKIN_IGNORE file in agile_grasp and build the BWI workspace again.

That's a significant step forward.

@jack-oquin
Copy link
Member Author

Of course, that does not prove that the package can be built in a clean environment, so the git-buildpackage step is still a valuable test.

jack-oquin added a commit to utexas-bwi/agile_grasp that referenced this issue Mar 8, 2018
jack-oquin added a commit to utexas-bwi/agile_grasp that referenced this issue Mar 9, 2018
@jack-oquin
Copy link
Member Author

It turns out that the git-buildpackage approach produces package files in the parent directory, just like fakeroot debian/rules binary does.

@jack-oquin
Copy link
Member Author

The next step is creating a Personal Package Archive. To start with, I'm trying to use LaunchPad.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants