Use colcon as the build tool

[esteve]

Status

PRODUCTION / DEVELOPMENT

Description

Use the ROS2 build tool (colcon) to build Autoware in preparation of integrating more ROS2 code.

Extra requirements

colcon requires the following packages:

sudo apt-get install -y python3-pip python3-setuptools
sudo pip3 install -U setuptools
sudo apt-get install -y python3-colcon-common-extensions

[esteve]

This is now ready for review. These changes switch the build tool to colcon, the difference between catkin_make and colcon is that colcon can build both ROS1 (catkin) and ROS2 (ament) packages, whereas catkin_make can only build ROS1. This would allow developers to add ROS2 packages to Autoware.ai and build them from the same workspace without using two different build tools.

[esteve]

This does not compile as is because it needs an updated Docker image. @kfunaoka could you add me to the Dockerhub organization so I can upload images? Thanks.

[kfunaoka]

@esteve Sorry for the very very late responses. I've added you into DockerHub repo. Would you try?

[esteve]

@kfunaoka thank you so much for adding me to the Docker hub, I just tested it and uploaded an updated image for synquacer. I'll upload images for aarch and driveworks too.

[esteve]

Here's a CI run https://gitlab.com/esteve/Autoware/pipelines/37505434

Some things:

• build_cross builds fine because no tests are run
• build_kinetic fails because of tests in the velodyne driver, so they are not introduced by this pull request
• build_indigo aborts because colcon requires Python 3.5, but Ubuntu 14.04 ships with Python 3.4.

Once we drop Indigo, the last point won't be an issue.

[esteve]

I've updated the description with the colcon dependencies:

sudo apt-get install -y python3-pip python3-setuptools
sudo pip3 install -U setuptools
sudo apt-get install -y python3-colcon-common-extensions

[esteve]

CI now passes:

https://gitlab.com/esteve/Autoware/pipelines/39074470

[sgermanserrano]

Mark catkin scripts to be deprecated but leave them for now

[bdholt1]

I haven't tested it but I completely support this PR!

[kfunaoka]

@esteve Thank you very much! I've tried the colcon_release script at ac52f44a4f. The following error messages appear. It seems msgs are not generated. Should I do something?

In file included from /home/autoware/Autoware/ros/src/common/libvectormap/include/libvectormap/vector_map.h:14:0,
                 from /home/autoware/Autoware/ros/src/common/libvectormap/src/vector_map.cpp:4:
/home/autoware/Autoware/ros/src/data/packages/vector_map/include/vector_map/vector_map.h:40:40: fatal error: vector_map_msgs/PointArray.h: No such file or directory

In file included from /home/autoware/Autoware/ros/src/sensing/drivers/lidar/packages/velodyne/velodyne_pointcloud/src/lib/rawdata.cc:35:0:
/home/autoware/Autoware/ros/src/sensing/drivers/lidar/packages/velodyne/velodyne_pointcloud/include/velodyne_pointcloud/rawdata.h:31:40: fatal error: velodyne_msgs/VelodyneScan.h: No such file or directory

Travis also shows the error.
https://travis-ci.org/CPFL/Autoware/jobs/467423253

[sgermanserrano]

@kfunaoka I think I had the same issue, have you removed the CMakeLists.txt inside ros/src before executing colcon_release?
Regarding the pipeline error, this one from @esteve Gitlab CI completes successfully. The travis.yml file might be out of date as we moved to Gitlab CI.

[esteve]

@kfunaoka I remember @sgermanserrano having the same problem, I think removing CMakeLists.txt fixed it, but I also remember making other changes. Anyway, as @sgermanserrano says, Travis is no longer relevant since we switched to GitLab for the develop branch. Once Autoware 1.10 is released, we can remove Travis and depend on GitLab only.

[kfunaoka]

@sgermanserrano @esteve Thank you very much! Built successfully!

It seems the following ./run does not show GUI called runtime_manager.
Does ./run work in your environment?

$ cd Autoware/ros
$ ./run

The following manual commands does not work also. Should I do something else?

$ source install/setup.zsh
$ rosrun runtime_manager runtime_manager_dialog.py
[rosrun] Couldn't find executable named runtime_manager_dialog.py below /home/autoware/Autoware/ros/install/runtime_manager/share/runtime_manager

[esteve]

@kfunaoka thanks for testing it. I haven't tried running the runtime manager, let me check and I'll get back to you.

[esteve]

@kfunaoka I figured out why the runtime_manager is not available. The package files (i.e. CMakeLists.txt) are not properly written and thus the script is not installed. colcon requires all packages to be well-formed with installation targets, since the devel space no longer exists.

I'll fix the runtime_manager package and try again. Let's postpone this PR for 1.11

[kfunaoka]

@esteve Thank you for finding out the root cause!

[esteve]

@kfunaoka I've fixed the CMake scripts for the runtime manager, given that 1.10 is now delayed, I'd like to get this merged. Could you have a look and see if it works for you?

[esteve]

I'm doing further testing because I found a bug in the legacy catkin scripts, unfortunately this is not ready for merging yet.

[esteve]

I've fixed the issues and the catkin scripts work again, this should be mergeable again now.

[kfunaoka]

@esteve Thanks! I've faced the error at b641679:

$ ./colcon_release
...
Starting >>> vector_map
--- stderr: ndt_cpu
** WARNING ** io features related to openni2 will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
** WARNING ** visualization features related to openni2 will be disabled
---
Finished <<< ndt_cpu [37.6s]
Starting >>> amathutils_lib
--- stderr: velodyne_pointcloud                                               
CMake Error at src/lib/CMakeLists.txt:1 (find_package):
  By not providing "Findvector_map_msgs.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "vector_map_msgs", but CMake did not find one.

  Could not find a package configuration file provided by "vector_map_msgs"
  with any of the following names:

    vector_map_msgsConfig.cmake
    vector_map_msgs-config.cmake

  Add the installation prefix of "vector_map_msgs" to CMAKE_PREFIX_PATH or
  set "vector_map_msgs_DIR" to a directory containing one of the above files.
  If "vector_map_msgs" provides a separate development package or SDK, be
  sure it has been installed.

---
Failed   <<< velodyne_pointcloud	[ Exited with code 1 ]
--- stderr: ndt_gpu
** WARNING ** io features related to openni2 will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
** WARNING ** visualization features related to openni2 will be disabled
...
RuntimeError: Event loop is closed
[112.830s] ERROR:concurrent.futures:exception calling callback for <Future at 0x7f4951c4a898 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.5/asyncio/futures.py", line 442, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 532, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[112.846s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/lib/python3/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>
[112.846s] ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_fd2callback() done, defined at /usr/lib/python3/dist-packages/colcon_core/subprocess.py:220> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>

Are there any advices to resolve the problem?

[esteve]

@kfunaoka just did another run of ./colcon_release and noticed that vector_map_msgs was missing as a dependency. I fixed it it fc7785a, could you try again? Thanks :-)

[kfunaoka]

@esteve It works! Thank you. Since this branch is not in CPFL, GitLab CI is not triggered. Should this branch is pushed into CPFL/autoware as @sgermanserrano suggested?
https://autoware-developer.slack.com/archives/CEB3KFZK3/p1543230404015400

[esteve]

@kfunaoka thanks for testing it again. I've triggered a build on my Gitlab fork:

https://gitlab.com/esteve/Autoware/pipelines/42387470