Use GitHub Actions CI

[scpeters]

We'd like to try using GitHub Actions CI for maliput and other delphyne dependencies, which will be especially useful after open-sourcing these repositories.

I've started by looking at maliput, which is easiest because it doesn't have any private dependencies. To checkout private GitHub repositories, you need to add a GitHub token as a secret inside the repository, so it's much easier to configure GitHub actions for repositories with only public dependencies.

My first attempt is in the following branch:

• https://github.com/ToyotaResearchInstitute/maliput/compare/scpeters/actions_ci

It tries to avoid using packages.ros.org by installing python packages using pip and adding ament to the colcon workspace. I ran into some trouble configuring rosdep, so I'm going to try using https://github.com/ros-tooling/action-ros-ci to see if that's easier to configure.

[scpeters]

I've been experimenting with https://github.com/ros-tooling/action-ros-ci in the following branch:

• https://github.com/ToyotaResearchInstitute/maliput/tree/scpeters/action-ros-ci

It recommends using the https://github.com/ros-tooling/setup-ros action step to configure the system for using ROS (installing build tops, adding ROS apt repositories), which I plan to use for configuring base system. The action-ros-ci workflow doesn't have great support for private repositories though. As outlined in ros-tooling/action-ros-ci#224, despite the fact that actions/checkout@v2 can checkout a private repository without configuring an extra token, action-ros-ci does not use this. They added a parameter for specifying a token in ros-tooling/action-ros-ci#252, but I think there's still room for improvement there.

For now, I will use ros-tooling/setup-ros and configure the rest manually.

[agalbachicar]

Thanks for the update!

[agalbachicar]

Relates to #135

[scpeters]

It looks like ros-tooling/setup-ros installs DDS implementations, which are not needed, but also shouldn't hurt anything, so I will keep using it for now

[scpeters]

I happened to try a build with ROS eloquent instead of dashing, and it gave the following error:

--- stderr: maliput
CMake Error at /opt/ros/eloquent/share/ament_cmake_target_dependencies/cmake/ament_target_dependencies.cmake:52 (message):
  ament_target_dependencies() the passed package name 'maliput' was not found
  before
Call Stack (most recent call first):
  test/utilities/CMakeLists.txt:23 (ament_target_dependencies)
  test/utilities/CMakeLists.txt:36 (add_dependencies_to_test)


---

• https://github.com/ToyotaResearchInstitute/maliput/runs/1073729710

I'm guessing some of the ament syntax has changed since dashing, so we should keep this in mind when updating to a new version of ros2 / ament.

[scpeters]

first approach in maliput/maliput#343

[scpeters]

maliput/maliput#343 has merged. Next steps could include:

• more CI steps in maliput (clang, sanitizers, linters)
• adding GitHub actions to other packages (will require access tokens)

[agalbachicar]

Before moving forward with more CI steps, I think we should take another package, let's say maliput-dragway which consumes maliput as a dependency.

I think a reasonable path forward would be:

• maliput
• maliput-dragway
• maliput-multilane
• maliput-integration
• maliput-integration-tests
• malidrive
• delphyne
• delphyne-gui

On a separate note, which linters are you thinking of? We have clang-format that checks the code is properly linted and similarly we have for python. That is already checked via ctests. Am I missing something else?

[scpeters]

Before moving forward with more CI steps, I think we should take another package, let's say maliput-dragway which consumes maliput as a dependency.

I think a reasonable path forward would be:

• maliput
• maliput-dragway
• maliput-multilane
• maliput-integration
• maliput-integration-tests
• malidrive
• delphyne
• delphyne-gui

ok, shall I create a new GitHub user account to hold read access to these repositories so we can use its token in the Actions CI?

On a separate note, which linters are you thinking of? We have clang-format that checks the code is properly linted and similarly we have for python. That is already checked via ctests. Am I missing something else?

sorry, I meant static analyzers, like scan-build

[agalbachicar]

ok, shall I create a new GitHub user account to hold read access to these repositories so we can use its token in the Actions CI?

@stonier @liangfok would you provide us with that account or shall we create it?

[scpeters]

Todo @scpeters: reach out to https://github.com/ros-tooling/setup-ros maintainers to see if they are willing to create a separate action that doesn't install DDS implementations

[scpeters]

Steps required for enabling GitHub Actions CI on a new package (starting with https://github.com/ToyotaResearchInstitute/maliput-dragway)

1. If it doesn't already exist, create a dedicated GitHub user account
2. If it doesn't already exist, create a Personal Access Token for that dedicated GitHub user account with repo scope. Be sure to store it somewhere secure because you can only view it immediately after creation.
3. Grant the dedicated GitHub user account read access to all relevant private repositories. (Strictly speaking, the dedicated GitHub user account only needs read access to the dependencies of the repositories to be tested. For example, if we were only planning to enable Actions CI for maliput-dragway, then it would only need read access to maliput. If you want to set up permissions and secrets once and not have to keep editing them, it's probably easiest to just give read-access to dsim-repos-index and all private repositories in the dsim.repos file).
4. Add the Personal Access Token created in step 2 as a secret to the repository for which Actions CI is to be enabled with a consistent name like MALIPUT_TOKEN or MALIPUT_READONLY_TOKEN. (For example, this can be added to maliput-dragway at https://github.com/ToyotaResearchInstitute/maliput-dragway/settings/secrets.) maliput doesn't need this secret, but all packages that depend on it will need the secret in order to enable Actions CI.

cc @liangfok @stonier

[scpeters]

Todo @scpeters: reach out to https://github.com/ros-tooling/setup-ros maintainers to see if they are willing to create a separate action that doesn't install DDS implementations

it looks like the logic for installing DDS implementations is intermingled with compilers and development tools in the package_manager/apt.ts file.

DDS-specific lines:

• https://github.com/ros-tooling/setup-ros/blob/39f9fe76c8bf5baaa/src/package_manager/apt.ts#L7
• https://github.com/ros-tooling/setup-ros/blob/39f9fe76c8bf5baaa/src/package_manager/apt.ts#L30-L34
• https://github.com/ros-tooling/setup-ros/blob/39f9fe76c8bf5baaa/src/package_manager/apt.ts#L39-L40
• https://github.com/ros-tooling/setup-ros/blob/39f9fe76c8bf5baaa/src/package_manager/apt.ts#L52-L53

[scpeters]

Todo @scpeters: reach out to https://github.com/ros-tooling/setup-ros maintainers to see if they are willing to create a separate action that doesn't install DDS implementations

it looks like the logic for installing DDS implementations is intermingled with compilers and development tools in the package_manager/apt.ts file.

DDS-specific lines:

• https://github.com/ros-tooling/setup-ros/blob/39f9fe76c8bf5baaa/src/package_manager/apt.ts#L7
• https://github.com/ros-tooling/setup-ros/blob/39f9fe76c8bf5baaa/src/package_manager/apt.ts#L30-L34
• https://github.com/ros-tooling/setup-ros/blob/39f9fe76c8bf5baaa/src/package_manager/apt.ts#L39-L40
• https://github.com/ros-tooling/setup-ros/blob/39f9fe76c8bf5baaa/src/package_manager/apt.ts#L52-L53

ros-tooling/setup-ros#268

[scpeters]

experimenting with compiling maliput with both gcc and clang in maliput/maliput@b14c695, but it currently only has clang 6 installed. I need to copy the following logic in order to install the correct value of clang (currently version 8):

• https://github.com/ToyotaResearchInstitute/dsim-repos-index/blob/b5c7befb8d30a4a7dfc82dd9d2802c4faebcdf57/tools/etc/default.prereqs#L7-L93

[scpeters]

experimenting with compiling maliput with both gcc and clang in ToyotaResearchInstitute/maliput@b14c695, but it currently only has clang 6 installed. I need to copy the following logic in order to install the correct value of clang (currently version 8):

• https://github.com/ToyotaResearchInstitute/dsim-repos-index/blob/b5c7befb8d30a4a7dfc82dd9d2802c4faebcdf57/tools/etc/default.prereqs#L7-L93

I submitted an issue ros-tooling/setup-ros#276 requesting a parameter to make it easy for a specific version of clang be installed. Now that I think about it; it might make more sense for this to be done as a separate action, though I don't have any experience making actions.

[scpeters]

enabling Actions CI for dragway: maliput/maliput_dragway#9

[agalbachicar]

it might make more sense for this to be done as a separate action, though I don't have any experience making actions.

I've seen actions that install apt packages but haven't seen one that adds a new repo source. I would try to split this task into:

1- Installation
2- Verify the installation
3- Set the clang alternative
4- Use clang with colcon build.

What do you think?

[agalbachicar]

Also, I came across this PR: https://github.com/onqtam/doctest/pull/285/files that includes several compilers. It might help.

[agalbachicar]

@scpeters how difficult would it be to either setup a maliput-actions repository or put all the common stuff in here.

From reviewing all the PRs, I could see that the following is repeated:

• Base image setup.
• Checkout public repositories.
• Checkout private repositories.
• Dependency installation (mostly ros-related and build system tools).
• gcc
  • Build.
  • Test.

And in the future we will add:

• clang
  • Build
  • Test
• Sanitizers ...

If we can pass the tokens and the repositories when checking out the private repositories and the package names when testing, I think our action files will be considerably reduced.

BTW, I don't think we need to address it now, I think it would be nice to have and understand what it would take.

[scpeters]

@scpeters how difficult would it be to either setup a maliput-actions repository or put all the common stuff in here.

From reviewing all the PRs, I could see that the following is repeated:

• Base image setup.

• Checkout public repositories.

• Checkout private repositories.

• Dependency installation (mostly ros-related and build system tools).

• gcc

  • Build.
  • Test.

And in the future we will add:

• clang

  • Build
  • Test

• Sanitizers ...

If we can pass the tokens and the repositories when checking out the private repositories and the package names when testing, I think our action files will be considerably reduced.

BTW, I don't think we need to address it now, I think it would be nice to have and understand what it would take.

I think it could be nice to generalize these actions if it can reduce duplication of the configuration and build scripts. At the moment, these actions are still under development and don't involve too much duplication, so I'd prefer to continue with the present setup for now but keep this in mind in case there is more duplication that we want to mitigate

[agalbachicar]

Status: all repositories that require a gcc build have Github Actions.

[scpeters]

our Jenkins CI has some logic to switch branches for package dependencies if they have a branch name matching the current branch of the leaf package to be tested:

• https://github.com/ToyotaResearchInstitute/dsim-repos-index/blob/edaf1afa8951b9548f411eee3500252994be1286/ci/jenkins/pull_sources#L30-L41

this is not enabled for GitHub Actions yet, but it could be. it would require setting fetch-depth: 0 in actions/checkout to checkout the full history of all branches and tags

[agalbachicar]

About scan_build:

• Copy to each repository the script to parse the suppression list.
• List all dependencies' suppression list to each leaf package.

[agalbachicar]

Per maliput/delphyne#717 (comment) investigation, we should:

• Fix all the occurrences of uses: ros-tooling/setup-ros@0.0.25, to be:

    - uses: ros-tooling/setup-ros@0.0.25
      env:
        ACTIONS_ALLOW_UNSECURE_COMMANDS: true

• Notifiy about the error in https://github.com/ros-tooling/setup-ros

See https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/ for context about the change.

[scpeters]

I think if we bump to ros-tooling/setup-ros@0.0.26, then we won't need the ACTIONS_ALLOW_UNSECURE_COMMANDS workaround. Testing in maliput/maliput#365

[scpeters]

our Jenkins CI has some logic to switch branches for package dependencies if they have a branch name matching the current branch of the leaf package to be tested:

• https://github.com/ToyotaResearchInstitute/dsim-repos-index/blob/edaf1afa8951b9548f411eee3500252994be1286/ci/jenkins/pull_sources#L30-L41

this is not enabled for GitHub Actions yet, but it could be. it would require setting fetch-depth: 0 in actions/checkout to checkout the full history of all branches and tags

due to maliput/maliput#368 requiring changes in some downstream packages, I'm adding logic for checking out matching branch names if available in those packages:

• Use maliput::test_utilities and try same branch name in actions maliput_dragway#18
• Use maliput::test_utilities and try same branch name in actions maliput_multilane#30
• https://github.com/ToyotaResearchInstitute/malidrive/pull/728
• Use maliput::test_utilities and try same branch name in actions maliput_integration_tests#24

it still needs to be added to a few other repositories

[agalbachicar]

Implementation has finished.