Adding CI support for health monitoring of Dockerfile builds

[ruffsl]

Issue

Currently this repo has no internal continuous integration, thus maintainers are given little notice of build issues unless attempting to rebuild locally; typically via someone noticing lapse of refreshing of images on Dockerhub even as upstream images are updated or via official library maintainer feedback during periodic PRs while adding new changes. I'd like to recommend adding an amount of continuous integration for monitoring build health, for at least our official library repos/tags and perhaps for local PRs as well.

Context

This need arises from an issue during the latest release cycle where a subtle change in a upstream parent image, a sudden packages omission in ubuntu, silently resulted in build failures of the official ROS Dockerhub images for a considerable about of time, leaving our official images stale and vulnerable from lack of updates. Specifically, the debian package locals was previously required to generate an expected charmap in setting the system locale. This has been mitigated by moving setup specific commands after the building image has been primed with appropriate apt installs in previous layers. However one should prepare for the unexpected, thus notifications of such occurrence would help improve our response time.

Additionally, CI support of PR validation would also be beneficial, e.g. checking all modified Dockerfiles in a PR continue to build. This would be helpful as a maintainers are not always in possession of suitable network bandwidth to test build Dockerfiles locally, i.e. using --no-cache. This occurred while I was attempting to fix locals issue for past distro tags, my ROS generation script prematurely bumped up the ros meta-package version numbers. Specifically the version info scraped from the rosdistro repo was ahead of what was at the time synced into OSRF's package repo, thus resulting in a build failure I didn't anticipate.

Proposal

I'd like to propose using some CI service such as travis or the like to trigger builds when say either a respective parent image is pushed to DockerHub's registry, or when a sync to a respective distro/version is released onto the package repo. For good measure, a weekly periodic build may also be useful to spot intermittent issues. Ideally I would've liked to tie in directly to whatever the official library managers use to monitor images, but it doesn't seem such exists yet or isn't exposed externally.

We could use webhooks from DockerHub's registry to sense pushes to upstream parent images, and perhaps something equivalent can be done syncs to OSRF's package repo. Github integration for PRs shouldn't be too tricky, although adjusting it to focus on only affected tags may require some more machinery. Another tricky aspect is that due to the nesting of tags our images use to optimize image layer reuse, build order of tags while testing will need to respect the order of parent image hierarchy. If folks have ideas or example in testing dockerfiles using CI, please voice it here.

[tianon]

E: Version '7.7.0-1*' for 'gazebo7' was not found

😉

[tianon]

We've had a lot of success with Travis CI (which is free for open source projects -- we use it for almost all https://github.com/docker-library repos), and it has a new "cron" feature which I've used on a project or two to force periodic builds with great success.

Getting that set up involves logging into https://travis-ci.org, browsing to https://travis-ci.org/profile/osrf, enabling the repo you want, and adding a .travis.yml file to the repo to tell it how to build. There are more settings over at https://travis-ci.org/osrf/docker_images/settings (and that's where "cron" is); I'm a fan of "Build only if .travis.yml is present" and the "Auto Cancellation" features (neither of which are on by default), but YMMV.

[tfoote]

@ruffsl I enabled travis integration for this repo

[ruffsl]

Ok, speaking with @tfoote , We've decided a plan would be to try and create three jobs:

• A weekly cron job that test builds all dockerfiles every week
  • The success of which would be rendered from a matrix of repo/tag badges on the readme
• A daily cron job that checks the dockerfiles with the package repo index
  • This would then use the github API to create a new PR
  • As well as a copy/paste command to make an upstream PR to the official library
• A PR cron job that triggers test builds for altered dockerfiles in the PR
  • This would give build feedback on manual and auto generated PRs

[ruffsl]

Ok, so I've worked up some CI using travis, currently residing on my fork: https://github.com/ruffsl/docker_images/pull/1

So far it creates a job for each repo/tag combo then calls a script that runs the auto generation script for that given repo/tag. Checking travis environment, it determines if the job was triggered from a PR or scheduled cron.

If a PR, then it checks that the auto generation script made no alterations, thus the dockerfiles are up to date, then attempts to build the repo/tag dockerfiles in order via the local make script.

If a cron event, then the repo/tag sub directory is also checked for changes made by the auto generation script. If changes were made then it should submit an auto generated PR for the patch to upstream on one does not yet exists: that bit is still a TODO. If no files were altered, this is also attempts to build the repo/tag dockerfiles in order via the local make script.

A small issue I have with this so far is that if I wanted to create a README with a matrix of build status badges, travis doesn't seem to natively support per-job status images. Because the repo/tags are enumerated using Travis's matrix build ability, via the list of environmental variables, I'm not sure how to render the matrix of build status badges for each. I think normally this can be done on a github branch basis, not a travis job basis. Related: travis-ci/travis-ci#1809

[tfoote]

That's great. Hopefully we can get everything green and then we won't need to worry about the matrix display.