Decide on main CI system

[hkjn]

At time of writing, we have travis-ci.org integration for this repo, and as of today via #88 we also have experimental circleci.com integration.

As described in #39 (comment), Travis has issues with slow builds preventing us from switching to using it for the Armbian OS image builds. We are exploring whether CircleCI is better here.

There's starting to be quite a few different options / ideas about CI, so this issue attempts to gather them into one place so we can make a decision for what system to use that will serve our needs best:

1. Shift Gitlab
2. CircleCI
3. Travis
4. Semaphore2

To expand on the Shift Gitlab alternative, we have a self-hosted Enterprise Edition Gitlab instance which we are evaluating currently. Once we figure out internally what the long-term future is for that tool (which we could use for source code management, CI/CD, deploy pipelines etc) and how we'd work with external contributors like for the BitBox Base project, it might be preferable to use it for CI needs as a default choice.

The needs for the BitBox Base project that I can see are:

1. Run QA checks (tests, linters, etc)
2. Do builds of individual applications, like the middleware
3. Build the main Armbian image
4. (Possibly) Upload build results or otherwise deploy them

[hkjn]

For the purposes of this issue, I'll mention that I tried out setting up integration with the Shift Gitlab instance:

1. I added a bitbox-base project under my own user on the instance
2. configuration locally in the repo was trivial, see hkjn@56962cd
3. the only additional step required was configuring a specific test runner for the project, i.e an agent that can accept tasks and perform builds

With those steps, the main make rule for the master...hkjn:20190621-add-gitlab branch is off to the races. I'll comment here once it finishes. If we do decide to deploy CI to the Shift Gitlab instance, we have some open questions around:

1. how often to produce builds (every commit to every branch, or only master, or ..)
2. whether to use separate or shared test runners
3. how to best make build results accessible to external contributors

Edit: The first attempt to run the CI on Shift's Gitlab ran into this issue, where the job is terminated due to the log file becoming too large, after ~43 min of runtime. We've tried tweaking the config.toml for the job runner to allow for larger log output. The next issue likely would be that the timeout of 60min would be hit. I changed the Settings > CI / CD > General pipelines > Timeout setting to 6h for this test runner to work around that.

Edit 2: The next issue we hit was that some of the files generated in the build/ directory end up being owned by the superuser (the default inside Docker containers, which gets shared to the host system via the volume mount in our docker run commands). On subsequent builds, the Gitlab test runner can't remove these files, so the build immediately fails. Unfortunately, these steps happen before the Gitlab test runner even starts the commands in the .gitlab-ci.yml config, so there's no way to remove the files there. It seems that a workaround to this would be to run the Docker commands inside the container as the same unprivileged user as on the system, to preserve the file ownership.

[hkjn]

As a datapoint, we found that the experimental CircleCI integration could successfully build Armbian OS images (unlike Travis). It took 2h and 54min to finish the build.

[hkjn]

As another datapoint, on the Shift Gitlab instance we managed to produce an Armbian image successfully, after 19m and 39s:

b4ec202a2170eb3c23e1dfec2f103dc1a64dd3e0f03b469d69232cbb4990da8b  output/images/Armbian_5.89_Rockpro64_Debian_stretch_default_4.4.180.img

I'd suggest we address the issues mentioned earlier and move in the direction of using Shift's Gitlab exclusively, unless there's reasons to do something else.

[hkjn]

The experimental Gitlab build integration I mentioned in my last comment in master...hkjn:20190621-add-gitlab branch mostly works fine, and gives superior performance than circleci.com (not surprisingly, since that is a free service while Gitlab is a self-hosted CI system using a decently powerful VM).

The reason I say it "mostly" works fine is that there is an issue where the Dockerized build runs as the super-user (id -u is 0) inside the container, and the bind mount of the build/ directory ends up being owned by the super-user on the host as well. This makes subsequent runs fail, since the non-privileged Gitlab test runner attempts to clean up files that are not part of the repo. Since this step occurs before Gitlab even starts running the build scripts we provide, it's also hard to work around the issue ourselves.

The best path forward would be to run using the same uid and gid inside the build container as on the host. That both achieves the specific goal of allowing Gitlab to build our project without tripping over itself on subsequent builds, and also prevents surprising developers when their unprivileged user doesn't have access to delete files under build/. I've started on this work, and it doesn't look too difficult.

[hkjn]

it doesn't look to [sic] difficult.

Famous last words..

I've sent out #98, which does achieve the result that build/ is owned by the non-privileged user, however in order to achieve this for users who (wisely!) require sudo to run docker commands, the host user id needs to be explicitly set:

$ sudo make docker-build-go BUILDER_UID=$(id -u)

[hkjn]

There were further issues with 0e5efd5 (the commit added in #98) that we only noticed post-merge, which caused the sudo make command to fail if the BUILDER_UID parameter wasn't specified.

I've sent #99 to revert 0e5efd5 until we can figure out a way to achieve a method to run builds while maintaining the ownership of the build/ directory for the non-privileged user, as I described in an earlier comment:

The reason I say it "mostly" works fine is that there is an issue where the Dockerized build runs as the super-user (id -u is 0) inside the container, and the bind mount of the build/ directory ends up being owned by the super-user on the host as well. This makes subsequent runs fail, since the non-privileged Gitlab test runner attempts to clean up files that are not part of the repo. Since this step occurs before Gitlab even starts running the build scripts we provide, it's also hard to work around the issue ourselves.

[Stadicus]

@Tomasvrba, please take a look at this issue so that we can discuss how to proceed

[Stadicus]

Closing due to inactivity. Not a pressing issue, can take this up next year.