Maintainers can rely on a CI system to enforce build and test checks for PRs for Linux

[CodeMonkeyLeet]

Need to set up a continuous integration pipelines for Linux.

To explicitly scope the goals for this story, most of the CI systems explored only support hosted builds by default or in their basic/free plans, which precludes running on Azure SGX machines. As such, this story primarily covers enforcing basic engineering checks during PR submission:

• Enforcing use of validation scripts for license checking and code formatting
• Building all flavors (debug, release, relwithdebinfo)
• Building against different architectural versions (legacy AESM libraries, NGSA libraries for coffeelake)
• Running all tests supported in simulation mode

For enabling CI to run on Azure SGX machines, see #374.
If we can get a single solution, this can be merged with the Windows one #51.

[anitagov]

Narrowed down to 3 contenders - CircleCI, Travis.CI and GoCD, all of which provide Enterprise options i.e. capability to run the builds on our own servers or own private cloud. CircleCI and Travis.CI do not support Windows builds right now but GoCD does. GoCD supports Windows and Linux builds but it has not been easy to configure the build agents.

[anitagov]

• Checked out CircleCI extensively using a Public GitHub repository that I created for this purpose. Primarily ran the jobs using Docker images/environment and created a YAML file to define the test environment setup.
• Checked out Travis.CI using another Public GitHub repository (similar to the one used for CircleCI). Ran the jobs on a Trusty (Linux) environment using Docker images created earlier. Builds took upto twice as long compared to CircleCI runs.

Docker based deployment of build environment and execution of SGX tests in Dockerized environment POC: - Was able to create a docker container on a KabyLake Linux VM and get OE SDK working. All the tests passed in SGX mode. Saved this docker image for team's use/checkout. The image comes up with the SGX driver but the aesmd service needs to be restarted.

[anitagov]

Tried GoCD again and configured the GoCD server with a demo build pipeline. However, the GoCD agent did not come up and hence the build was stalled. Suspect that the port used by GoCD is being blocked by the firewall.

[anitagov]

For our immediate CI workflows on Linux, my thought is to go with CircleCI which provides 1500 builds and 1 container per month FREE. But we will be able to test only in simulator mode. I am not familiar with using the existing CircleCI webhooks to provide notification when a CI build successfully completes. But the new Checks API from GitHub will soon be integrated and available with CircleCI. I used email notification which works well.

TravisCI for private repositories is not FREE and also seems very slow compared to CircleCI. So, not pursuing this right now.

[anitagov]

Created oeciteam service account for OE CircleCI use. Also a docker hub account for team (oeciteam).
#349 Pull request created for the CircleCI test and checkins

• Checked in Mike's handy precommit script that will enforce license-checks and formatting check.
• Anita created script build-testall that builds and tests for the specified build-type (Debug/Release/RelWithDebInfo), platform (SGX1 or SGX1-FLC) and test-mode (simulation or hardware)
• John created a docker script with the Ubuntu and basic library packages. Also updated cmake to the latest so that skipped tests are not flagged as Fail but as Success.
• We created a YAML script (.circleci/config.yml) and this had all 3 jobs working with the Docker image for the SGX1 platform.
• The results for SGX1 platform are - Success for Debug and RelWithDebInfo and a known failure with the Release build (tests.libcxxrt/test_exception).

[anitagov]

Friday's update:
For the SGX1-FLC jobs to succeed, the private libsgx libraries are required to be on the OE GitHub as CircleCI cannot access Intel's private GitHub.
Added a new parameter NODOWNLOAD to prereqs/Makefile to facilitate the coping of the libsgx packages to ngsa directory.
With this change, all the 3 jobs on the SGX1-FLC platform are running. Samples test Fails and is a known failure.
Total time for all 6 jobs is about 27 minutes.

[anitagov]

Scripts pre-commit and build-testall are part of master along with a config.yml which enforces pre-commit (#PR 430).

With the following optimizations in place in the ci-work branch (#349 ), the run time came down to ~24 minutes.

1. Using Docker image 1.2 with reduced set of packages - only install-prereqs, sudo, git, wget and updated cmake.
2. Doing pre-commit first and a requirement for the other jobs to start. So, if pre-commit fails, the CI test will FAIL.

Here is a picture of the 7 jobs in this workflow:

• Several features such as docker layer caching, ability to skip builds, etc. are not available with the container FREE model.

• Merging the 6 jobs into two did improve the time. However, if one of the configurations fails, the remaining build/tests will not be run. We do not get the granularity of which build in the job failed either.

[anitagov]

CI work branch (PR #349) has been merged to the master finally as all outstanding build/test issues have been fixed. Here is the snapshot of the CI clean run with merge to master.
Setup email forwarding for build updates from the service account to OpenEnclave Maintainers.