Proposal: Migrate Cortex CI/CD from CircleCI to GitHub Actions

[AzfaarQureshi]

Is your feature request related to a problem? Please describe.

The current CI/CD provider for Cortex, CircleCI, offers useful features but it also leaves room for a better developer experience. A better developer experience would include having easy access to test results, logs, and artifacts which is not currently possible with CircleCI. Currently, these are not available to developers until they provide CircleCI with OAuth access to their GitHub account. In addition, CircleCI workflows do not work out-of-box for forked repositories. One can personally setup an account with CircleCI and link it to their forked repository. However, their workflows would be subject to the free plan restrictions and can become a barrier for long term contribution when limited free-tier minutes run out. Alternatively, if one does not have a CircleCI account setup, CI tests will only run once a PR is opened upstream. This results in lower quality and unstable initial PRs which wastes development cycles. Another consideration that most developers working in large organizations have is security. Sharing sensitive data, such as OAuth tokens or private repositories to a third party creates potential security vulnerabilities which may be an issue for organizations with strict security requirements.

Describe the solution you'd like

We propose that Cortex migrates from CircleCI to GitHub Actions as its CI/CD provider. Here are the reasons why:

Ease Of Use

CI workflows written in GitHub Actions work out of the box on forked repositories. Currently with CircleCI, contributors are unable to run the CI pipeline until after an upstream pull request is made. Adopting GitHub Actions will ensure more stable and higher quality PRs before community effort is spent reviewing them. This helps increases developer velocity by reducing feedback cycles.

Transparency

GitHub Actions help us align more with the core open source value of transparency. Test results, logs, and artifacts can be viewed freely from within the GitHub repository rather than having this information withheld behind a third party service.

Rich Feature-set

GitHub Actions supports a wider variety of workflows than CircleCI. Github Actions has thirty unique workflow triggers while CircleCI only has two: push and schedule. Workflows like pull request templates, closing stale issues and running custom jobs on new releases are all natively supported with GitHub Actions.

Future Proofing

GitHub Actions provides ARM variants of OS containers while CircleCI does not. This is becoming increasingly relevant to users of Cortex as all the major cloud providers such as Amazon Web Services, Microsoft Azure, Google Cloud Platform, and IBM Cloud, provide ARM compute instances which their deployments could be running on. MacBooks are also switching to ARM so having a CI pipeline which supports tests in these environments is important for the long-term.

Lower Costs

GitHub Actions is completely free for open source repositories. CircleCI only provides an annual credit stipend (works out to be $240/month) which may not be able to serve the needs of larger open source projects as the community grows or workflows become more intensive.

Pros vs Cons of Migrating

Addressing cons

Ability to SSH into test containers
The ability to SSH into test containers is useful especially when debugging the workflows itself. However, any problems relating to the tests themselves can be debugged locally. This is a useful feature but not a vital one. There are also community workarounds available ( 1,2) to achieve the same functionality.

Lack of pipeline insights
Historical analysis and test insights is not natively supported in GitHub Actions. We could manually add telemetry and build a dashboard to display the same metrics if need be.
In our estimation however, CI performance metrics are mainly useful for projects with Continuous Deployment. Due to the periodic nature of releases in open source projects, using CI performance metrics to build marginally faster CI will not result in a productivity impact. There are also no cost concerns with GitHub Actions (no CI minute restriction for open source) that we would have with CircleCI.

Docker Layer Caching and Test Parallelism
These features are not currently being used by the Cortex CI pipeline, so lack of feature parity here will not block migration.

Furthermore, these features are primarily important in repositories with Continuous Deployment as Docker Layer Caching and Test Parallelism help cut CI time between consecutive jobs. Due to the periodic nature of releases in open source projects, having marginally faster CI does not matter in the long run as there are often months between releases so losing these features will not be a long-term hindrance.

Describe alternatives you've considered

TravisCI could be a potential alternative but we recommend moving to GitHub Actions

Additional context

cc - @alolita @tomwilkie @markcartertm @shovnik @jeromeinsf

[pstibrany]

Related to #2340.

[bboreham]

The PR that closed this is "1 of 3" so I guess it should remain open.