Open agnostic CI/CD tooling

[bbros-dev]

Note: This issue was raised by issue #1995

One of the chores in forking a project is setting up the CI/CD tooling so that you can investigate any issues that arise in the course of making a change.

Your best case scenario is that a project has accepted PR's adding CI/CD configurations for different service providers, and one of them is the one you use: In this scenario your chore is to fork and activate the repo your CI/CD provider.

Next best case, they have a CI/CD configuration but not for your CI/CD provider: Now you have to translate/port the configuration and setup. This is irritating, but not uncommon. More irritating is if they insist their CI/CD provider is the one to rule all others and reject a PR for your CI/CD provider.

Worst case there is no publicly visible CI/CD, so nothing in the repo to guide you: What was at best a configuration chore now becomes a time sink. Most irritating is if they insist no CI/CD provider is good enough for them, and reject open CI/CD configurations. Thankfully this is quite rare nowadays.

Firecracker seems it might be that worst case scenario.

Do you accept PR's adding other CI/CD providers?

Ack the test suite might look something like: ./tools/devtool test -- --ignore integration_tests/performance.

[raduweiss]

Hi @bbros-dev . We've missed this. Apologies!

Indeed, right now our CI/CD is not publicly visible. I think this is mostly came about because we were working on Firecracker as a private repo for about a year before open sourcing it, and dealing with external CI providers at that time was tricky.

We're currently looking at moving to Buildkite, and we can definitely think about taking PRs for other CI/CD providers.

But I'm not sure what configuration you're referring to, off the top of my had I imagine that, as you exemplify above, what changes is basically the system you test against. The actual test run should be ./tools/devtool test -- $USER_PARAMS ... or am I missing something? Are you referring to adding additional tests?

[bbros-dev]

we can definitely think about taking PRs for other CI/CD providers.

Great news - lowers the hurdle for devs to runs tests on any changes they make.

Are you referring to adding additional tests?

No just acknowledging that even using an open CI/CD service, it still may be necessary to skip the integration tests.

We're still mystified by what broke the test linked to in #1995, working that out is a huge, avoidable, time sink.

Seeing the test suite status marked green when there was a failing test meant we lost confidence in the CI status indicators on PRs.

We really stumbled on it while trying to workout if there was something about the test suite that would bite us if we tried to set things up in CircleCI.

[raduweiss]

@bbros-dev , looks like there's a few different issues packed in here. I've tried to group them below. Personally, these all sound like good high-level ideas and the best way to move forward would be to open a PR and get down to the details. If we get one or two of these in we can probably even write a useful doc for how people can contribute more.

Transparency for how the master branch required status checks run

We've initially hooked the master branch status checks to AWS-internal systems and it kind of stayed that way (with the exception of the x86_64 AMD Zen status check which runs via a public BuildKite pipeline).

We're currently looking into using BuildKite instead for all existing / future status checks, but our use case has a bunch of constraints and we don't expect this to happen very soon.

Current CI/CD has opaque failures due to the CI/CD, and not the PR

This is/was just a miss on our part. We've pun in effort to fix these kinds of issues in the past months, and haven't seen them lately. Let us know if you still see any, we'd like our CI/CD runners to not behave like that.

Adding other master branch status checks in this repository

This is a workable model, if those status checks are not required/blocking.

I assume this would involve both the addition of some CI/CD configuration to the repository (which would be neatly tucked away in a ".dir" or similar), potentially other changes that would broaden how/where tests run, while still keeping the tests running in their current environments, and the addition of a webhook for that particular CI/CD. Each such check could have a subset of tests configured.

We'd love to see this a PR i this direction, and we can address the specifics there.

Making these checks required would be separate, and much more involved discussion since it has deep operational implications for us.

Adding support for other CI/CD tools but no status checks here

Same as above, but no webhook. Also workable if users find this useful. Looking forward to PRs.

Additional support in the testing tooling for other testing use cases

Also open to this.

A god example of this is #2104. We're open here too and we can just take things on a case by case basis. While we strive to maintain Firecracker the binary program minimalist and laser-focused on the use case, we'd love to review any contributions that make this repository useful to anyone else.

[bbros-dev]

the best way to move forward would be to open a PR and get down to the details.

Agreed. Right now we are setting up Drone based custom/private CI/CD (self-hosted) that might be useful here. It is a different workflow (CI/CD is run before a PR is issued) that envisions upstream having some other CI/CD tooling. We'll know how mcuh might be useful here once we have the kinks ironed out.

We've pun in effort to fix these kinds of issues in the past months, and haven't seen them lately.

Happy for you to close #1995 if you think your changes covered that.

Making these checks required would be separate, and much more involved discussion since it has deep operational implications for us.

I don't think anyone would object to this repo nominating an 'official' CI/CD setup that PR's are evaluated against, leaving the community to maintain/synchronize the 'alternate' CI/CD setup/config files.

Thanks for the feedback.

[dianpopa]

The current discussion does not seem to have any follow up items. Feel free to re-open otherwise.