Set up Semaphore CI

[Brian Strauch (brianstrauch)]

Checklist

1. [CRUCIAL] Is the change for CP or CCloud functionalities that are already live in prod?
   • yes: ok

What

Use Semaphore CI instead of Azure Pipelines! With the help of caching, total time is < 10min (as opposed to ~30min with the current system)

TODOs:

• Only trigger CI for PRs and commits to main (project settings)
• https://docs.semaphoreci.com/examples/golang-continuous-integration/#managing-module-dependencies-in-private-repositories-on-github
• Cache Go modules
• Prevent fork builds (https://docs.microsoft.com/en-us/azure/devops/pipelines/security/repos?view=azure-devops#forks)
• Windows
• Test reports

[David Hyde (DABH)]

Brian Strauch (@brianstrauch) What is the latest on this? Wondering if it makes sense to migrate to Semaphore e.g. this week.

Also, since CPD in theory is using prod versions of services now, maybe it's a bit more stable, maybe we should try not skipping the system tests pipeline step for now, and/or trying to run system tests from semaphore? wdyt? Can always just skip it again if it's still flaky

[Brian Strauch (brianstrauch)]

I've been working on this on-and-off. There's still a bit more to do (see the TODO list above), and I'm not sure if we have Windows support yet... and yes! I would love to have Semaphore run our system tests too.

[Brian Strauch (brianstrauch)]

Waiting on https://confluentinc.atlassian.net/browse/DP-8693

[David Hyde (DABH)]

LGTM up to couple questions/comments

[David Hyde (DABH)]

Any reason to kill this? Why not keep it until we migrate this to sem as well?

[Brian Strauch (brianstrauch)]

It's been disabled for almost a whole year, so I thought we would want to start from scratch, but sure, I suppose I can add it back just in case.

[David Hyde (DABH)]

Why??

[Brian Strauch (brianstrauch)]

This was added as a hack to fix a weird panic(). That's been fixed, so it should be fine to remove it. We're just running confluent -h here so there should be no reason that the contents of the config file change the output.