Should we move build pipeline to GitHub actions?

[mconnew]

@tibor19

[Kralizek]

As someone who is evaluating the same decision on other projects, which are the pros and cons of this?

[mconnew]

I don't know the answer to these questions yet, but I have a bunch of questions I'd like to find answers to as part of the decision making process:

What differences are there between CPU time allowed between the two options. CoreWCF is just going to be gaining more and more test scenarios over time and I expect the full test suite to take a long time eventually. The full WCF set of tests on NetFx takes hours to run so this might be an issue in the future. How much extra does it cost to buy more compute time for running PR's and CI builds? In theory could keep CI on Azure Devops as only maintainers need to care about CI builds, but then you end up managing and supporting two slightly different pipeline definitions.

One problem with Azure DevOps is very poor permissions granularity for with GitHub integration when running the pipelines. When paired with a GitHub repo, you have to give quite a lot of repo permissions for someone to be allowed to re-run a failed pipeline build. You might be comfortable enough with someone to be able to do that, but not comfortable enough to give them the rest of the capabilities which they get with it. This one is probably the strongest thing pushing me to thinking GitHub actions are a better choice.

Another future issue for CoreWCF is running multiple containers as part of the test run. Eventually we're going to have a docker container to fire up an adhoc domain controller, another one to run a Windows client to have .NET Framework clients (for interop testing), and a third one for the CoreWCF tests to live in. We could end up with a 4th one for running things like MSMQ or other message queue servers. I don't know how flexible GitHub is for that.

Code Coverage reports. Although it's broken right now, Azure Devops has a nice ability to publish code coverage reports and to track trends in code coverage from one build to the next. I don't know what the abilities are around that in GitHub.

Range of OS images supported. Although we aren't testing on it, and I'm not sure if it's really needed, but running tests on MacOS is possible in Azure DevOps. I don't know if GitHub has that or not. At the moment, it's not relevant, and I don't expect it to become so, but it's a datapoint worth knowing. We might also need to manage our own OS images in the future for Windows auth testing as for Linux I think we might need to have root access to set up the MIT kerberos stack to get things working. I'm not 100% sure on this as I have a vague memory that we might be able to get away with a local config using environment variables to set things up.

Secret store management. We have multiple secrets we would need to manage for the pipeline. One of them is for signing packages in the CI pipeline, another potential one is pushing packages to nuget which is currently done manually by me. I don't know how those secrets are managed, so I don't know if I'll be happy with the mechanism.

Anyway, that's my braindump of things I've been thinking about around this decision. Hopefully some of it might be useful for you.

[Kralizek]

Thanks. I would say that the cpu limits is the most concrete problem. Luckily my projects fits nicely within the boundaries of the free tier of AppVeyor that handles quite nicely all the rest of your concerns.

Is being part of the .net foundation giving any resources in terms of build tools?

[mconnew]

Yes, being a part of the .NET Foundation does come with some resources in Azure Devops.

[StingyJack]

One problem with Azure DevOps is very poor permissions granularity for with GitHub integration when running the pipelines. When paired with a GitHub repo, you have to give quite a lot of repo permissions for someone to be allowed to re-run a failed pipeline build. You might be comfortable enough with someone to be able to do that, but not comfortable enough to give them the rest of the capabilities which they get with it.

if someone wants to rerun a pipeline build they can close and reopen the GH PR to achieve that. I'm an admin and still have to do it that way when I make changes to a build pipeline because rerunning a PRV build does not use an updated definition.

IDK about GH actions, but DevOps pipelines manages secrets pretty well and tries very hard to hide the values anywhere they could leak out in the logs or UI. At my current job, we also use Azure and AWS secret managers for some things in our build pipelines. They dont get leaked either.

[g7ed6e]

Code Coverage reports. Although it's broken right now, Azure Devops has a nice ability to publish code coverage reports and to track trends in code coverage from one build to the next. I don't know what the abilities are around that in GitHub.

Isn’t sonarqube saas free for public project ?
https://www.sonarsource.com/plans-and-pricing/