Use Azure Pipelines instead of Travis (and maybe other builds)?

[jnm2]

Hey, are we interested in using this for some or all of our builds? We don't have parallel jobs on Travis and sometimes Travis machines have network issues. This looks pretty exciting:

https://azure.microsoft.com/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/

Starting today, Azure Pipelines provides unlimited CI/CD minutes and 10 parallel jobs to every open source project for free. All open source projects run on the same infrastructure that our paying customers use. That means you’ll have the same fast performance and high quality of service.

Image from the link:

The link also shows a GitHub integration.

[ChrisMaddock]

If someone was interested enough to set this up, I'd like to see us first use it alongside Travis for a while, before making a decision. Just to be sure this doesn't have the same reliability problems for OSX as Travis does.

Looks cool though! 🙂

[rprouse]

Great idea. I'd like to see us make more use of the builds there and eventually the releases. If anyone wants to jump in, I've made sure that @ChrisMaddock and @jnm2 are both project admins of https://nunit.visualstudio.com/NUnit/. If any other @nunit/framework-and-engine-team member wants to help out with our Azure Pipeline builds, let me know.

[rprouse]

I've made a bit of progress on this tonight. I've improved the existing Windows CI build to build, test and package. I also have the macOS build doing the same although I am getting a consistent test failure on macOS. The nice thing is I publish test results, so you don't have to scan the logs to see the test failures!

I am working on the Linux build. I am having trouble getting a 1.1 and 2.0 version of .NET Core installed on the build agents...

[rprouse]

Linux is now working. I just needed to switch from the Use .NET Core Version tasks to a general task that does an apt-get install of the version that was missing.

All builds on all platforms do a build, test and package then publish as artifacts. They are currently building without dev build numbers like on AppVeyor. We should update the Cake script to do an IsRunningOnVSTS and use the same logic there as on AppVeyor.

Once we get to that step, I would love to move on to the next step of building the release branch changes and auto-deploying/releasing them. We'll need to think about how we maintain release notes and the changes.txt at that point. Automate all the things?

[rprouse]

@mikkelbu I've added you as another team member of the DevOps Pipeline (aka VSTS) project, so you should be able to get in and make changes if needed. Let me know if you run into permissions problems and I will fix.

[jnm2]

All Azure builds are currently reporting success when they should report failure.

[chrispat]

Interesting discussion here. I am a PM on the Azure Pipelines team. Please feel free to ping me with any questions.

[jnm2]

@chrisrpatterson I have a question, if I may I take you up on that. I apologize that it's a little long in order to be clear.

Problem

We have 18 test runs that we'd like to publish, 6 from each OS (see list).
(The reason why the number cannot be fewer than 18 is because these tests have the exact same names. Each test runs 18 times, covering every combination of OS and target framework. If we went with fewer test run uploads, Azure DevOps would merge the tests with the same names and we would not be able to tell which OS and target framework the test had failed for.)

The 'Publish test results' task only supports creating one test run, so if we have 18 test runs, we need to repeat this block 18 times in our YAML:

  - task: PublishTestResults@2
    displayName: Publish test results
    inputs:
      testResultsFormat: NUnit
      testResultsFiles: test-results\net45\*.xml
      mergeTestResults: true
      testRunTitle: net45/Windows
    condition: succeededOrFailed()

This makes the YAML dense and hard to read. Also, the specific six target frameworks we support may change in the future. We'll invariably forget that we have this coupling to the specific test-results\net45\*.xml paths and only remember to fix the YAML once we notice test results have been going missing. It's just not very satisfactory.

Attempt at automation

Since we're using a Cake build script, it seems like a good idea to automate these uploads using C# to print this out as soon as each set of files is ready, just like the 'Publish test results' task:

##vso[results.publish type=NUnit;mergeResults=true;config='Release';runTitle='netcoreapp2.0/Windows';publishRunAttachments=true;resultFiles=D:\a\1\s\test-results\netcoreapp2.0\nunit.framework.tests.xml,D:\a\1\s\test-results\netcoreapp2.0\nunitlite.tests.xml]

Publishing from our own scripts instead of publishing using the 'Publish test results' task has strong benefits:

• We don't have to repeat that chunk of YAML 18 times. The YAML is now easy to read. Also, we won't have to keep out six specific frameworks matching in the YAML, so we won't have missing test results if we add a target framework because we forgot to update the YAML file.

• We can watch test results roll in as each of the test runs completes, rather than waiting until the 'PowerShell' step is finished and then doing the 'Publish test results' step. This is less essential, but its a cool feature which would never be possible when using the 'Publish test results' task to publish the test results.

Problem with attempted automation

➡ It only works if the build jobs happen to pick up v2 agents out of the queue. Most of the time, the Windows build picks up a v1 agent which totally ignores the ##vso[results.publish...] logging command.

How does the 'Publish test results' task normally solve this? It prevents v1 agents from getting picked up because it has a task.json which specifies "minimumAgentVersion": "2.0.0".

There is no equivalent of minimumAgentVersion for our azure-pipelines.yml. The best that pipelines YAML can do is demands: agent.version -equals 2.142.2 or whatever the exact agent version happens to be today, but that breaks as soon as the agents are updated to version 2.142.3. So this is no good.

The only way to get an agent with version greater than or equal to 2.0.0 is to add a dummy 'Publish test results' task which uploads nothing and leaves warnings in our build summaries, or else abandon the idea of automation and go back to the 18 variations of this block of YAML in our azure-pipelines.yml:

  - task: PublishTestResults@2
    displayName: Publish test results
    inputs:
      testResultsFormat: NUnit
      testResultsFiles: test-results\net45\*.xml # Or net 40, or 5 others
      mergeTestResults: true
      testRunTitle: net45/Windows # Or net40/Linux, or 16 others
    condition: succeededOrFailed()

I found this request which is marked Closed - Not Enough Info: https://developercommunity.visualstudio.com/content/problem/75687/build-demands-agentversion-gtversion-21020.html

So this is me striving to provide more info. Was this helpful? Would you like me to move my question to a better forum? Unless your team knows of any workarounds, we'll probably go the 18-copies-of-mundane-YAML route.

Much thanks!

[jnm2]

@chrisrpatterson I apologize for bugging you. What's the best avenue for me to pursue to get the issue resolved? Is there something I can clarify?

[chrispat]

@jnm2 The automation example you have seems fine. I am curious to know how you have V1 agents at all as they are not really supported anymore. There also should be no v1 agents in the hosted pool at all.

[jnm2]

@chrisrpatterson The v1 agents were showing up at the end of November:

Uploading test results only happens if the jobs happen upon the correct agents. When it works, it says "Agent: Hosted VS2017 2." When it fails, it says "Agent: Hosted Agent." Similar for Linux and macOS.

I wonder if this is what's going on:

https://github.com/Microsoft/azure-pipelines-tasks/blob/f8526d84cde186125cc589433fb950d437d1ef77/Tasks/PublishTestResultsV2/task.json#L20

"minimumAgentVersion": "2.0.0"

Now that I'm not using task: PublishTestResults@2 in the YAML, it's not requiring the capable agents.

I didn't realize it was a point-in-time issue, but that makes sense. I'll try the automation route again. Thanks for the info!

[rprouse]

Closing this as done.

[jnm2]

@chrispat I apologize! I think I figured out what was going on.

Uploading test results only happens if the jobs happen upon the correct agents. When it works, it says "Agent: Hosted VS2017 2." When it fails, it says "Agent: Hosted Agent." Similar for Linux and macOS.

I was using Cake's IsRunningOnTFS which returns false if AGENT_NAME starts with "Hosted"! The reason test results weren't showing up had nothing to do with the agent version but rather Cake's check being unreliable and our script thinking it was not on Azure Pipelines and therefore not even printing out ##vso[results.publish etc.

My fault for not seeing this earlier. Thank you very much for your help!