Skip to content

Conversation

@stevenbrix
Copy link
Contributor

Testing in Helix

This PR enables running our DRT tests on Helix. I'd recommend seeing the official Helix readme if you are interested in some of the general Helix concepts. I'll briefly outline what we are doing that is a bit unique:

  1. Helix has built-in support for running xUnit tests. Since we are not using xUnit, we have to manually setup our machines so that they work with QualityVault and STI. During the build, we create a payload directory that contains the infrastructure we need. A single project (in our case DrtXaml) is responsible for creating this directory (see instances of the MSBuild property CreateTestPayload).
  2. After the build is done, we utilize Arcade's AfterSolutionBuild.targets extension point to finish creating the rest of the payload if the -test parameter is passed to the build. Here we add the just built DRTs and if -ci was not passed into to the build, run the tests.
  3. Helix allows you to specify a HelixCorrelationPayload directory, where this directory gets deployed to the Helix machine, and is made available in your various helix commands with the HELIX_CORRELATION_PAYLOAD environment variable. We use the payload directory created described above.
  4. Helix and Azure Pipelines can report xUnit logs, so QualityVault was updated to produce an xUnit compatible log. We then copy that log to a known location for it to be picked up. This location can be in any subfolder of the HelixWorkItem working directory.

How we are running tests

  1. Currently, the HelixQueues that we selected are Windows.10.Amd64.Open;Windows.7.Amd64.Open;Windows.10.Amd64.ServerRS5.Open. Essentially, this translates to: "Latest Windows 10", "Windows 7", and "Windows 10 RS5 Server" addition. This enables tests to run on some of the most interesting and important SKUs, without overloading the Helix servers and/or making CI runs take an unnecessarily long time to run.
  2. In a similar fashion, we only run test passes for Release builds, mainly to save time and resources running duplicate tests in a near-identical environment.

Fixes #782

@ghost ghost added the PR metadata: Label to tag PRs, to facilitate with triage label May 31, 2019
@rladuca rladuca requested a review from a team June 1, 2019 04:18
@rladuca rladuca added * NO MERGE * metadata: The PR is not ready for merge yet (see discussion for detailed reasons) Enhancement Requested Product code improvement that does NOT require public API changes/additions labels Jun 1, 2019
@rladuca
Copy link
Member

rladuca commented Jun 1, 2019

Just marking NO MERGE until Preview 6 clears.

@stevenbrix stevenbrix force-pushed the dev/stevenbrix/helix branch 2 times, most recently from 1f01c84 to 1bec645 Compare June 4, 2019 17:59
@stevenbrix stevenbrix removed the * NO MERGE * metadata: The PR is not ready for merge yet (see discussion for detailed reasons) label Jun 4, 2019
@stevenbrix
Copy link
Contributor Author

I removed the no merge tag since the preview6 branch has snapped.

For more detailed information on some of the workarounds, see this comment:
#816 (comment)

@stevenbrix stevenbrix requested review from fabiant3-zz and miguep June 4, 2019 20:54
note: this definitely won't work, it's just some basic initial infrastructure for us to build on once we have a better idea of what to do

use dummy HelixWorkItem initially

updating pipeline.yml

fix Release only condition
@stevenbrix stevenbrix merged commit 628e8f1 into master Jun 6, 2019
@vatsan-madhavan vatsan-madhavan deleted the dev/stevenbrix/helix branch August 23, 2019 20:13
@ghost ghost locked as resolved and limited conversation to collaborators Apr 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Enhancement Requested Product code improvement that does NOT require public API changes/additions PR metadata: Label to tag PRs, to facilitate with triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Get CI builds running in Helix

6 participants