Use Arcade

[ryanbrandenburg]

First pass at updating this repo to use dotnet/Arcade.

[natemcmaster]

A good start. The devil's in the details.

[ryanbrandenburg]

@natemcmaster I worked around the skip problems here, but I'm not a big fan or the fix since it's pretty hacky. Thoughts or suggestions?

[ryanbrandenburg]

@bricelam is looking into the binding issues.

[ryanbrandenburg]

@tmat did the heavy lifting here and said:

Xunit has an option called preEnumerateTheories:
“Gets a flag that determines whether theories are pre-enumerated. If enabled, then the discovery system will return a test case for each row of test data; they are disabled, then the discovery system will return a single > test case for the theory.”
The console runner forces this option to false (setting it to true in xunit.runner.json does not have any effect), while vstest runner sets it to true.
As written, the tests that validate that ConditionalTheoryDiscoverer works implicitly assume this option is true – that is that “each theory produces a test case for each row of test data”.

as well as pasting the snippet you commented on.

I wasn't able to come up with a better solution while keeping the current behavior of skipping the appropriate TheoryData, but then again I don't have much understanding of XUnit discovery mechanics. I can get spun up on it if this is a no-go, but I figured I'd ask what you think before spending that time.

[tmat]

BTW, you can continue using dotnet vstest to run tests if you find the xunit console test runner insufficient.

If you go this route I'd suggest copying /eng/common/build.ps1 to /eng/build.ps1, updating it to execute tests using dotnet vstest when -test is passed and not pass /t:Test to MSBuild function. The same for build.sh. Roslyn also uses a different runner that optimizes test scheduling when running tests on desktop CLR: https://github.com/dotnet/roslyn/blob/master/eng/build.ps1#L403-L405

[ryanbrandenburg]

The success is a lie. There's a problem in Arcade wherein when you have a failing test with warnaserrors=false the test failures just show as warnings. dotnet/arcade#1589

[ryanbrandenburg]

🆙📅 @natemcmaster we're worked through the two blockers, If I could get your re-review I'll do one last pass of checking the nupkg metadata and we should be good to go.

[ryanbrandenburg]

Talked to Nate, things that still remain:

1. Publishing packages.
2. Producing Source packages.
3. Verifying that all the packages we used to produce are still produced.
4. Update arcade version.

[ryanbrandenburg]

🆙📅 Source packages are now produced and we're publishing artifacts. I'm going to test our Manifest publishing from the official build if @natemcmaster doesn't have an objection.

[natemcmaster]

Almost there! I'm picking over this with a fine-tooth comb now because I think we're close. Lots of nits, some stuff we can do later. My primary concerns about this change are that it will break our package deployments to NuGet, our ability to build patches and final release candidates, and that we have broken some a few other tools, like BaselineGenerator.

[ryanbrandenburg]

This is blocked by the failures of JSInterop. I'm going to need either @javiercn or @SteveSandersonMS's help to fix these tests before we can merge.