Add live Azure DevOps test result publishing#8297
Draft
Evangelink wants to merge 1 commit into
Draft
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds opt-in live Azure DevOps test-result publishing to the existing Azure DevOps reporting extension, using REST APIs plus multi-process run coordination.
Changes:
- Adds new
--publish-azdo-*CLI options and wires the live publisher intoAddAzureDevOpsProvider(). - Implements REST client, result batching, run-id coordination, and localized resource strings.
- Adds unit coverage and updates all-extensions help/info expectations.
Show a summary per file
| File | Description |
|---|---|
Directory.Packages.props |
Adds System.Text.Json package version. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Microsoft.Testing.Extensions.AzureDevOpsReport.csproj |
References System.Text.Json. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsCommandLineOptions.cs |
Adds live publishing option names. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsCommandLineProvider.cs |
Registers and validates new CLI options. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsExtensions.cs |
Wires publisher as data consumer and session lifetime handler. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsLivePublishingModels.cs |
Adds live publishing constants and DTOs. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsRunIdCoordinator.cs |
Adds multi-process run-id coordination. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsClient.cs |
Adds Azure DevOps REST client. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsPublisher.cs |
Adds live result publishing lifecycle and batching. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/IAzureDevOpsTestResultsClient.cs |
Adds internal client abstraction. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/AzureDevOpsResources.resx |
Adds localized strings for options and warnings. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.cs.xlf |
Updates Czech localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.de.xlf |
Updates German localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.es.xlf |
Updates Spanish localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.fr.xlf |
Updates French localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.it.xlf |
Updates Italian localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ja.xlf |
Updates Japanese localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ko.xlf |
Updates Korean localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.pl.xlf |
Updates Polish localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.pt-BR.xlf |
Updates Portuguese localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ru.xlf |
Updates Russian localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.tr.xlf |
Updates Turkish localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.zh-Hans.xlf |
Updates Simplified Chinese localization resources. |
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.zh-Hant.xlf |
Updates Traditional Chinese localization resources. |
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs |
Updates help/info expectations for new options. |
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsLivePublishingTests.cs |
Adds live publishing unit tests. |
Copilot's findings
- Files reviewed: 26/26 changed files
- Comments generated: 6
Comment on lines
+93
to
+94
| TryDeleteFile(runIdFilePath); | ||
| TryDeleteFile(ownerFilePath); |
Comment on lines
+182
to
+186
| private static bool ShouldRetry(Exception exception, CancellationToken userCancellationToken, CancellationToken requestCancellationToken, int attempt) | ||
| => attempt < MaxAttempts | ||
| && !userCancellationToken.IsCancellationRequested | ||
| && !requestCancellationToken.IsCancellationRequested | ||
| && exception is HttpRequestException or IOException or SocketException or TaskCanceledException; |
Comment on lines
+196
to
+199
| await _runIdCoordinator.FinalizeRunAsync( | ||
| _coordinatedRun, | ||
| cancellationToken => _client.UpdateTestRunStateAsync(_publishConfiguration, CurrentRunId.Value, finalState, cancellationToken), | ||
| testSessionContext.CancellationToken).ConfigureAwait(false); |
|
|
||
| testApplicationModuleInfo ??= new Mock<ITestApplicationModuleInfo>(); | ||
| testApplicationModuleInfo.Setup(x => x.TryGetAssemblyName()).Returns("MyTests"); | ||
| testApplicationModuleInfo.Setup(x => x.GetCurrentTestApplicationFullPath()).Returns("Q:\\src\\testfx-worktrees\\azdo-live\\artifacts\\MyTests.dll"); |
| configuration.AccessToken, | ||
| new CreateTestRunRequest(configuration.RunName, true, new BuildReference(configuration.BuildId), AzureDevOpsLivePublishingConstants.InProgressTestRunState)); | ||
|
|
||
| CreateTestRunResponse response = await SendAsync<CreateTestRunResponse>(request, cancellationToken).ConfigureAwait(false); |
| </data> | ||
| <data name="OptionDescription" xml:space="preserve"> | ||
| <value>Eanble Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.</value> | ||
| <value>Enable Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.</value> |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of the brainstorm in #5951 — adds opt-in live test-result publishing to the Azure DevOps Tests tab via the REST API. No
Microsoft.TeamFoundationServer.Clientdependency; justHttpClient+System.Text.Json.Why
Today, getting results into the AzDO Tests tab requires generating a TRX and adding a downstream
PublishTestResults@2task. Two real problems:This PR publishes results to a single AzDO run as tests finish, with no TRX dependency.
What
Two new opt-in CLI options on
Microsoft.Testing.Extensions.AzureDevOpsReport:--publish-azdo-test-resultsSYSTEM_ACCESSTOKEN/SYSTEM_COLLECTIONURI/SYSTEM_TEAMPROJECTis missing.--publish-azdo-run-name <name><assembly> (<tfm>) on <agent>(or<stage>/<job>when present, sanitized). Requires--publish-azdo-test-results.Wired into the existing
AddAzureDevOpsProvider()— no new public API.How it works
Authorization: Basic base64(":<SYSTEM_ACCESSTOKEN>")(AzDO's standard PAT-as-basic-auth pattern). Token never logged.POST {project}/_apis/test/runs?api-version=7.1at session start, batchedPOST .../results?api-version=7.1asTestNodeUpdateMessages flow in,PATCH .../runs/{id}?api-version=7.1toCompleted(orAbortedon cancellation) at session end.AzureDevOpsRunIdCoordinatorwrites aazdo-runid.<buildId>.{owner,json,participant.<pid>.json}set underTestResultsDirectory. The first process creates the run; joiners read the id and publish into the same run. The owner waits for participants to drain (bounded timeout, defaults to 30s) before finalizing.HttpRequestException,IOException,SocketException, andTaskCanceledException(3 attempts, exponential backoff, 429 honorsRetry-After). All callbacks catchException ex when ex is not OperationCanceledExceptionand log a warning — publishing failures never fail the test run.Timeoutand per-request linkedCancellationTokenSource(15s).Highlights from the expert-reviewer round
Implementation went through one full round of
expert-reviewer. Critical/major issues addressed:FinalizeRunAsyncwait (no infinite hang on recycled PIDs).azdo-runid.jsonfrom a prior build no longer poisons the next build (ExpiresAtvalidated; coordination filenames keyed byBUILD_BUILDID; owner failures clean up the owner file).Exception ex when ex is not OperationCanceledExceptionsoJsonException/IOException/SocketExceptionno longer escapeIDataConsumercallbacks.HttpClient.Timeout+ per-request linked CTS — eliminates publisher-wide deadlock on a hung endpoint._lastFlushTimeis nowlong ticks+Interlocked.Read/Exchange(no tornDateTimeOffsetreads).ReadAsByteArrayAsync(ct)/ReadAsStringAsync(ct)under#if NET.HttpRequestException or IOException or SocketException or TaskCanceledException.[JsonPropertyName]on every wire DTO (request and response).automatedTestStoragenow strips the extension so it matches legacy TRX-uploaded history keys.Environment.ProcessId/int.MaxValuefor alive/dead PIDs and unique temp dirs underPath.GetTempPath()with cleanup-on-failure.Owner re-election on owner-process crash is deferred behind a clear
TODO(AzureDevOpsRunIdCoordinator.cs) with periodic lease renewal so a clean run never trips the bounded wait.Tests
127 unit tests pass. New coverage in
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsLivePublishingTests.cs:Retry-After.HelpInfoAllExtensionsTestsexpectations updated for the new options (both--helpand--infoblocks, alphabetical order preserved).Build status (local)
.\.dotnet\dotnet.exe build src\Platform\Microsoft.Testing.Extensions.AzureDevOpsReport\Microsoft.Testing.Extensions.AzureDevOpsReport.csproj -c Debug— 0 warnings, 0 errors..\.dotnet\dotnet.exe test test\UnitTests\Microsoft.Testing.Extensions.UnitTests\Microsoft.Testing.Extensions.UnitTests.csproj— 127/127 passed..\build.cmd -pack— 0 warnings, 0 errors.Out of scope (deliberate)
Checklist
/t:UpdateXlf, not hand-edited).\build.cmdgreen (0 warnings, 0 errors)Refs #5951