Skip to content

Add live Azure DevOps test result publishing#8297

Draft
Evangelink wants to merge 1 commit into
mainfrom
dev/amauryleve/azdo-live-publishing
Draft

Add live Azure DevOps test result publishing#8297
Evangelink wants to merge 1 commit into
mainfrom
dev/amauryleve/azdo-live-publishing

Conversation

@Evangelink
Copy link
Copy Markdown
Member

Part 1 of the brainstorm in #5951 — adds opt-in live test-result publishing to the Azure DevOps Tests tab via the REST API. No Microsoft.TeamFoundationServer.Client dependency; just HttpClient + System.Text.Json.

One of three PRs derived from the #5951 brainstorm. The others:

See issue comment for the broader plan.

Why

Today, getting results into the AzDO Tests tab requires generating a TRX and adding a downstream PublishTestResults@2 task. Two real problems:

  1. Long pipelines can't see what passed/failed until the very end.
  2. Parallel jobs race on TRX uploads.

This PR publishes results to a single AzDO run as tests finish, with no TRX dependency.

What

Two new opt-in CLI options on Microsoft.Testing.Extensions.AzureDevOpsReport:

Option Type Purpose
--publish-azdo-test-results zero-arity Enable live publishing. Auto-no-op (with warning) if any of SYSTEM_ACCESSTOKEN / SYSTEM_COLLECTIONURI / SYSTEM_TEAMPROJECT is missing.
--publish-azdo-run-name <name> string Override the run name. Default: <assembly> (<tfm>) on <agent> (or <stage>/<job> when present, sanitized). Requires --publish-azdo-test-results.

Wired into the existing AddAzureDevOpsProvider() — no new public API.

How it works

  • AuthAuthorization: Basic base64(":<SYSTEM_ACCESSTOKEN>") (AzDO's standard PAT-as-basic-auth pattern). Token never logged.
  • Run lifecyclePOST {project}/_apis/test/runs?api-version=7.1 at session start, batched POST .../results?api-version=7.1 as TestNodeUpdateMessages flow in, PATCH .../runs/{id}?api-version=7.1 to Completed (or Aborted on cancellation) at session end.
  • Batching — flush at 100 results or every 5s, with a global semaphore guaranteeing one in-flight POST per run.
  • Multi-processAzureDevOpsRunIdCoordinator writes a azdo-runid.<buildId>.{owner,json,participant.<pid>.json} set under TestResultsDirectory. The first process creates the run; joiners read the id and publish into the same run. The owner waits for participants to drain (bounded timeout, defaults to 30s) before finalizing.
  • Resilience — REST calls retry on HttpRequestException, IOException, SocketException, and TaskCanceledException (3 attempts, exponential backoff, 429 honors Retry-After). All callbacks catch Exception ex when ex is not OperationCanceledException and log a warning — publishing failures never fail the test run.
  • HttpClient — single static instance with a 30s Timeout and per-request linked CancellationTokenSource (15s).

Highlights from the expert-reviewer round

Implementation went through one full round of expert-reviewer. Critical/major issues addressed:

  • C1: bounded FinalizeRunAsync wait (no infinite hang on recycled PIDs).
  • C2: stale azdo-runid.json from a prior build no longer poisons the next build (ExpiresAt validated; coordination filenames keyed by BUILD_BUILDID; owner failures clean up the owner file).
  • C3: participant marker race fixed — owner also writes its own marker; existing marker file is overwritten rather than silently dropped.
  • C4: broadened catches to Exception ex when ex is not OperationCanceledException so JsonException/IOException/SocketException no longer escape IDataConsumer callbacks.
  • C5: finite HttpClient.Timeout + per-request linked CTS — eliminates publisher-wide deadlock on a hung endpoint.
  • M1: _lastFlushTime is now long ticks + Interlocked.Read/Exchange (no torn DateTimeOffset reads).
  • M2: flush loop now drains the queue past the threshold while holding the semaphore.
  • M4: cancellation-aware ReadAsByteArrayAsync(ct) / ReadAsStringAsync(ct) under #if NET.
  • M6: retry predicate widened to HttpRequestException or IOException or SocketException or TaskCanceledException.
  • M7: explicit [JsonPropertyName] on every wire DTO (request and response).
  • M9: automatedTestStorage now strips the extension so it matches legacy TRX-uploaded history keys.
  • M10/M11: tests use Environment.ProcessId / int.MaxValue for alive/dead PIDs and unique temp dirs under Path.GetTempPath() with cleanup-on-failure.

Owner re-election on owner-process crash is deferred behind a clear TODO (AzureDevOpsRunIdCoordinator.cs) with periodic lease renewal so a clean run never trips the bounded wait.

Tests

127 unit tests pass. New coverage in test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsLivePublishingTests.cs:

  • Run creation and id persistence.
  • Result-batch flush at size and time thresholds, plus session-end forced flush.
  • State mapping (Passed / Failed / Skipped / Timeout / Cancelled / Error).
  • 429 retry honors Retry-After.
  • Network/JSON failure → warning logged, session continues normally.
  • Run-id coordinator: create, join (multi-process), expired-file rejection, BuildId mismatch rejection.
  • Owner finalize bounded wait.

HelpInfoAllExtensionsTests expectations updated for the new options (both --help and --info blocks, alphabetical order preserved).

Build status (local)

  • .\.dotnet\dotnet.exe build src\Platform\Microsoft.Testing.Extensions.AzureDevOpsReport\Microsoft.Testing.Extensions.AzureDevOpsReport.csproj -c Debug0 warnings, 0 errors.
  • .\.dotnet\dotnet.exe test test\UnitTests\Microsoft.Testing.Extensions.UnitTests\Microsoft.Testing.Extensions.UnitTests.csproj127/127 passed.
  • .\build.cmd -pack0 warnings, 0 errors.

Out of scope (deliberate)

Checklist

  • Critical & Major review findings addressed
  • Localized via resx + xlf (regenerated with /t:UpdateXlf, not hand-edited)
  • Help/info acceptance test expectations updated
  • No new public API
  • .\build.cmd green (0 warnings, 0 errors)
  • Owner re-election on crash (deferred — TODO with lease renewal in place)
  • PR feedback addressed

Refs #5951

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 16, 2026 16:46
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds opt-in live Azure DevOps test-result publishing to the existing Azure DevOps reporting extension, using REST APIs plus multi-process run coordination.

Changes:

  • Adds new --publish-azdo-* CLI options and wires the live publisher into AddAzureDevOpsProvider().
  • Implements REST client, result batching, run-id coordination, and localized resource strings.
  • Adds unit coverage and updates all-extensions help/info expectations.
Show a summary per file
File Description
Directory.Packages.props Adds System.Text.Json package version.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Microsoft.Testing.Extensions.AzureDevOpsReport.csproj References System.Text.Json.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsCommandLineOptions.cs Adds live publishing option names.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsCommandLineProvider.cs Registers and validates new CLI options.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsExtensions.cs Wires publisher as data consumer and session lifetime handler.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsLivePublishingModels.cs Adds live publishing constants and DTOs.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsRunIdCoordinator.cs Adds multi-process run-id coordination.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsClient.cs Adds Azure DevOps REST client.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsPublisher.cs Adds live result publishing lifecycle and batching.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/IAzureDevOpsTestResultsClient.cs Adds internal client abstraction.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/AzureDevOpsResources.resx Adds localized strings for options and warnings.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.cs.xlf Updates Czech localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.de.xlf Updates German localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.es.xlf Updates Spanish localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.fr.xlf Updates French localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.it.xlf Updates Italian localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ja.xlf Updates Japanese localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ko.xlf Updates Korean localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.pl.xlf Updates Polish localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.pt-BR.xlf Updates Portuguese localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.ru.xlf Updates Russian localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.tr.xlf Updates Turkish localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.zh-Hans.xlf Updates Simplified Chinese localization resources.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/AzureDevOpsResources.zh-Hant.xlf Updates Traditional Chinese localization resources.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs Updates help/info expectations for new options.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsLivePublishingTests.cs Adds live publishing unit tests.

Copilot's findings

  • Files reviewed: 26/26 changed files
  • Comments generated: 6

Comment on lines +93 to +94
TryDeleteFile(runIdFilePath);
TryDeleteFile(ownerFilePath);
Comment on lines +182 to +186
private static bool ShouldRetry(Exception exception, CancellationToken userCancellationToken, CancellationToken requestCancellationToken, int attempt)
=> attempt < MaxAttempts
&& !userCancellationToken.IsCancellationRequested
&& !requestCancellationToken.IsCancellationRequested
&& exception is HttpRequestException or IOException or SocketException or TaskCanceledException;
Comment on lines +196 to +199
await _runIdCoordinator.FinalizeRunAsync(
_coordinatedRun,
cancellationToken => _client.UpdateTestRunStateAsync(_publishConfiguration, CurrentRunId.Value, finalState, cancellationToken),
testSessionContext.CancellationToken).ConfigureAwait(false);

testApplicationModuleInfo ??= new Mock<ITestApplicationModuleInfo>();
testApplicationModuleInfo.Setup(x => x.TryGetAssemblyName()).Returns("MyTests");
testApplicationModuleInfo.Setup(x => x.GetCurrentTestApplicationFullPath()).Returns("Q:\\src\\testfx-worktrees\\azdo-live\\artifacts\\MyTests.dll");
configuration.AccessToken,
new CreateTestRunRequest(configuration.RunName, true, new BuildReference(configuration.BuildId), AzureDevOpsLivePublishingConstants.InProgressTestRunState));

CreateTestRunResponse response = await SendAsync<CreateTestRunResponse>(request, cancellationToken).ConfigureAwait(false);
</data>
<data name="OptionDescription" xml:space="preserve">
<value>Eanble Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.</value>
<value>Enable Azure DevOps report generator to write errors to the output in a way that AzureDev Ops understands.</value>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants