Skip to content

Auto-upload test artifacts and build tags to Azure DevOps#8295

Draft
Evangelink wants to merge 1 commit into
mainfrom
dev/amauryleve/azdo-artifact-upload
Draft

Auto-upload test artifacts and build tags to Azure DevOps#8295
Evangelink wants to merge 1 commit into
mainfrom
dev/amauryleve/azdo-artifact-upload

Conversation

@Evangelink
Copy link
Copy Markdown
Member

Part 3 of the brainstorm in #5951 — adds opt-in pipeline-level artifact upload + build tags to Microsoft.Testing.Extensions.AzureDevOpsReport. Pure ##vso logging commands — no Azure DevOps authentication required.

This is one of three PRs derived from the #5951 brainstorm. The other two:

  • Live test-result publishing
  • Flaky-history annotations + quarantine awareness

See comment for the broader plan.

Why

Today HangDump, CrashDump, CodeCoverage, and TRX all drop files into TestResultsDirectory, but nothing teaches Azure DevOps to publish them as build artifacts — users have to add a separate PublishBuildArtifacts@1 task. Similarly, build queries can't easily filter "runs that produced a hang/crash dump" without grepping logs.

What

Four new opt-in CLI options on Microsoft.Testing.Extensions.AzureDevOpsReport:

Option Type Purpose
--report-azdo-upload-artifacts <mode> off (default) | tags-only | files | all Enable build tags, file upload, or both.
--report-azdo-upload-artifact-name <name> string Override the artifact container name. Default: TestResults_<assemblyName>_<tfm>.
--report-azdo-upload-artifact-include <glob> repeatable File globs (relative to TestResultsDirectory) to include. Default: **/*.
--report-azdo-upload-artifact-exclude <glob> repeatable Exclusion globs. Default: none.

At session end (when TF_BUILD=true and the mode is not off), the new AzureDevOpsArtifactUploader:

  • Emits ##vso[build.addbuildtag]has-crashdump and/or has-hangdump based on the producer UID of consumed TestNodeFileArtifact events (CrashDumpProcessLifetimeHandler / HangDumpProcessLifetimeHandler — robust to localization).
  • Emits ##vso[build.addbuildtag]has-test-failures if the session contained any failed test.
  • Enumerates TestResultsDirectory via IFileSystem, applies include/exclude globs (case-sensitive on Linux, case-insensitive on Windows), and emits ##vso[artifact.upload containerfolder=…;artifactname=…]<path> for each match — properly escaping ;, \r, \n via the existing AzDoEscaper.

All output goes through IOutputDevice (no direct Console.WriteLine). All env/path/file access goes through IEnvironment / IConfiguration / IFileSystem. Exception isolation: callbacks never propagate non-cancellation exceptions out of the lifetime handler.

Highlights from the expert-reviewer round

The implementation went through one round of expert-reviewer. Critical/major issues addressed:

  • Failure isolation: all callbacks now try/catch everything except OperationCanceledException; warnings logged via ILogger.
  • Validator: rejects --report-azdo-upload-artifact-{include,exclude,name} when --report-azdo-upload-artifacts is absent or off.
  • Race-free flags: monotonic crash/hang/failure flags use Interlocked.Exchange + Volatile.Read.
  • Cross-platform paths: new PathComparison helper (OrdinalIgnoreCase on Windows, Ordinal elsewhere) applied to glob construction, prefix matching, and sort.
  • Dump detection: producer-UID-based only — no longer matches on localized DisplayName or path substring.
  • Out-of-dir filter: files resolved outside TestResultsDirectory (e.g. via symlinks) are skipped.
  • Lazy TFM resolution via new shared TargetFrameworkMonikerHelper (also adopted by the existing AzureDevOpsReporter).
  • Pre-existing typo EanbleEnable fixed in Resources/AzureDevOpsResources.resx.

Tests

  • 12 new unit tests in test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsArtifactUploaderTests.cs covering mode wiring (incl. off no-op), tag emission (exactly once, even under concurrent message events), glob include/exclude, escaping, missing TF_BUILD, null/whitespace TestResultsDirectory, out-of-dir filtering, and validator rejection.
  • HelpInfoAllExtensionsTests expectations updated for new options (both --help and --info blocks, alphabetical order preserved).

Build status (local)

  • .\.dotnet\dotnet.exe build src\Platform\Microsoft.Testing.Extensions.AzureDevOpsReport\Microsoft.Testing.Extensions.AzureDevOpsReport.csproj -c Debug0 warnings, 0 errors.
  • .\.dotnet\dotnet.exe test test\UnitTests\Microsoft.Testing.Extensions.UnitTests\Microsoft.Testing.Extensions.UnitTests.csproj566/566 passed.
  • .\build.cmd -pack0 warnings, 0 errors.

Out of scope (deliberate)

  • Result-level attachments (attach a dump to its failing test in the AzDO Tests tab) — depends on the REST client introduced by the live-publishing PR.
  • Code-coverage attachment under the Coverage tab — same reason.

Both can land as small follow-ups once that PR is merged.

Checklist

  • Critical & Major review findings addressed
  • Localized via resx + xlf (regenerated with /t:UpdateXlf, not hand-edited)
  • Help/info acceptance test expectations updated
  • No new public API
  • .\build.cmd green (0 warnings, 0 errors)
  • PR feedback addressed

Refs #5951

Add opt-in Azure DevOps artifact upload support, help/info updates, and unit coverage for the new reporter behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 16, 2026 16:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in pipeline-level artifact upload + build-tag feature to Microsoft.Testing.Extensions.AzureDevOpsReport. It introduces four new CLI options that, when running under Azure DevOps (TF_BUILD=true), emit ##vso[artifact.upload …] and ##vso[build.addbuildtag]… logging commands at session end based on consumed FileArtifact/TestNodeUpdateMessage events and a glob filter over the test results directory.

Changes:

  • New AzureDevOpsArtifactUploader (data consumer + session lifetime handler) plus supporting PathComparison / TargetFrameworkMonikerHelper helpers and AzureDevOpsArtifactUploadMode enum.
  • New CLI options (--report-azdo-upload-artifacts, --report-azdo-upload-artifact-{name,include,exclude}) with validator that rejects absolute globs, unknown modes, and artifact options when uploads are off; resx/xlf strings added; pre-existing Eanble→Enable typo fixed.
  • Help/info acceptance expectations updated and 12 new unit tests covering modes, tag emission, escaping, globbing, missing TF_BUILD, and validator rejection.
Show a summary per file
File Description
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsArtifactUploader.cs New uploader implementing the artifact-upload + build-tag logic.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsCommandLineOptions.cs New option/mode constants.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsCommandLineProvider.cs Registers and validates the new options.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsExtensions.cs Registers the uploader as data consumer + session lifetime handler.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsReporter.cs Switched TFM resolution to shared helper.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/PathComparison.cs OS-aware path string comparer/comparison.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/TargetFrameworkMonikerHelper.cs Shared TFM resolution helper.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Microsoft.Testing.Extensions.AzureDevOpsReport.csproj Adds Microsoft.Extensions.FileSystemGlobbing reference.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/AzureDevOpsResources.resx New strings + Eanble→Enable typo fix.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/xlf/*.xlf Regenerated translations for all locales.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsArtifactUploaderTests.cs 12 new unit tests for uploader + validator.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs Adds AzDO extension + new option entries to help/info expectations.

Copilot's findings

  • Files reviewed: 24/24 changed files
  • Comments generated: 1

public sealed class AzureDevOpsArtifactUploaderTests
{
private const string ArtifactUploadOptionsRequireUploadArtifactsMessage = "'--report-azdo-upload-artifact-include', '--report-azdo-upload-artifact-exclude', and '--report-azdo-upload-artifact-name' require '--report-azdo-upload-artifacts' to be set to a value other than 'off'.";
private const string ResultsDirectory = @"Q:\results";
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants