Harden build-failure-analysis Azure DevOps fetch (raise artifact cap, fail closed, timeouts)

[YuliiaKovalova]

Robustness fixes to the shared Azure DevOps fetch step, all found while validating the equivalent workflow on dotnet/roslyn#85046 end to end against real failing builds. The same fixes are going to microsoft/testfx (microsoft/testfx#10835) and dotnet/arcade, which carry sibling copies of this workflow.

1. Raise the per-artifact download cap: 500 MB -> 2 GB

MAX_ZIP_BYTES is a download guard, not a size expectation. An artifact that exceeds it is silently dropped from the analysis rather than reported, so a too-tight value hides the exact failure the workflow exists to explain.

On roslyn this fired for real: a 636 MB analyzer-logs artifact tripped the 500 MB cap and the run completed "successfully" having analysed nothing (run 33161086176). After raising the cap the same build produced a correct root cause and an inline fix suggestion (run 33161255171).

This repo has more headroom than the others -- the largest *_Logs_Attempt* artifact on definition 101 currently measures ~153 MB -- so this is preventative here rather than urgent. The cumulative budgets (MAX_TOTAL_ZIP_BYTES 3 GB compressed, MAX_TOTAL_BYTES 4 GB uncompressed) and MAX_UNZIP_BYTES are unchanged and are what actually bound the runner's disk and network, so this raises no real resource ceiling.

2. Fail closed on Azure DevOps fetch / non-JSON responses

A failed curl or an HTML error page produced an empty .value/.records, which is indistinguishable from a healthy build -- the step reported "no failed jobs" and the agent stayed inert. A transient ADO outage therefore looked green.

All metadata fetches now go through an ado_get helper that checks the curl exit status and validates the body parses as JSON, reporting a data-resolution failure instead. It returns a status code rather than calling emit_none directly, because emit_none inside a command substitution would only exit the subshell. The timeline fetch already had equivalent fail-closed handling and is left as is.

3. Bound Azure DevOps request time, and make the download deadline hard

Retries without timeouts meant a stalled endpoint hung the step until the whole job timed out. Two rounds of review sharpened this considerably:

• Metadata requests use --connect-timeout 10 --max-time 20 --retry-max-time 40. These are small JSON documents from an endpoint that normally answers in well under a second, and there are three of them before any download begins, so their combined retry windows must not eat into the job on their own.
• --max-time is per attempt, so --retry 3 alone would have permitted four full transfers plus backoff — roughly 20 minutes against timeout-minutes: 15. The download loop now has a wall-clock deadline (DOWNLOAD_BUDGET=420, MAX_ATTEMPT_SECONDS=120) and every transfer derives both --max-time and --retry-max-time from the time actually left.
• --retry-max-time only gates whether a new retry may start, so the whole invocation is additionally wrapped in timeout "${TIME_LEFT}". That is what makes the deadline real; without it a retry begun just inside the window could overshoot by a further attempt.

A transfer killed at the deadline is treated like any other failed download: the leg is reported missing and the analysis is disabled rather than run on a partial picture.

5. Enforce the download budgets, and never analyze a stale binlog

• The compressed-budget and time-budget guards now actually break out of the loop. ulimit -f takes 512-byte blocks and the block count is rounded up, so a small remaining allowance still buys at least one block instead of flooring to zero and failing every write. (This is a budget, not a byte-exact ceiling: a transfer can overshoot its cap by under 512 bytes before the size check rejects it.)
• The binlog directory is cleared immediately after it is created. The extract and upload steps glob the whole directory, so anything left by an earlier run on the same runner would otherwise be attributed to this build.

4. Validate GITHUB_OUTPUT once up front

Previously unchecked; the success path writes six outputs. The guard now proves the path is writable with a zero-byte append, not merely non-empty — a set-but-unwritable path passed the old check and then failed every append, emitting no outputs at all. Both the no-op and success paths fail loudly rather than writing into nothing.

Validation

• gh aw compile build-failure-analysis build-failure-analysis-command --strict — 2 succeeded, 0 warnings; both .lock.yml files regenerated and committed. The diff contains no uses: changes.
• shellcheck and bash -n over the fetch script extracted from every generated .lock.yml — clean.
• A checker parses the generated workflows and asserts the invariants directly, so they cannot silently rot: the worst-case download time fits the job's timeout-minutes; every retrying curl bounds its whole retry window; the download is wrapped in timeout; every budget guard actually leaves the loop; and the binlog directory is cleared. It reproduced a real regression during this work — a bad scripted substitution had replaced a break with a literal None, which both bash -n and shellcheck happily accept.
• End to end on a fork against real failing Azure DevOps builds, most recently run 33172351332: all seven jobs green, the large log artifact downloaded with no cap or budget warnings, and a correct root cause with an inline fix suggestion.

The same fixes are applied to the sibling copies of this workflow in dotnet/roslyn, dotnet/runtime, dotnet/sdk and dotnet/arcade.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR hardens the “build-failure-analysis” workflow’s Azure DevOps fetch logic to avoid silent no-op analyses and reduce hang risk when Azure DevOps is slow/unavailable. It updates the shared fetch step and then regenerates the compiled workflow lock files so the runtime workflows reflect those robustness changes.

Changes:

• Adds an ado_get helper to validate Azure DevOps metadata fetches (curl status + JSON parse) and applies timeouts to metadata requests.
• Raises the per-artifact compressed download guard (MAX_ZIP_BYTES) from 500 MB to 2 GB and adds a connect-timeout to key requests.
• Regenerates *.lock.yml workflows to incorporate the updated shared fetch logic and newer gh-aw compiler output.

Reviewed changes

Copilot reviewed 1 out of 3 changed files in this pull request and generated 1 comment.

File	Description
.github/workflows/shared/build-failure-analysis-fetch.md	Implements the hardened ADO fetch helper, timeouts, and increases per-artifact download cap; adds a GITHUB_OUTPUT check.
.github/workflows/build-failure-analysis.lock.yml	Regenerated compiled workflow reflecting the shared fetch updates (plus compiler/version refresh).
.github/workflows/build-failure-analysis-command.lock.yml	Regenerated compiled slash-command workflow reflecting the shared fetch updates (plus compiler/version refresh).

[YuliiaKovalova]

Follow-up in b0c2a7e796: two further defects found while reviewing the sibling
backports apply here too, so they're fixed in this PR as well.

1. The GITHUB_OUTPUT guard only checked the variable was non-empty; a
   set-but-unwritable path passed it and then failed on every append, leaving
   the step with no outputs instead of the intended controlled no-op. It now
   probes the path with a zero-byte append.
2. The cumulative compressed budget rejected only a non-positive remaining
   allowance, so a tiny positive remainder still started a transfer that could
   never yield a usable archive. A new MIN_ZIP_BYTES (1 MB) floor treats such
   a remainder as exhausted.

Validated in the reference implementation by a new test-fetch-guards.sh
(13/13) and by fork E2E run
33167910190,
which is green end to end and downloaded the full artifact set with no cap or
budget warnings.

Note

This comment was generated with GitHub Copilot.