[CI] Azure DevOps TF10216 outage fails SAS, test-library, and publish tasks

[dalexsoto]

Failure signature

Azure DevOps-backed tasks fail before tests run with:

TF10216: Azure DevOps services are currently unavailable. Try again later.

The same service outage affected unrelated tasks and PRs within minutes.

Confirmed occurrences

PR / head	Build and job	Failing task	Outcome
#26180 / 897e85a4b8e8e206c57ea59a7b0c3088abf18e15	14718322 - monotouch_ios, attempt 3	Generate BosStorageMirror SAS tokens	All task retries returned TF10216; the job failed. The stage attempt remains active.
#26207 / 84ada67506d107a5b8979d2bccb4aa1201b46d3b	14719170 - introspection, attempt 2	Download test libraries	TF10216 prevented the download; the job failed. The stage attempt remains active.
#26213 / 6acb80dc4985ae3a81ea8b6487ba2a3ec13148a5	14719548 - Publish results, attempt 2	Download secrets / OIDC token	Repeated TF10216 responses failed the publish-results job. The build also had a separate checkout failure after the PR merged.

First observed: 2026-07-21 23:59 UTC. Latest observed: 2026-07-22 00:04 UTC. Recurrence: 3 jobs across 3 distinct monitored PRs.

Classification and impact

This is an Azure DevOps service/infrastructure outage, not a source or test regression. It blocks credential acquisition, artifact/test-library access, and result publication before the affected work can run.

Investigation

• Inspected each Azure build timeline and failing task.
• Correlated the exact TF10216 response across separate agents, tasks, builds, and PRs.
• Searched open and closed dotnet/macios issues using TF10216, the full service-unavailable message, and the affected task names; no matching issue was found.

Workaround

Retry failed jobs after Azure DevOps service health recovers, subject to mergeability and retry-budget gates. Do not submit another retry while an equivalent stage attempt is queued or running. Azure logs have limited retention.

[dalexsoto]

PR #26180 retry outcome: build 14718322 simulator_tests attempt 3 completed failed at 00:08 UTC after the monotouch_ios job hit TF10216. With the build terminal and no equivalent attempt active, separately counted attempt 4 was accepted at 00:24 UTC. Reconciled budget: 5 lifetime retries / 3 on the current head.

[dalexsoto]

PR #26207 retry outcome: build 14719170 simulator_tests attempt 2 completed failed because introspection hit TF10216; linker_macos succeeded. Attempt 3 was accepted at 00:50 UTC to rerun only introspection. Budget: 2 lifetime/current-head retries.

[dalexsoto]

Final retry outcome: merged PR #26207 build 14719170 simulator_tests attempt 3 completed successfully at 02:04 UTC. Introspection passed, the overall build succeeded, and the PR merged at 02:08 UTC.

[dalexsoto]

Final retry outcome: merged PR #26180 build 14718322 simulator_tests attempt 4 completed successfully. The replacement attempt ran after the TF10216 failure, the overall build succeeded, and the PR merged at 02:37 UTC.