[Engsys] Configure cibuildwheel for compiled dev requirements in live test jobs

[danieljurek]

Description

Storage live test jobs have been failing on every scheduled run since at least 2026-07-10 — 22 of 24 jobs in build 6634674 never got as far as running a test.

Root cause

All four storage packages list ../azure-storage-extensions as a relative dev requirement. build_whl_for_req → create_package detects ext_modules and shells out to cibuildwheel, which then cannot reach the network the way it needs to:

OS	Failure
Linux (16 jobs)	pip inside the manylinux container can't reach pypi.org — PIP_INDEX_URL doesn't cross the Docker boundary. No matching distribution found for setuptools>=77.0.3
Windows (4 jobs)	nuget can't reach api.nuget.org (blackholed to 192.0.2.88) while fetching a CPython
macOS	file-datalake / file-share passed (public egress); blob / queue hit the 60-minute job timeout

The build pipeline already solves this in steps/build-package-artifacts.yml, but that template is only reachable from ci.yml. The live test chain is completely disjoint from it:

ci.yml    -> archetype-sdk-client.yml -> jobs/ci.yml         -> steps/build-package-artifacts.yml  # has workarounds
tests.yml -> archetype-sdk-tests.yml  -> jobs/live.tests.yml -> steps/build-test.yml               # had none

Every workaround there is a step-scoped ##vso[task.setvariable], so nothing propagates across chains.

Fix

Add the equivalent steps to jobs/live.tests.yml, gated on the service directory actually containing a package configured for cibuildwheel. steps/build-test.yml is deliberately not touched — it's shared with ci.tests.yml and the blast radius would be every package in the repo.

Unlike the build pipeline, these wheels are only pip installed into the job's own venv, so the matrix is cut to the single wheel that interpreter can use:

Platform	Before	After
Linux x86_64	8 wheels	1 — cp310-manylinux_x86_64
Windows AMD64	4 wheels	1 — cp310-win_amd64
macOS arm64	3 wheels	1 — cp310-macosx_arm64

Everything dropped is untestable in that job anyway: wrong CPU arch, wrong libc, or a PyPy interpreter the job never runs. The surviving wheel is cp310-abi3, forward-compatible with the 3.11–3.14 jobs via the limited API. This also means no QEMU (aarch64) and no pythonarm64 (win_arm64), both of which the build pipeline has to provision.

Two details worth calling out:

• CIBW_ENVIRONMENT_PASS_LINUX, not CIBW_ENVIRONMENT — the latter would clobber the package's own [tool.cibuildwheel].environment (CFLAGS).
• CIBW_SKIP='pp* *musllinux*' rather than CIBW_ENABLE='' — enable uses InheritRule.APPEND, so an env value appends to the pyproject enable = ["pypy"] instead of replacing it, and PyPy still builds.

azure-storage-extensions is currently the only package in the repo with [tool.cibuildwheel], so no other service is affected.

Validation — build 6636078

Step	Result
Check for compiled dev requirements	24/24 succeeded (Found compiled package: azure-storage-extensions)
Configure cibuildwheel	24/24 succeeded
Authenticate to NuGet	4 succeeded (Windows), 20 correctly skipped
Pre-provision CPython	4 succeeded (Windows), 20 correctly skipped

Run Tests went from 2 passing to 13 passing, and 0 jobs failed at dev-requirement install (was 22 of 24).

Exactly one wheel per platform. Linux built in 38 seconds:

21:37:44  Building wheel for package azure-storage-extensions
21:38:22  Found whl azure_storage_extensions-0.2.0-cp310-abi3-manylinux_2_5_x86_64....whl

Pre-existing failures this uncovers

These are not regressions — the pipeline died before reaching them for weeks, so a real backlog accumulated. Filing separately:

• blob — 40 failed / 2193 passed, identical on Linux and macOS (deterministic, not flake): 18 test_arrow*, 10 test_blob_encryption*, 8 access-tier. Ruled out the new wheel as a cause: azure-storage-extensions ships only checksums, and blob's _encryption.py never imports it.
• file-share — 1–2 failed / 799 passed: test_abort_copy_file → NoPendingCopyOperation, a copy-completes-before-abort race. Flaky.
• queue — Test Samples fails on all 4 OS with exit 99, which is dispatch_checks.py:470 wrapping an exception in the dispatcher, not a sample failure (duration 0.00 confirms). Dies while streaming network_activity_logging.py, whose output contains raw non-UTF8 bytes from an encrypted queue message. summarize() never prints the exception text, so the cause is swallowed.
• datalake (Windows) — 3 failed / 472 passed.

Unrelated observation

live.tests.yml declares a BeforeTestSteps parameter (line 20) that it never forwards to build-test.yml, so live-test callers setting it have them silently dropped. ci.tests.yml:154 forwards correctly. Not addressed here.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes. — N/A, pipeline-only change, no shipping package affected
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message.

Testing Guidelines

• Pull request includes test coverage for the included changes. — validated end-to-end in build 6636078 (table above)

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Configures live-test jobs to build compiled development requirements reliably with cibuildwheel across supported platforms.

Changes:

• Detects service packages using cibuildwheel.
• Limits builds to the native CPython wheel and forwards Linux feed authentication.
• Pre-provisions CPython from Azure Artifacts on Windows.