ci: use Entra authentication for ACR cleanup - #2584
Merged
Merged
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hey @ranadeepsingh 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
5 tasks
ranadeepsingh
changed the title
This was referenced Aug 1, 2026
ranadeepsingh
force-pushed
the
ci/release-branch-compat
branch
from
August 1, 2026 10:50
edea596 to
996a747
Compare
ranadeepsingh
force-pushed
the
ci/fix-acr-cleanup-auth
branch
from
August 1, 2026 10:50
c54e2ff to
28ab3c3
Compare
Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: #2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ranadeepsingh
force-pushed
the
ci/release-branch-compat
branch
from
August 1, 2026 12:10
996a747 to
8cc85fb
Compare
ranadeepsingh
force-pushed
the
ci/fix-acr-cleanup-auth
branch
from
August 1, 2026 12:10
28ab3c3 to
20200b6
Compare
## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: #2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ranadeepsingh
added a commit
that referenced
this pull request
Aug 4, 2026
* Fix and streamline release branch compatibility checks ## Summary Run release compatibility checks for both GitHub target-branch formats and replace redundant compile, setup, credential, and per-package SBT tasks with one cached, project-scoped validation process. ## Prompting Intent The engineer asked to fix the silently skipped ReleaseBranchCompat job and simplify it before enabling it so the check is both reliable and efficient. ## Linked Sources - Base CI hardening PR: #2581 - Evidence build with skipped phase: ADO build 229176406 - Parent stack layer: ci/databricks-impact-gating ## Rationale The target condition accepts both values observed across Azure Repos and GitHub PR providers. A single SBT process retains full test compilation and the intended core, VW, and OpenCV compatibility suites while removing repeated build loading, root-wide IntelliJ setup, unnecessary Key Vault access, and Azure CLI authentication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use Entra authentication for ACR cleanup (#2584) * Use Entra authentication for ACR cleanup Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: #2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant CI authentication and Conda work (#2585) ## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: #2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ranadeepsingh
added a commit
that referenced
this pull request
Aug 4, 2026
* Gate Databricks CPU and GPU tests independently ## Summary Classify changed paths against the actual Databricks CPU and GPU runtime surfaces, emit separate fail-open decisions, and gate each matrix leg independently. ## Prompting Intent The engineer asked to determine exactly when Databricks tests should run, lock down the path rules, and deliver the work as a stacked pull request above PR #2581. ## Linked Sources - Base CI hardening PR: #2581 - GitHub stacked PR documentation: https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-stacked-pull-requests - ADO timing audit: build 229176406 ## Rationale CPU and GPU decisions are separated because most module changes cannot affect the expensive GPU notebooks. Unknown paths and shared build or test infrastructure remain fail-open, while explicit test-only and unrelated tooling paths skip safely. This preserves coverage while avoiding unrelated GPU capacity waits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: fix and streamline release branch compatibility (#2583) * Fix and streamline release branch compatibility checks ## Summary Run release compatibility checks for both GitHub target-branch formats and replace redundant compile, setup, credential, and per-package SBT tasks with one cached, project-scoped validation process. ## Prompting Intent The engineer asked to fix the silently skipped ReleaseBranchCompat job and simplify it before enabling it so the check is both reliable and efficient. ## Linked Sources - Base CI hardening PR: #2581 - Evidence build with skipped phase: ADO build 229176406 - Parent stack layer: ci/databricks-impact-gating ## Rationale The target condition accepts both values observed across Azure Repos and GitHub PR providers. A single SBT process retains full test compilation and the intended core, VW, and OpenCV compatibility suites while removing repeated build loading, root-wide IntelliJ setup, unnecessary Key Vault access, and Azure CLI authentication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use Entra authentication for ACR cleanup (#2584) * Use Entra authentication for ACR cleanup Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: #2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant CI authentication and Conda work (#2585) ## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: #2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ranadeepsingh
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 4, 2026
## Summary Merge the reviewed microsoft#2582 squash commit back into microsoft#2581 after a concurrent master refresh displaced the child merge, restoring the cumulative microsoft#2582- microsoft#2585 pipeline changes on top of current master. ## Prompting Intent The engineer asked to diagnose the failing tests on microsoft#2581 while completing, reviewing, and fully validating the stacked CI pull requests without losing work already merged through the stack. ## Linked Sources - Parent PR: microsoft#2581 - Databricks gating PR: microsoft#2582 - Release compatibility PR: microsoft#2583 - ACR authentication PR: microsoft#2584 - CI setup cleanup PR: microsoft#2585 - Failed Fabric build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229191951 ## Rationale The old red build was caused by a transient DNS failure resolving the Fabric service endpoint, not by the CI changes. During the investigation, a branch refresh raced with the stacked merges and moved microsoft#2581 past microsoft#2582's squash commit. Merging the official squash commit preserves the exact reviewed child tree and its history instead of reconstructing or duplicating those changes, while retaining the latest master updates already present on the parent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues/PRs
What changes are proposed in this pull request?
trigger: noneandpr: none.synapseml-clean-acrservice connection.az storage blob exists --auth-mode login.The previous four weekly runs failed with
KeyBasedAuthenticationNotPermittedbecause the storage account disallows key-based access.Operational prerequisite
The
synapseml-clean-acrservice connection must be provisioned/authorized in ADO before this schedule can run. Its identity needs blob read/exists access tommlspark/acrbackup, ACR repository list/delete, and permission to create runs formmlsparkacrexport3.How is this patch tested?
Does this PR change any dependencies?
Does this PR add a new feature?