test: clean up Fabric E2E artifacts - #2615
Merged
Rana Singh (ranadeepsingh) merged 3 commits intoAug 9, 2026
Merged
Conversation
## Summary Run the existing stale-artifact cleanup before Fabric E2E tests and delete each Lakehouse and Spark Job Definition created by a suite after it finishes. Restrict stale cleanup to SynapseML test naming patterns and add regression coverage for cleanup ordering, failure preservation, and pipeline wiring. ## Prompting Intent The engineer asked to make the remaining SynapseML pull requests merge-ready, diagnose failing checks to root cause, keep fixes lean and performant, and avoid changing the original intent of unrelated PRs. Repeated PR microsoft#2604 runs reproduced a shared Fabric workspace artifact-quota failure, so this change isolates the CI infrastructure repair from that PR. ## Linked Sources - Original Fabric E2E infrastructure PR: microsoft#2495 - Blocked pull request: microsoft#2604 - Reproduced Azure Pipelines failure: https://msdata.visualstudio.com/A365/_build/results?buildId=230268903 ## Rationale A dedicated cleanup change avoids broadening PR microsoft#2604. Running the already-defined stale cleanup first recovers leaked capacity, while tracking and deleting only artifacts created by each suite prevents recurrence. Reverse-order, best-effort deletion removes job definitions before their backing store, tolerates already-deleted resources, and still surfaces every real cleanup failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
|
Hey Rana Singh (@ranadeepsingh) 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens Fabric E2E test hygiene to prevent shared-workspace quota exhaustion by ensuring stale and newly-created Fabric artifacts are cleaned up deterministically, and wires the cleanup into CI so the Fabric E2E job starts from a known-good state.
Changes:
- Run
FabricTestCleanupahead of Fabric smoke + notebook suites in the FabricE2E pipeline job. - Introduce an artifact tracker to delete per-suite created artifacts in reverse creation order from
afterAll, while continuing after deletions fail and tolerating “already deleted” artifacts. - Add unit tests for tracker ordering / error aggregation and a pipeline-wiring regression test.
Show a summary per file
| File | Description |
|---|---|
pipeline.yaml |
Executes FabricTestCleanup before Fabric E2E suites to reduce artifact leakage/quota pressure. |
tools/ci/tests/test_pipeline_yaml.py |
Adds a regression test ensuring cleanup runs before Fabric tests in the pipeline job. |
core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricTestArtifactTracker.scala |
New tracker utility to record artifact ids and delete them in reverse creation order with aggregated error handling. |
core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricTestArtifactTrackerSuite.scala |
Unit tests for deletion ordering, “not found” tolerance, and failure aggregation behavior. |
core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricNotebookTests.scala |
Uses the tracker for all created artifacts, restricts stale cleanup to SynapseML-generated names, and cleans up tracked artifacts in afterAll. |
Review details
Tip
Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 5/5 changed files
- Comments generated: 3
- Review effort level: Lite
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2615 +/- ##
==========================================
- Coverage 86.33% 86.20% -0.13%
==========================================
Files 335 335
Lines 18007 18007
Branches 1658 1658
==========================================
- Hits 15546 15523 -23
- Misses 2461 2484 +23 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
## Summary Address review feedback by logging artifact-specific cleanup failures, computing the stale cutoff once per cleanup pass, and waiting for notebook workers to terminate before deleting their artifacts. Add regression coverage for executor shutdown ordering. ## Prompting Intent The engineer asked to resolve all active pull-request comments while keeping CI fixes lean, performant, and free of cleanup races. The three review threads on PR microsoft#2615 identified diagnostics, consistency, and concurrency issues in the initial Fabric artifact lifecycle repair. ## Linked Sources - Cleanup failure diagnostics review: microsoft#2615 (comment) - Executor shutdown review: microsoft#2615 (comment) - Stale cutoff review: microsoft#2615 (comment) - Pull request: microsoft#2615 ## Rationale The original exceptions remain unwrapped so callers retain their exact failure types and suppressed errors, while logs now identify the affected artifact. A single cutoff avoids boundary drift. Graceful executor shutdown followed by forced interruption ensures deletion cannot race active notebook work; a hard failure is preferable to deleting resources still in use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
## Summary Restrict stale Spark Job Definition cleanup to the exact Fabric E2E notebook allowlist, cover forced executor interruption, guarantee test executor cleanup, and execute cleanup plus notebook suites in one sbt process. ## Prompting Intent Extensively review PR microsoft#2615 and make it merge-ready while keeping destructive cleanup narrowly scoped, preventing in-flight notebook work from racing artifact deletion, preserving test reliability, and reducing CI startup overhead. ## Linked Sources - Pull request: microsoft#2615 - Executor cleanup review: microsoft#2615 (comment) - Cleanup diagnostics review: microsoft#2615 (comment) - Cutoff consistency review: microsoft#2615 (comment) ## Rationale An exact notebook allowlist avoids deleting unrelated workspace artifacts that merely share the ExploreAlgorithms prefix. A timeout overload makes the forced-shutdown path deterministic to test without slowing production cleanup. Keeping both Fabric test commands in one sbt process preserves fail-fast ordering while avoiding a second JVM startup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
This was referenced Aug 9, 2026
Rana Singh (ranadeepsingh)
added a commit
that referenced
this pull request
Aug 10, 2026
* ci: add release branch compatibility check to PR validation (#2550) Add a ReleaseBranchCompat job that runs on every PR to master. It rebases each release branch (starting with spark4.0) onto the PR HEAD and runs sbt compile test:compile to catch breakage before it lands in master. - Non-blocking (continueOnError: true) - Matrix-based for easy expansion to more release branches - Reports merge conflicts and compile failures as warnings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: bump netty to 4.1.118 and drop duplicate pyspark in mmlspark/release demo image (#2557) Addresses MSRC case 110886 / incident 31000000570827. The mmlspark/release image (built from tools/docker/demo/Dockerfile) ships Spark 3.5.4, which pins netty 4.1.96.Final. That version is flagged for multiple CVEs (CVE-2023-44487, CVE-2024-29025, CVE-2025-24970, ...). Spark has not bumped netty in any 3.5.x release. netty 4.1.x is binary-compatible, so we replace all netty-*-4.1.96.Final*.jar files in /opt/spark/jars/ with 4.1.118.Final right after the Spark extract. This includes netty-codec-http2 (the specific artifact named by the finder). Also removes 'pyspark' from the conda install line. It was pulling a complete second Spark install (PySpark 4.0.1) into /usr/local/lib/python*/site-packages/pyspark/ that nothing in the demo image actually used (SPARK_HOME points at /opt/spark) and that doubled the surface area scanners report on. Validated locally: - /opt/spark/jars/netty-*-4.1.96.Final*.jar: 0 matches after build - /opt/spark/jars/netty-*-4.1.118.Final*.jar: full set present - /usr/local/lib/.../pyspark: no longer exists - spark-submit --version: works - spark.range(5).count(): returns 5 Jetty (shaded inside hadoop-client-runtime-3.3.4.jar at 9.4.43) is OUT OF SCOPE for this PR; that requires a Spark/Hadoop swap and will be tracked separately. * chore: add SynapseML local setup skill (#2558) * chore: add SynapseML local setup skill ## Summary Add a project-scoped SynapseML agent skill that diagnoses local toolchain state, selects JDK 11 for SBT commands, runs a safe local Spark smoke test, and flags live-service tests before agents run them. ## Prompting Intent The engineer asked the agent to create a skill that helps any future agent get SynapseML working locally after the PR 2556 review exposed a local Java 21 and Scala 2.12 compiler-bridge failure. The engineer also asked to create a PR for the skill addition before continuing the original external PR review. ## Linked Sources - User request in current session: create a skill that will help any agent be able to get SynapseML working locally. - Follow-up user request in current session: create a PR for that skill addition and continue using it to review PR 2556. - Existing project-scoped skill convention: .agents/skills/code-review/SKILL.md. - Local validation output: doctor_status=ok, JDK 11 dry-run selected JAVA_HOME, smoke test passed, Azure Search tests flagged review_required. ## Rationale A project-scoped SynapseML skill keeps local setup guidance with the repository where future agents need it. The scripts use explicit parameters rather than session state, force JDK 11 for Scala 2.12 SBT commands, and include a live-service guard so agents do not accidentally create or delete Azure Search resources while validating changes. * chore: move SynapseML setup skill to Copilot path ## Summary Move the SynapseML local setup skill from `.agents/skills/` to `.github/skills/` so it uses the documented Copilot project-skill discovery path. ## Prompting Intent The engineer asked whether the `.agents` folder was correct and whether Copilot would pick it up. Investigation found that the local skill-authoring reference documents `.github/skills/<name>/` and `.claude/skills/<name>/` as project skill locations, so the open skill PR needed a path correction. ## Linked Sources - User question in current session: is this .agent folder correct? will copilot pick this up? - Skill-authoring reference: /home/brwals/.copilot/installed-plugins/copilot-toolkit-marketplace/common/skills/create-skill/references/REFERENCE.md - Existing PR: https://github.com/microsoft/SynapseML/pull/2558 ## Rationale The existing `.agents/skills/code-review` directory was only evidence of a repo-local convention, not evidence of Copilot discovery. Moving the new skill to `.github/skills/synapseml-local-setup/` keeps the same skill content while placing it in the documented project-skill path. * feat: Add v1 OpenAI endpoint support and remove legacy completions API (#2560) * Add v1 OpenAI Endpoint support and remove legacy completions API * Fix FuzzingUnitTest * Add test to increase code coverage * Make v1 api assumption cleaner * Add OpenAICompletion deprecation * Remove deprecation warnings * Fix RAI test for OpenAIPrompt * Revert "Add OpenAICompletion deprecation" This reverts commit fa708e25cbacebd959bd2ca3ef9573a9d0bb8670. * Revert "Fix RAI test for OpenAIPrompt" This reverts commit 3ed60449fe0ef594e742b24c701040e2b3624e8d. * Revert "Remove deprecation warnings" This reverts commit 9a40c5c21ff9ffbdb0ed04cd3eb10703a5569165. * Reapply "Remove deprecation warnings" This reverts commit 987484ceee2c2dd89b1b04ef104b0c716695ef6a. * Reapply "Fix RAI test for OpenAIPrompt" This reverts commit f06f1ade547f4dbfec418159af5795aa626d57ab. * Reapply "Add OpenAICompletion deprecation" This reverts commit 10715cd4c20ef80b0b318e27710a3027f5bd5236. * chore: migrate SynapseML skills to Copilot path (#2559) ## Summary Move the remaining SynapseML repo skill from `.agents/skills/` to `.github/skills/` so Copilot CLI can discover all repo-versioned skills from the documented project-skill path. Add README pointers under `.agents/` for tools or agents that inspect the older convention. ## Prompting Intent The engineer asked to migrate everything to the correct Copilot CLI path and suggested keeping a generic agents pointer. The goal was to make existing skills discoverable by Copilot while avoiding future confusion about `.agents/skills`. ## Linked Sources - User request in current session: migrate everything to the correct path for Copilot CLI and keep a generic agents pointer. - Skill location reference: /home/brwals/.copilot/installed-plugins/copilot-toolkit-marketplace/common/skills/create-skill/references/REFERENCE.md - Prior merged skill PR: https://github.com/microsoft/SynapseML/pull/2558 ## Rationale `.github/skills/<name>/` is the documented Copilot CLI project-skill location. Keeping only README pointers under `.agents/` preserves a breadcrumb for other agent conventions without leaving duplicate or stale `SKILL.md` files in a path Copilot CLI may not load. * fix: add speechtotextsdk improvements (#2562) * add speechtotextsdk improvements * Fix ffmpeg output args * add ffmpeg url check * fix: address speech recording review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: make OpenAIPrompt RAI test resilient Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "test: make OpenAIPrompt RAI test resilient" This reverts commit fccce86149476005b0506e7562e81eb3ef1f620b. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: remove Acrolinx integration config (#2570) * chore: remove Acrolinx integration config AB#5391146 AB#5391147 ## Summary Remove the retired Acrolinx repository configuration from SynapseML and add the Feature Registry pointer and repo-specific design notes for Feature 5391136. ## Prompting Intent Engineer asked the agent to complete the Acrolinx removal request from the Microsoft Learn authoring tools PM. The repository cleanup needed to remove stale source-controlled Acrolinx state while preserving Feature Registry traceability for the administrative webhook removal and the June 30 contract-expiration risk. ## Linked Sources - ADO Feature: https://msdata.visualstudio.com/A365/_workitems/edit/5391136 - Design Spec task: https://msdata.visualstudio.com/A365/_workitems/edit/5391146 - Deployment task: https://msdata.visualstudio.com/A365/_workitems/edit/5391147 - Feature Registry specs: https://msdata.visualstudio.com/A365/_git/FeatureRegistry?path=/Features/active/5391136 - Teams request: https://teams.microsoft.com/l/message/19:81ff723c-eac9-4b2a-ba9f-844542135555_cc1adbf9-6510-43d6-a849-adba51e66d59@unq.gbl.spaces/1782314980087?context=%7B%22contextType%22%3A%22chat%22%7D - Acrolinx config before cleanup: https://github.com/microsoft/SynapseML/blob/b0fa222cfdde5d0a2cbb2bc6a35630bbb61bc0e3/.acrolinx-config.edn ## Rationale Deleting `.acrolinx-config.edn` is the least invasive source change because the Acrolinx contract is ending and the repo-level webhook was already removed through GitHub administration. Keeping the Feature Registry folder in the repo gives future maintainers a durable pointer to the reason for the cleanup without adding runtime or build behavior. * chore: keep Feature Registry metadata out of SynapseML AB#5391146 AB#5391147 ## Summary Remove the Feature Registry scaffold files from the SynapseML cleanup branch so the public repository PR only deletes the retired Acrolinx config. ## Prompting Intent Engineer clarified that Feature Registry metadata must not be included in the external SynapseML repository. The agent adjusted the existing cleanup PR to keep registry tracking in FeatureRegistry only while preserving the Acrolinx source cleanup. ## Linked Sources - ADO Feature: https://msdata.visualstudio.com/A365/_workitems/edit/5391136 - SynapseML PR: https://github.com/microsoft/SynapseML/pull/2570 - FeatureRegistry PR: https://msdata.visualstudio.com/A365/_git/FeatureRegistry/pullrequest/2169703 - User correction: do not include Feature Registry metadata in the external repo ## Rationale Keeping the public SynapseML PR scoped to `.acrolinx-config.edn` avoids adding internal Feature Registry process artifacts to an external repository. Feature-level tracking remains in the FeatureRegistry PR and ADO work items. * fix: route AnalyzeText document errors to errorCol (#2569) * fix: route AnalyzeText document errors to errorCol Move Azure AI Language document-level errors returned inside HTTP 200 AnalyzeText responses from the response payload into the configured error column after auto-batch flattening. Preserve transport error precedence and add a no-network regression test for mixed document success/error responses. AB#4638662 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: pin PR validation sbt launcher Use the sbt launcher version from project/build.properties instead of installing the latest apt sbt package. This keeps the JDK 11 PR validation job on the repository's sbt 1.10.11 launcher and avoids sbt 2.x rejecting JDK 11 before scalastyle can run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use pinned sbt wrapper in PR validation Invoke the downloaded sbt launcher explicitly so the GitHub runner does not resolve its preinstalled sbt 2.x binary under JDK 11. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: prefer pinned sbt on PATH Keep PR validation commands as plain sbt while placing the repository-version launcher first on PATH for subsequent workflow steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: avoid ordering assumption in AnalyzeText error test Partition collected rows by error nullability instead of relying on collect order, addressing PR review feedback about Spark DataFrames being unordered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore SynapseML Azure pipeline (#2573) fix: restore Azure pipeline * ci: pin MLflow for protobuf compatibility AB#5480878 (#2580) Pin the shared Python test environment to MLflow 2.21.3, matching the Databricks test dependency. This constrains protobuf to a compatible major version and invalidates the stale conda cache that breaks Python test collection. * fix: modernize OpenAI and LangChain support for GPT-5.1 (#2572) test: migrate OpenAI tests and examples to GPT-5.1 * ci: migrate Databricks GPU pool to T4 AB#5478524 (#2579) ci: migrate Databricks GPU pool to T4 * fix: correct LightGBM improvement tolerance semantics (#2578) fix: correct LightGBM improvement tolerance semantics * chore(deps): bump amannn/action-semantic-pull-request (#2554) Bumps [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request) from 5.4.0 to 6.1.1. - [Release notes](https://github.com/amannn/action-semantic-pull-request/releases) - [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md) - [Commits](https://github.com/amannn/action-semantic-pull-request/compare/v5.4.0...v6.1.1) --- updated-dependencies: - dependency-name: amannn/action-semantic-pull-request dependency-version: 6.1.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com> * ci: use pre-installed Azure CLI in ADO jobs(#2545) * chore(deps): bump ossf/scorecard-action from 2.3.1 to 2.4.4 (#2597) Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.3.1 to 2.4.4. - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](https://github.com/ossf/scorecard-action/compare/0864cf19026789058feabb7e87baa5f140aac736...2d1146689b8cda280b9bc96326124645441f03bc) --- updated-dependencies: - dependency-name: ossf/scorecard-action dependency-version: 2.4.4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump actions/setup-java from 5.6.0 to 5.7.0 (#2600) Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5.6.0 to 5.7.0. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](https://github.com/actions/setup-java/compare/03ad4de0992f5dab5e18fcb136590ce7c4a0ac95...b6effb05e454b25005698d916606bdc6ffcbf961) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: 5.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com> * docs: add T4 GPU local RAG quickstart (#2588) * docs: add T4 GPU local RAG quickstart ## Summary Add an end-to-end local RAG notebook that performs sentence embedding, exact retrieval, and Phi-4-mini generation on a Databricks T4 worker. Register the notebook in the active GPU smoke suite and documentation sidebar with pinned model dependencies. ## Prompting Intent Reassess the unmerged GPU demo from PR #2271 against current master. Add a maintainable integration example only if it fills a gap beyond the standalone GPU KNN, Hugging Face CausalLM/Phi, and PDF Q&A notebooks; use current T4 assumptions, avoid TensorRT-LLM and custom CUDA setup, provide deterministic smoke assertions, and make no unrelated pipeline changes. ## Linked Sources - Original proposal: https://github.com/microsoft/SynapseML/pull/2271 - GPU KNN component: https://github.com/microsoft/SynapseML/pull/2157 - Local embedding component: https://github.com/microsoft/SynapseML/pull/2236 - Hugging Face CausalLM/Phi component: https://github.com/microsoft/SynapseML/pull/2301 - Current Databricks T4 validation platform: https://github.com/microsoft/SynapseML/pull/2579 - PDF Q&A reference: https://github.com/microsoft/SynapseML/blob/master/docs/Explore%20Algorithms/AI%20Services/Quickstart%20-%20Document%20Question%20and%20Answering%20with%20PDFs.ipynb ## Rationale The existing notebooks document the individual building blocks but not their local, service-free composition. Exact PyTorch cosine scoring keeps the tutorial small and fully testable on the active T4 suite without reviving the disabled RAPIDS pipeline or its obsolete CUDA/TensorRT initialization. The notebook uses supported current-master models, max_new_tokens rather than conflicting sequence limits, and a PR smoke mode that exercises every GPU stage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: harden GPU RAG reproducibility checks ## Summary Pin both Hugging Face repositories to immutable commit snapshots, load the Phi model and tokenizer from the same local snapshot with remote code disabled, and strengthen retrieval validation against input-order fallback. ## Prompting Intent Address independent review findings on PR #2588 by removing mutable model resolution and trust_remote_code, then make the smoke test prove that GPU similarity ranking—not corpus order—selects the answer document. ## Linked Sources - Follow-up pull request: https://github.com/microsoft/SynapseML/pull/2588 - Original proposal: https://github.com/microsoft/SynapseML/pull/2271 - Pinned embedding snapshot: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/tree/1110a243fdf4706b3f48f1d95db1a4f5529b4d41 - Pinned Phi snapshot: https://huggingface.co/microsoft/Phi-4-mini-instruct/tree/cfbefacb99257ffa30c83adab238a50856ac3083 ## Rationale SentenceTransformer accepts an immutable revision for its complete model/tokenizer snapshot. HuggingFaceCausalLM loads its tokenizer separately, so Phi is first resolved to one pinned worker-local snapshot and both loaders receive that path. Transformers 4.49 natively supports the checkpoint's phi3 architecture, allowing remote model code to remain disabled. A persisted corpus ordinal and independent Python sort over all GPU scores prove the top-k result differs from the first input rows and has strict score ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: format GPU RAG notebook cells ## Summary Apply the repository-pinned Black 22.3 Jupyter formatter to the updated GPU RAG notebook cells. ## Prompting Intent Resolve the Python Style CI failure on PR #2588 without changing notebook behavior or broadening the patch. ## Linked Sources - Pull request: https://github.com/microsoft/SynapseML/pull/2588 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229241355 ## Rationale Black's Jupyter formatter omits the terminal newline stored in each code cell. Formatting only the touched notebook aligns its JSON representation with the CI environment while preserving all model-pinning and retrieval assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: pin GPU RAG hub client and FP16 ## Summary Pin huggingface-hub 0.26.0 in the Databricks GPU libraries and notebook setup, verify the Hugging Face dependency set in unit tests, and force Phi model loading to FP16 on T4 hardware. ## Prompting Intent Address the second independent re-review of PR #2588 by making snapshot_download's client version reproducible and preventing Phi's BF16 checkpoint metadata from selecting an unsupported native dtype on T4 GPUs. ## Linked Sources - Pull request: https://github.com/microsoft/SynapseML/pull/2588 - Repository environment pin: environment.yml - Hugging Face Hub 0.26.0: https://pypi.org/project/huggingface-hub/0.26.0/ - Pinned Phi configuration: https://huggingface.co/microsoft/Phi-4-mini-instruct/blob/cfbefacb99257ffa30c83adab238a50856ac3083/config.json ## Rationale Version 0.26.0 is already the repository-pinned lower bound used with Transformers 4.49.0, so installing that exact version on the GPU cluster makes snapshot resolution deterministic without introducing a new dependency choice. Phi advertises bfloat16 in its configuration, while NVIDIA T4 compute capability 7.5 lacks native BF16; passing the supported float16 dtype explicitly avoids architecture-dependent auto selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: guard accelerate GPU dependency pin ## Summary Assert that the Databricks GPU library manifest retains accelerate==0.26.0 alongside the pinned Hugging Face dependencies. ## Prompting Intent Address the remaining actionable review feedback on PR #2588 by preventing the runtime dependency used for distributed Phi loading from drifting without a focused unit-test failure. ## Linked Sources - Pull request: https://github.com/microsoft/SynapseML/pull/2588 - Reviewed GPU library manifest: core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala ## Rationale The package is already explicitly pinned in GPULibraries, so extending the existing parsed-manifest test is the smallest regression guard and avoids duplicating library configuration or changing runtime behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(deps): bump github/codeql-action/autobuild from 4.37.3 to 4.37.5 (#2601) * chore(deps): bump github/codeql-action/autobuild from 4.37.3 to 4.37.5 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ranadeepsingh <16433904+ranadeepsingh@users.noreply.github.com> Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com> * chore(deps): bump postcss from 8.5.19 to 8.5.25 in /website (#2603) Bumps [postcss](https://github.com/postcss/postcss) from 8.5.19 to 8.5.25. - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/postcss/postcss/compare/8.5.19...8.5.25) --- updated-dependencies: - dependency-name: postcss dependency-version: 8.5.25 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com> * chore: migrate artifact links off retiring Azure CDN (#2589) * chore: migrate artifact links off retiring Azure CDN ## Summary Replace all 400 current-master references to mmlspark.azureedge.net with the repository-owned mmlspark Blob Storage origin across runtime package configuration, release output, examples, documentation, notebooks, and every published documentation version. ## Prompting Intent Recreate the intent of the stale CDN-removal PR on current master only after verifying the supported artifact destination and Azure CDN retirement path. Audit each endpoint use by semantics, preserve package and content paths, validate live artifacts and package resolution, and avoid changing or closing the original PR. ## Linked Sources - Original proposal: https://github.com/microsoft/SynapseML/pull/2326 - Azure CDN retirement FAQ: https://learn.microsoft.com/en-us/azure/cdn/classic-cdn-retirement-faq - Azure CDN migration guidance: https://learn.microsoft.com/en-us/azure/cdn/migrate-tier - Azure Front Door/CDN comparison: https://learn.microsoft.com/en-us/azure/frontdoor/front-door-cdn-comparison ## Rationale SynapseML's release pipeline publishes artifacts directly to the mmlspark storage account, the repository already uses that public Blob Storage origin extensively, and byte-for-byte URL checks confirmed the CDN currently proxies the same content. Using the verified origin removes the retiring CDN hostname without inventing an unverified Front Door name, while preserving Maven, documentation, R-package, model, dataset, and icon path semantics. Historical links that already return 404 retain the same status and are not broadened into unrelated artifact-repair work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: make R setup independent of retired CDN ## Summary Repair current and versioned R setup guidance so each release installs its six published, version-matched component archives and resolves SynapseML JVM artifacts through Blob Storage. Document the compatibility bypass required by already-published wrappers, correct the Databricks setup and LightGBM example, remove invalid HTML-page Maven repositories from the Docker demo, and add generator/docs regressions. ## Prompting Intent Investigate the review finding that published R archives still register the retired Azure CDN resolver. Make repository-controlled R installation work with that hostname unavailable, avoid claiming that externally published archives were rewritten, validate local and Databricks-oriented resolution paths, and state the exact external publishing prerequisite for a full artifact migration. ## Linked Sources - Original migration PR: https://github.com/microsoft/SynapseML/pull/2326 - Current migration PR: https://github.com/microsoft/SynapseML/pull/2589 - Maven repository review: https://github.com/microsoft/SynapseML/pull/2589#discussion_r3695558709 - Azure CDN retirement FAQ: https://learn.microsoft.com/en-us/azure/cdn/classic-cdn-retirement-faq - Azure Front Door migration guidance: https://learn.microsoft.com/en-us/azure/cdn/migrate-tier - Apache Spark package repository configuration: https://spark.apache.org/docs/3.5.0/configuration.html#runtime-environment ## Rationale Existing release archives cannot be repaired by a source-only change because their generated sparklyr metadata is already published. Version-matched component downloads plus an explicit Blob resolver and `extensions = character()` provide a tested repository-controlled path without racing or misrepresenting external publication. Future generated archives inherit the corrected resolver from PackageUtils; fully repairing historical metadata still requires an authorized regeneration and publish to the `mmlspark/rrr` container (or a replacement release). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: correct Spark 3.4 compatibility guidance ## Summary Correct the Spark Packages and Python installation snippets so both identify SynapseML 1.0.15 as the compatible release for Spark 3.4 while retaining SynapseML 1.1.3 for Spark 3.5. ## Prompting Intent Address the remaining actionable review feedback on PR #2589 in the existing branch, verify the surrounding compatibility guidance stays consistent, run targeted website validation and code review, and rerun the full PR checks. ## Linked Sources - Pull request and review feedback: https://github.com/microsoft/SynapseML/pull/2589 - Original migration context: https://github.com/microsoft/SynapseML/pull/2326 ## Rationale The Databricks, Fabric, and SBT guidance already distinguishes SynapseML 1.1.3 for Spark 3.5 from 1.0.15 for Spark 3.4. Updating only the two stale explanatory references restores consistency without changing the Spark 3.5 commands that the snippets demonstrate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add backward-compatible AAD auth to Azure Search (#2591) feat: add backward-compatible AAD auth for Azure Search * fix: preserve Spark partition topology when counting rows (#2593) ## Summary Count rows on the original DataFrame RDD so adaptive execution cannot coalesce a projected counting query into a different partition topology. Add a regression that exposes the old 20-to-fewer-partitions drift and verifies exact per-partition counts. ## Prompting Intent Recreate the valid intent behind ancient PR #2282 from current master only after reproducing issue #2278. Isolate distributed startup, feature-width bounds, and native pointer lifetime separately; use TDD and submit only a proven root cause with real regression coverage. ## Linked Sources - Reported failure: https://github.com/microsoft/SynapseML/issues/2278 - Superseded ancient proposal: https://github.com/microsoft/SynapseML/pull/2282 ## Rationale The literal-only projection was cheaper, but AQE could optimize it to fewer partitions than the training DataFrame. LightGBM then indexed that shortened count array with real task partition IDs, causing the primary ArrayIndexOutOfBoundsException and secondary connection failures. Counting the exact DataFrame RDD trades projection pruning for topology correctness. Feature-width validation and innerPredict cleanup were deliberately excluded because neither was demonstrated as the cause of #2278 or backed by a stable leak regression. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(deps): bump github/codeql-action/upload-sarif (#2606) Bumps [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) from 4.37.3 to 4.37.4. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81...f205ea1c3313d32999d8d6a48b4f6530d4437b38) --- updated-dependencies: - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github/codeql-action/upload-sarif (#2607) Bumps [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) from 4.37.4 to 4.37.5. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...d1ba80a13dd99fba24a470575428917156a28b43) --- updated-dependencies: - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: remove retired Azure AI Anomaly Detector components (#2605) The Azure AI Anomaly Detector service has been retired by Microsoft. Every `anomalydetector` REST endpoint now answers HTTP 410 (Gone), verified across paths, API versions and regions. * ci: land sbt bootstrap and stacked CI fixes (#2581) * fix: correct LightGBM improvement tolerance semantics ## Summary Require lower-is-better validation metrics to improve by more than improvementTolerance before resetting the early-stopping counter. Clarify the parameter documentation and add focused regression coverage for both metric directions and zero tolerance. ## Prompting Intent Investigate GitHub issue #2565 from a new branch based on master, determine whether the report is valid, and implement a complete fix suitable for an upstream SynapseML pull request. ## Linked Sources - GitHub issue: https://github.com/microsoft/SynapseML/issues/2565 ## Rationale The existing higher-is-better comparison already treats improvementTolerance as a minimum delta, while lower-is-better metrics accepted small regressions. A package-internal comparison helper makes the intended symmetric behavior directly testable without adding a slow native LightGBM fixture or changing public APIs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: harden LightGBM early stopping parameters ## Summary Expand improvement-tolerance coverage across representative LightGBM metrics and tolerance values. Preserve disabled early stopping when earlyStoppingRound is zero, validate both early-stopping parameters, and document their accepted ranges. ## Prompting Intent The engineer requested broader parameter testing to ensure the issue #2565 fix does not introduce downstream regressions. Cover related defaults, boundaries, metric families, invalid values, and early-stopping-round interactions before updating the pull request. ## Linked Sources - GitHub issue: https://github.com/microsoft/SynapseML/issues/2565 - Pull request: https://github.com/microsoft/SynapseML/pull/2578 - LightGBM 3.3.5 parameters: https://lightgbm.readthedocs.io/en/v3.3.5/Parameters.html#early-stopping-round ## Rationale Correct tolerance semantics classify more rounds as non-improving, so the wrapper must explicitly preserve LightGBM's zero-means-disabled behavior. Shared Spark parameter validators reject values that LightGBM does not support, while deterministic matrix tests cover the decision logic without depending on platform-specific native binaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: prevent sbt bootstrap Maven rate limits SynapseML's Azure Pipelines fans out ~30 hosted-agent matrix jobs that each cold-bootstrap the sbt launcher (org.scala-sbt:sbt:1.10.11, pinned in project/build.properties) and resolve Ivy dependencies from public Maven Central. When many fresh agents -- and several overlapping PR builds -- do this simultaneously, Maven Central returns HTTP 429 (rate limit) and "Setup repo" fails before any test runs (e.g. ADO build 229124511, UnitTests flaky). The pre-existing jittered retries only widened the window against a sustained throttle; they did not remove the thundering herd. Durable fix (cache-first, stagger as supplement): * templates/sbt_cache.yml (primary): Azure Cache@2 for the sbt launcher boot dir (~/.sbt/boot -- the artifact that 429s) and the Ivy cache (~/.ivy2/cache). In steady state, jobs restore these from Azure's cache service and never touch Maven Central. Keys derive from the bootstrap inputs (project/build.properties, project/plugins.sbt, build.sbt) so they invalidate exactly when those change; restoreKeys give a safe partial fallback and continueOnError keeps a cache miss/corruption non-fatal. * BuildAndCacheSbt prewarm job: warms those caches once per run, mirroring the existing BuildAndCacheCondaEnv job. * tools/ci/sbt_retry.sh: single tested helper replacing the duplicated inline retry blocks. Smooths only the cold-cache path with a bounded random start stagger (desynchronises concurrent cold bootstraps) plus bounded jittered exponential-backoff retries. Fails visibly on exhaustion -- no success fallback masking. Wired the shared cache template into every sbt-running job (Style, Publish, Databricks/Fabric E2E, BuildDocker, PythonTests, RTests, WebsiteSamplesTests, UnitTests, ReleaseBranchCompat) by reviving the dormant ivy_cache placeholders, and routed all `sbt setup` bootstraps through the helper. Tests (python -m pytest tools/ci/tests/): deterministically exercise the retry/backoff/stagger + visible-failure behaviour with a fake sbt, and assert pipeline.yaml parses, the cache keys invalidate on bootstrap inputs, and every sbt job is wired to the cache template + prewarm job. No LightGBM, Isolation Forest, GPU, or application changes. TLS verification, job coverage, and all tests are preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: serialize sbt cache prewarm before fan-out ## Summary Make the sbt bootstrap prewarm a mandatory gate before Azure Pipeline matrix jobs start. Add Coursier caching, require exact hits on the boot, Ivy, and Coursier caches before disabling the cold-cache stagger, wire the conditional release job, and strengthen pipeline tests around the dependency graph and cache lifecycle. ## Prompting Intent The engineer asked to fix Maven Central HTTP 429 setup failures in a new stacked PR. The solution must prevent fresh hosted agents from cold-bootstrapping sbt concurrently, allow at least the existing job fan-out after bootstrap is safe, retain bounded retry behavior for cache-service failures, and keep bootstrap failures visible rather than masking them. ## Linked Sources - Failing Azure job: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229124511&view=logs&jobId=e97036a1-bcdb-5cd5-905e-b0cf2c8f33cf - Parent PR investigation: https://github.com/microsoft/SynapseML/pull/2578#issuecomment-5147830518 - Stacked PR: https://github.com/microsoft/SynapseML/pull/2581 - Prewarm concurrency review: https://github.com/microsoft/SynapseML/pull/2581#discussion_r3693956516 ## Rationale A best-effort prewarm running beside the matrix does not protect the first run for a new dependency key, so every sbt-running job now waits for one successful warm job. Cache-service errors remain non-fatal and fall back to staggered retries, but a failed warm blocks fan-out to avoid recreating the thundering herd. Coursier is cached alongside sbt boot and Ivy because modern resolution uses all three stores, and the stagger is suppressed only when every cache is an exact hit so dependency-only changes remain desynchronized. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: skip Databricks E2E for non-impacting PRs ## Summary Add conservative pull-request impact detection for the six-leg Databricks E2E matrix. Clearly non-impacting documentation, website, GitHub metadata, CI helper, and isolated test-source changes skip Databricks, while all uncertain or runtime-affecting changes continue to run it. ## Prompting Intent The engineer asked to extend PR #2581 so expensive Databricks Azure Pipeline jobs are skipped when the pull request cannot affect notebook execution. The gate must preserve scheduled and branch coverage, avoid brittle CPU-shard mapping, and default to running whenever impact detection is incomplete or uncertain. ## Linked Sources - Stacked CI PR: https://github.com/microsoft/SynapseML/pull/2581 - Full green baseline build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229164855 - Azure multi-job output variables: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops#job-output-variables-used-in-other-job-conditions ## Rationale Use one fail-open decision for the complete Databricks matrix because the five CPU partitions mix notebooks across modules and are not stable ownership boundaries. The detector skips only a narrow allowlist of clearly inert paths; runtime code, notebooks, build and pipeline files, Databricks test utilities, shared TestBase infrastructure, unknown paths, empty diffs, and fetch or classifier failures all keep E2E enabled. Non-PR builds always run to preserve scheduled and release coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: gate Databricks CPU and GPU tests independently (#2582) * Gate Databricks CPU and GPU tests independently ## Summary Classify changed paths against the actual Databricks CPU and GPU runtime surfaces, emit separate fail-open decisions, and gate each matrix leg independently. ## Prompting Intent The engineer asked to determine exactly when Databricks tests should run, lock down the path rules, and deliver the work as a stacked pull request above PR #2581. ## Linked Sources - Base CI hardening PR: https://github.com/microsoft/SynapseML/pull/2581 - GitHub stacked PR documentation: https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-stacked-pull-requests - ADO timing audit: build 229176406 ## Rationale CPU and GPU decisions are separated because most module changes cannot affect the expensive GPU notebooks. Unknown paths and shared build or test infrastructure remain fail-open, while explicit test-only and unrelated tooling paths skip safely. This preserves coverage while avoiding unrelated GPU capacity waits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: fix and streamline release branch compatibility (#2583) * Fix and streamline release branch compatibility checks ## Summary Run release compatibility checks for both GitHub target-branch formats and replace redundant compile, setup, credential, and per-package SBT tasks with one cached, project-scoped validation process. ## Prompting Intent The engineer asked to fix the silently skipped ReleaseBranchCompat job and simplify it before enabling it so the check is both reliable and efficient. ## Linked Sources - Base CI hardening PR: https://github.com/microsoft/SynapseML/pull/2581 - Evidence build with skipped phase: ADO build 229176406 - Parent stack layer: ci/databricks-impact-gating ## Rationale The target condition accepts both values observed across Azure Repos and GitHub PR providers. A single SBT process retains full test compilation and the intended core, VW, and OpenCV compatibility suites while removing repeated build loading, root-wide IntelliJ setup, unnecessary Key Vault access, and Azure CLI authentication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use Entra authentication for ACR cleanup (#2584) * Use Entra authentication for ACR cleanup Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: https://github.com/microsoft/SynapseML/pull/2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant CI authentication and Conda work (#2585) ## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: https://github.com/microsoft/SynapseML/pull/2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: extend Docker validation timeout ## Summary Raise the BuildDocker job timeout from 60 to 120 minutes and add a pipeline regression test that preserves enough time for both sequential image builds. ## Prompting Intent The engineer asked to diagnose and fix the remaining failure on #2581 and to continue full validation until the parent PR is ready, without hiding genuine test failures. ## Linked Sources - Parent PR: https://github.com/microsoft/SynapseML/pull/2581 - Failed PR build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229579403 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229578121 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229580525 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229576499 ## Rationale The Dockerfiles and image behavior were unchanged, but recent hosted-agent builds required roughly 51 minutes when successful and exceeded the default one-hour job cap in multiple master and PR runs. A 120-minute job budget keeps both image validations mandatory while tolerating current registry and package download latency. This is safer and more targeted than skipping an image or doubling agent usage by splitting the builds into parallel jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: address PR review and protect package publishing ## Summary Resolve the blocking PR #2581 review findings by making ACR archival digest-safe, correcting PipelineRun names and sbt cache invalidation, warming cold agents before direct sbt calls, and validating the canonical package version before publishing. ## Prompting Intent The engineer asked to rebase PR #2581 onto current master, audit the new review feedback, fix valid actions, ensure the pipelines continue to publish package versions safely, review the complete change, and rerun Azure validation. ## Linked Sources - Integration PR and review threads: https://github.com/microsoft/SynapseML/pull/2581 - Stacked CI changes: https://github.com/microsoft/SynapseML/pull/2582 - Stacked CI changes: https://github.com/microsoft/SynapseML/pull/2583 - Stacked CI changes: https://github.com/microsoft/SynapseML/pull/2584 - Stacked CI changes: https://github.com/microsoft/SynapseML/pull/2585 - ACR transfer guidance: https://learn.microsoft.com/azure/container-registry/container-registry-transfer-images - ACR image deletion behavior: https://learn.microsoft.com/azure/container-registry/container-registry-delete ## Rationale Immutable manifest digests prevent mutable tags such as latest from reusing the wrong backup or deleting an unarchived manifest. Per-agent warming is limited to unavailable or inexact cache restores so exact hits remain fast, while the prewarm job still verifies dependency resolution. Package versions are resolved from the SBT source of truth and release publication fails before side effects when the v-tag disagrees. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Support lossless string identifiers in SAR (#2594) * feat: Add lossless string identifier support to SAR Refs #2275 Refs #2283 ## Summary Add deterministic, reversible user and item identifier mappings to SAR so string and wide numeric IDs are never cast into lossy caller-visible values. Persist mappings with the model, preserve identifier types in scores and recommendations, define null and unknown-ID behavior, restore typed item recommendation APIs, and add Scala and Python regression coverage. ## Prompting Intent Recreate the intent of the stale SAR string-ID change on current master without copying its lossy casts. Keep the SparkML API coherent and backward compatible for numeric users, use TDD, validate serialization and schema behavior, expose Python wrappers, and exercise targeted compile, style, code generation, Scala, and Python/JVM checks before opening a replacement PR. ## Linked Sources - Feature request: https://github.com/microsoft/SynapseML/issues/2275 - Original pull request: https://github.com/microsoft/SynapseML/pull/2283 - Current SAR implementation at the starting revision: https://github.com/microsoft/SynapseML/tree/7d9fabcc/core/src/main/scala/com/microsoft/azure/synapse/ml/recommendation - Repository review policy: .github/skills/code-review/SKILL.md ## Rationale Use model-owned typed mappings instead of composing RecommendationIndexer because that stage stringifies numeric identifiers, exposes index columns, and cannot recover every original type. Contiguous deterministic indices keep the existing matrix implementation viable, while persisted DataFrame parameters make decoding reversible after save/load. Inner mapping joins intentionally drop null or unseen scoring IDs, strict type validation prevents ambiguous conversions, and legacy numeric models fall back to identity mappings. The approach accepts a deterministic global sort and persisted mapping storage in exchange for lossless, reproducible SparkML behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Address SAR identifier compatibility review Refs #2275 Refs #2594 ## Summary Resolve the four independent review findings on SAR string identifier support. Preserve typed IDs in ranking train/validation splits, accept only round-trip-safe numeric scoring casts, retain established integer recommendation schemas for safely representable numeric IDs, and rank only factor IDs that have real mappings. Add focused Scala and Python regressions and remove unnecessary mapping cache and interaction-count work identified during review. ## Prompting Intent The engineer asked to fix all medium correctness and compatibility findings on PR #2594, add a regression for each, rerun targeted Scala, code generation, formatting, and Python/JVM validation, then update the existing PR and request re-review without weakening lossless string or wide numeric behavior. ## Linked Sources - Pull request and review context: https://github.com/microsoft/SynapseML/pull/2594 - Feature request: https://github.com/microsoft/SynapseML/issues/2275 - Original pull request: https://github.com/microsoft/SynapseML/pull/2283 - Repository review policy: .github/skills/code-review/SKILL.md ## Rationale Use Spark structs and array functions instead of Double UDF payloads so split schemas remain typed. Numeric scoring IDs are temporarily cast only when casting back reproduces the input, preventing overflow and fractional aliasing while retaining unknown-ID drop semantics. Recommendation decoding conditionally uses the historical integer schema only when every ID round-trips through Int; strings and wide or fractional numeric IDs remain lossless. Candidate indices are intersected with both factors and mappings before top-K so gaps cannot consume recommendation slots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Make SAR numeric identifier handling ANSI-safe Refs #2275 Refs #2594 ## Summary Use ANSI-safe try_cast expressions for numeric identifier compatibility and legacy mappings. Persist whether model-owned user and item mappings safely round-trip through IntegerType, reuse those flags when selecting recommendation output schemas, and limit destination-index collection to mapping-less legacy models. Add ANSI overflow, persisted-flag, legacy-default, and recommendation-planning regressions. ## Prompting Intent The engineer asked to resolve the second independent review of PR #2594: prevent CAST_OVERFLOW under spark.sql.ansi.enabled=true, eliminate repeated mapped-model recommendation scans and index collection, add focused regressions, rerun Scala/codegen/Python validation, update the existing PR, trigger Azure Pipelines, and request another re-review. ## Linked Sources - Pull request and review context: https://github.com/microsoft/SynapseML/pull/2594 - Feature request: https://github.com/microsoft/SynapseML/issues/2275 - Original pull request: https://github.com/microsoft/SynapseML/pull/2283 - Repository review policy: .github/skills/code-review/SKILL.md ## Rationale Use Spark SQL try_cast in both cast directions rather than pre-cast comparisons so out-of-range values become null and are filtered even with ANSI mode enabled. Compute compatibility once while fitting and persist it with conservative false defaults for legacy models, avoiding full mapping scans on every recommendation call. New model mappings are contiguous, so mapped models rank the score vector directly; only mapping-less legacy models collect actual candidate indices to preserve gapped-ID correctness. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: restore Spark release compatibility checks (#2608) Configure a deterministic repository-local Git committer identity before replaying PR commits onto the Spark 3.5 and Spark 4.1 release branches. Distinguish genuine merge conflicts from rebase infrastructure failures and preserve successful rebase diagnostics. * ci: make Spark 4.1 compatibility validation reliable (#2611) * ci: preserve sbt retry helper during release replay ## Summary Stage the sbt retry helper outside the repository before switching to Spark release branches, and parameterize the shared cache template so it can invoke that stable path after rebase. ## Prompting Intent Investigate why Spark 3.5 and Spark 4.1 compatibility checks still failed after PR #2608, reproduce the failure with PR #2595 changes, implement the complete hotfix, and validate the real release replay path. ## Linked Sources - Failing PR: https://github.com/microsoft/SynapseML/pull/2595 - Prior identity hotfix: https://github.com/microsoft/SynapseML/pull/2608 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229969678 ## Rationale The rebased working tree intentionally comes from the Spark release branch, so master-only CI helpers cannot remain repository-relative. Copying the helper to Agent.TempDirectory preserves release-specific dependency resolution and avoids moving cache warming ahead of the rebase, where exact cache hits could hide missing release dependencies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: replay only release-relevant PR changes ## Summary Replace commit-history rebasing with a three-way application of the synthetic PR merge tree's release-relevant patch onto each Spark release branch. ## Prompting Intent Validate the compatibility hotfix with PR #2595's real source changes while ensuring CI-only commits do not conflict with old Spark branches that predate the current pipeline and helper files. ## Linked Sources - Validation PR source: https://github.com/microsoft/SynapseML/pull/2595 - Prior identity hotfix: https://github.com/microsoft/SynapseML/pull/2608 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229969678 ## Rationale The compatibility job needs the effective PR content on the release tree, not unrelated CI and documentation commits. Building the patch from the synthetic merge commit preserves GitHub's merge result, handles source branches behind master, retains three-way conflict detection, and avoids requiring commit identity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: focus release compatibility on Spark 4.1 compilation ## Summary Remove the redundant Spark 3.5 release matrix leg and replace broad Spark 4.1 runtime suites with full test compilation of the effective PR patch. ## Prompting Intent Explain why the release compatibility jobs exist and keep fixing the failures exposed by validation PR #2610, accounting for master already targeting Spark 3.5. ## Linked Sources - Original compatibility PR: https://github.com/microsoft/SynapseML/pull/2550 - Streamlining PR: https://github.com/microsoft/SynapseML/pull/2583 - Integration validation PR: https://github.com/microsoft/SynapseML/pull/2610 - Azure validation build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229984834 ## Rationale Normal PR validation already compiles and tests master on Spark 3.5, so replaying onto the older spark3.5 maintenance snapshot duplicates coverage and introduces unrelated JVM drift. Spark 4.1 test compilation catches cross-version source and test API breakage, while the existing master test fan-out supplies runtime coverage without rerunning broad, memory-heavy suites on a constrained compatibility agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: pin GitHub Actions to full-length commit SHAs (#2602) * fix: skip VW barrier execution for single-partition training (#2592) ## Summary Use barrier execution only when VowpalWabbit training enables it and the prepared dataset has more than one partition. Add Spark-stage execution-path assertions for enabled multi-partition, disabled multi-partition, and enabled single-partition training. ## Prompting Intent Revalidate the current VowpalWabbitBaseLearner behavior, restore the minimal source-compatible rule proposed by the ancient PR, prove it through TDD and targeted VW validation, and keep the change isolated from LightGBM. ## Linked Sources - Original proposal: https://github.com/microsoft/SynapseML/pull/1912 - Initial CI failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229206650 - Repository review guidance: .github/skills/code-review/SKILL.md - Local validation guidance: .github/skills/synapseml-local-setup/SKILL.md ## Rationale Keep the public API and multi-partition synchronization behavior unchanged with a short-circuit partition-count guard. Observe Spark stage metadata rather than adding a test-only PipelineStage subclass, and use a marker job to drain asynchronous listener events deterministically; this proves the selected execution path without entering global stage discovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: reserve LightGBM worker ports until network init (#2595) * fix: preserve hand-written Python package initializers (#2590) * fix: validate OpenAIPrompt Java post-processing options (#2576) Co-authored-by: Ranadeep Singh <ranadeep.dtu@gmail.com> * test: clean up Fabric E2E artifacts (#2615) * test: clean up Fabric E2E artifacts ## Summary Run the existing stale-artifact cleanup before Fabric E2E tests and delete each Lakehouse and Spark Job Definition created by a suite after it finishes. Restrict stale cleanup to SynapseML test naming patterns and add regression coverage for cleanup ordering, failure preservation, and pipeline wiring. ## Prompting Intent The engineer asked to make the remaining SynapseML pull requests merge-ready, diagnose failing checks to root cause, keep fixes lean and performant, and avoid changing the original intent of unrelated PRs. Repeated PR #2604 runs reproduced a shared Fabric workspace artifact-quota failure, so this change isolates the CI infrastructure repair from that PR. ## Linked Sources - Original Fabric E2E infrastructure PR: https://github.com/microsoft/SynapseML/pull/2495 - Blocked pull request: https://github.com/microsoft/SynapseML/pull/2604 - Reproduced Azure Pipelines failure: https://msdata.visualstudio.com/A365/_build/results?buildId=230268903 ## Rationale A dedicated cleanup change avoids broadening PR #2604. Running the already-defined stale cleanup first recovers leaked capacity, while tracking and deleting only artifacts created by each suite prevents recurrence. Reverse-order, best-effort deletion removes job definitions before their backing store, tolerates already-deleted resources, and still surfaces every real cleanup failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128 * fix: harden Fabric artifact cleanup ## Summary Address review feedback by logging artifact-specific cleanup failures, computing the stale cutoff once per cleanup pass, and waiting for notebook workers to terminate before deleting their artifacts. Add regression coverage for executor shutdown ordering. ## Prompting Intent The engineer asked to resolve all active pull-request comments while keeping CI fixes lean, performant, and free of cleanup races. The three review threads on PR #2615 identified diagnostics, consistency, and concurrency issues in the initial Fabric artifact lifecycle repair. ## Linked Sources - Cleanup failure diagnostics review: https://github.com/microsoft/SynapseML/pull/2615#discussion_r3739655643 - Executor shutdown review: https://github.com/microsoft/SynapseML/pull/2615#discussion_r3739655659 - Stale cutoff review: https://github.com/microsoft/SynapseML/pull/2615#discussion_r3739655670 - Pull request: https://github.com/microsoft/SynapseML/pull/2615 ## Rationale The original exceptions remain unwrapped so callers retain their exact failure types and suppressed errors, while logs now identify the affected artifact. A single cutoff avoids boundary drift. Graceful executor shutdown followed by forced interruption ensures deletion cannot race active notebook work; a hard failure is preferable to deleting resources still in use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128 * test: harden Fabric artifact cleanup ## Summary Restrict stale Spark Job Definition cleanup to the exact Fabric E2E notebook allowlist, cover forced executor interruption, guarantee test executor cleanup, and execute cleanup plus notebook suites in one sbt process. ## Prompting Intent Extensively review PR #2615 and make it merge-ready while keeping destructive cleanup narrowly scoped, preventing in-flight notebook work from racing artifact deletion, preserving test reliability, and reducing CI startup overhead. ## Linked Sources - Pull request: https://github.com/microsoft/SynapseML/pull/2615 - Executor cleanup review: https://github.com/microsoft/SynapseML/pull/2615#discussion_r3739655659 - Cleanup diagnostics review: https://github.com/microsoft/SynapseML/pull/2615#discussion_r3739655643 - Cutoff consistency review: https://github.com/microsoft/SynapseML/pull/2615#discussion_r3739655670 ## Rationale An exact notebook allowlist avoids deleting unrelated workspace artifacts that merely share the ExploreAlgorithms prefix. A timeout overload makes the forced-shutdown path deterministic to test without slowing production cleanup. Keeping both Fabric test commands in one sbt process preserves fail-fast ordering while avoiding a second JVM startup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128 --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128 * fix: repair Spark 4.1 CI regressions ## Summary Repair the full Spark 4.1 validation failures exposed by Azure build 230432302: use an scoverage release that supports Scala 2.13.17, restore codegen-owned Python package initializers, preserve recommendation wildcard exports, make optional Python matrix variables shell-safe, and reuse the migrated shared T4 pool with the DBR 18.0 runtime. ## Prompting Intent The engineer asked to keep spark4.1 synchronized with master, run all validation, diagnose every failing check rather than adding broad exceptions, preserve Spark 4.1-specific runtime differences, and leave PR #2617 ready to merge with lean and performant fixes. ## Linked Sources - Pull request: https://github.com/microsoft/SynapseML/pull/2617 - Failed Azure validation: https://dev.azure.com/msdata/A365/_build/results?buildId=230432302 - T4 pool migration: https://github.com/microsoft/SynapseML/pull/2579 - Python initializer preservation: https://github.com/microsoft/SynapseML/pull/2590 - scoverage Scala 2.13.17 support: https://github.com/scoverage/scalac-scoverage-plugin/releases/tag/v2.4.0 ## Rationale Upgrade scoverage instead of disabling coverage so PR validation retains instrumentation. Keep hand-written initializer sources empty where codegen owns exports so generated classes and manual function modules compose correctly, while retaining the existing recommendation model API. Reuse the already-provisioned T4 instance pool because Databricks pools are runtime-agnostic; the cluster still runs DBR 18.0 and strict node-type/capacity validation remains intact. Use Bash parameter expansion for optional matrix variables so absent Azure macros are not interpreted as commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128 * fix: repair remaining Spark 4.1 CI regressions ## Summary Fix the remaining cognitive Python, core R, and Databricks GPU failures on the Spark 4.1 synchronization PR. Correct generated OpenAIPrompt inheritance, preserve nested R pipeline stages and ANSI identifier behavior, retain the Databricks runti…
Rana Singh (ranadeepsingh)
added a commit
that referenced
this pull request
Aug 10, 2026
* fix: make Fabric test artifact names unique across parallel runs ## Summary Append a compact UUID to Fabric Spark Job Definition and store artifact display names so parallel CI agents cannot collide within the same timestamp second. Keep legacy artifact names eligible for stale cleanup and add deterministic naming and cleanup-recognition coverage. ## Prompting Intent Audit the merged Fabric cleanup behavior under parallel PR validation, diagnose any cross-run failures, and create a lean fix that preserves artifact ownership boundaries and the original cleanup intent while making concurrent runs reliable. ## Linked Sources - Parallel failure evidence: https://dev.azure.com/msdata/A365/_build/results?buildId=230394225 - Fabric cleanup PR: #2615 - Affected validation PR: #2575 ## Rationale Use random UUIDs rather than finer timestamp precision because independent agents can still observe identical clock values. Retain the timestamp for human diagnostics and stale-artifact matching, make the UUID suffix optional only in the exact cleanup allowlist so pre-fix leaks remain removable, and leave normal teardown scoped to tracked object IDs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128 * fix: keep Fabric store artifact names alphanumeric ## Summary Remove the separator before UUIDs in Lakehouse and Warehouse display names, while preserving the separator for Spark Job Definitions. Update stale-cleanup recognition and regression tests to enforce each artifact type's accepted format. ## Prompting Intent Monitor the parallel-validation fix in live Fabric CI, diagnose any failure precisely, and correct it without weakening uniqueness, cleanup safety, or the original intent of the pull request. ## Linked Sources - Failed validation build: https://dev.azure.com/msdata/A365/_build/results?buildId=230398662 - Fix pull request: #2616 - Original collision build: https://dev.azure.com/msdata/A365/_build/results?buildId=230394225 ## Rationale Fabric accepts hyphens in Spark Job Definition display names but rejected the UUID-separated Lakehouse name as invalid. Concatenating the hexadecimal UUID directly keeps store names strictly alphanumeric, preserves the full collision-resistant identifier, and lets cleanup continue to use an exact artifact-type-specific allowlist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128 --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d39bfc-927c-418a-90a8-e0f2cd8fc128
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues/PRs
Unblocks the Fabric E2E checks currently failing on #2604 and #2612 with
PowerBIMetadataWorkspaceArtifactsQuotaExceededException.What changes are proposed in this pull request?
FabricTestCleanupsuite before Fabric smoke/notebook tests.afterAll.Why is this separate from #2604?
The quota failure is shared Fabric test infrastructure debt introduced with #2495, not a release-compatibility regression. Keeping the repair separate preserves #2604's original scope and prevents every successful PR build from leaking more workspace artifacts.
How is this patch tested?
FabricTestArtifactTrackerSuite: 4 tests passed.tools/ci/tests/test_pipeline_yaml.py: 15 tests passed.