PR Pipeline - Build/Pack and Local Tests

[benrr101]

Description

As the goal of this sprint, we want to create a new PR pipeline that simplifies all the complexities of the old one, reduces the test coverage to tests that can run simply and quickly, and completes quickly. This PR introduces the basics of that with a few more changes to come.

• Test execution is gated by a build/pack stage - if PR doesn't build/pack successfully, test run fails fast. ~3min
• Test matrix is split into the following platforms as stages in parallel:
  • [Windows/Linux]
  • [net462/net8.0/net9.0/net10.0]
• Test projects that run as jobs within each stage in parallel:
  • Abstractions ~2min
  • Azure ~2min
  • SqlClient Funtional ~3min
  • SqlClient Manual - Local ~30min
  • SqlClient Unit ~6min

This puts the total run time at ~40min in its current state. Compared to the current pipeline which has full runtimes of anywhere from 1.5hrs to 3hrs, this is a massive improvement.

Noteworthy notes have been added inline.

Remaining work

• SqlClient Manual tests against Azure SQL
• Code coverage collection and publishing
• Operationalize the pipeline via triggers

Testing

• Test run on internal ADO
  https://sqlclientdrivers.visualstudio.com/ADO.Net/_build/results?buildId=154247&view=results ✅
• Test run on public ADO
  https://dev.azure.com/SqlClientDrivers/public/_build/results?buildId=154249&view=results ✅
• Due diligence run of sqlclient-nonofficial
  https://sqlclientdrivers.visualstudio.com/ADO.Net/_build/results?buildId=154257&view=results 🔄
• Due diligence run of sqlclient-kerberos
  https://sqlclientdrivers.visualstudio.com/ADO.Net/_build/results?buildId=154260&view=results 🔄

[Copilot]

Pull request overview

This PR introduces a new faster PR validation pipeline under eng/pipelines/pr/, consolidating build/pack gating and parallelized local test execution across Windows/Linux and multiple TFMs. It also centralizes .NET install/tool restore pipeline templates and adjusts manual-test configuration and selected flaky/Debug-incompatible tests to fit the faster validation flow.

Changes:

• Added a new PR pipeline with build/pack, secret generation, SQL Server setup, and parallel test stage/job templates.
• Moved shared pipeline install/tool-restore assets into eng/pipelines/common/steps and updated most consumers.
• Updated manual-test utilities/configuration and a few test cases to reduce flaky or Debug-mode failures.

Reviewed changes

Copilot reviewed 43 out of 45 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
eng/pipelines/pr/pr-pipeline.yml	Adds the new PR pipeline entry point and platform matrix.
eng/pipelines/pr/variables/pr-variables.yml	Defines PR pipeline variables and test config variable group.
eng/pipelines/pr/stages/pack-stage.yml	Adds build/pack gating stage for all packages.
eng/pipelines/pr/stages/generate-secrets-stage.yml	Adds reusable SA password generation stage.
eng/pipelines/pr/stages/test-stages.yml	Adds per-platform parallel test stage generation.
eng/pipelines/pr/jobs/test-buildproj-job.yml	Adds generic build.proj test job template.
eng/pipelines/pr/jobs/test-sqlclientmanual-job.yml	Adds SqlClient manual-test job with local SQL/config generation.
eng/pipelines/pr/steps/build-buildproj-step.yml	Adds PR-specific build.proj build step.
eng/pipelines/pr/steps/pack-buildproj-step.yml	Adds PR-specific build.proj pack step.
eng/pipelines/pr/steps/test-buildproj-step.yml	Adds PR-specific build.proj test step.
eng/pipelines/pr/steps/install-dotnet.yml	Wraps common .NET install/tool restore for PR jobs.
eng/pipelines/pr/steps/configure-sqlserver-step.yml	Dispatches SQL Server setup by OS.
eng/pipelines/pr/steps/configure-sqlserver-windows-step.yml	Adds Windows local SQL Server setup.
eng/pipelines/pr/steps/configure-sqlserver-linux-step.yml	Adds Linux local SQL Server setup.
eng/pipelines/common/steps/install-dotnet.yml	Moves shared .NET install template and updates ARM64 script path.
eng/pipelines/common/steps/install-dotnet-arm64.ps1	Moves ARM64 .NET install helper script.
eng/pipelines/common/steps/restore-dotnet-tools.yml	Adds shared dotnet tool restore step.
eng/pipelines/common/variables/common-variables.yml	Narrows common variables to pipeline-wide values.
eng/pipelines/onebranch/variables/onebranch-variables.yml	Moves OneBranch-specific variables out of common variables.
eng/pipelines/onebranch/sqlclient-official.yml	Points official pipeline at common variable template.
eng/pipelines/onebranch/sqlclient-non-official.yml	Points non-official pipeline at common variable template.
eng/pipelines/onebranch/jobs/build-buildproj-job.yml	Updates shared install/tool restore template paths.
eng/pipelines/onebranch/steps/build-buildproj-step.yml	Clarifies OneBranch build-step behavior.
eng/pipelines/common/templates/jobs/ci-build-nugets-job.yml	Updates shared install/tool restore template paths.
eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml	Updates shared install template path.
eng/pipelines/common/templates/jobs/ci-run-tests-job.yml	Updates shared template paths and removes obsolete config fields.
eng/pipelines/common/templates/steps/pre-build-step.yml	Updates shared install template path.
eng/pipelines/common/templates/steps/update-config-file-step.yml	Removes obsolete config-file parameters.
eng/pipelines/common/templates/steps/verify-nuget-package-step.yml	Updates shared install template path.
eng/pipelines/jobs/pack-abstractions-package-ci-job.yml	Updates shared install template path.
eng/pipelines/jobs/pack-azure-package-ci-job.yml	Updates shared install template path.
eng/pipelines/jobs/pack-logging-package-ci-job.yml	Updates shared install template path.
eng/pipelines/jobs/pack-sqlserver-package-ci-job.yml	Updates shared install template path.
eng/pipelines/jobs/test-abstractions-package-ci-job.yml	Updates shared install template path.
eng/pipelines/jobs/test-azure-package-ci-job.yml	Updates shared install template path.
eng/pipelines/stress/stress-tests-job.yml	Updates shared install template path.
eng/pipelines/kerberos/sqlclient-kerberos.yml	Partially updates tool restore template paths.
src/Microsoft.Data.SqlClient/tests/tools/Microsoft.Data.SqlClient.TestUtilities/config.default.json	Updates default test configuration schema.
src/Microsoft.Data.SqlClient/tests/tools/Microsoft.Data.SqlClient.TestUtilities/Config.cs	Updates test config model fields.
src/Microsoft.Data.SqlClient/tests/ManualTests/DataCommon/DataTestUtility.cs	Adds Debug-build detection helper.
src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/ApiShould.cs	Skips AE API tests in Debug and refactors a large-query async test.
src/Microsoft.Data.SqlClient/tests/Common/SqlDataReaderExtensions.cs	Adds async result-flushing helpers.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/SqlNotificationTest/SqlNotificationTest.cs	Increases callback timeout and marks a flaky test.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/SqlCommand/SqlCommandCancelTest.cs	Marks named-pipe timeout test as active issue.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/InstanceNameTest/InstanceNameTest.cs	Simplifies data-source parsing in instance-name test.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.36%. Comparing base (f873c1a) to head (218eb35).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4326      +/-   ##
==========================================
- Coverage   66.44%   64.36%   -2.09%     
==========================================
  Files         284      279       -5     
  Lines       43301    66219   +22918     
==========================================
+ Hits        28773    42623   +13850     
- Misses      14528    23596    +9068     

Flag	Coverage Δ
CI-SqlClient	?
PR-SqlClient-Project	64.36% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.