Skip to content

Quarantine flaky pipeline tests; add triage prompt#4476

Draft
paulmedynski wants to merge 2 commits into
mainfrom
dev/paul/flaky-round-2
Draft

Quarantine flaky pipeline tests; add triage prompt#4476
paulmedynski wants to merge 2 commits into
mainfrom
dev/paul/flaky-round-2

Conversation

@paulmedynski

@paulmedynski paulmedynski commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Description

Quarantines two intermittently-failing tests observed in the public CI pipelines on main at/after commit bf4cd14, and adds a reusable prompt for triaging pipeline test failures.

Test changes (engineering processes / test coverage):

  • SimulatedServerTests.ConnectionTests.TransientFault_RetryEnabled_ShouldSucceed_Async — under CI load the simulated transient error occasionally resurfaces on the retry login, so the async open propagates the SqlException instead of succeeding. Same transient-fault retry-timing family as its already-quarantined siblings. Marked [Trait("Category", "flaky")] with the observed failure signature.
  • ConnectionPoolTest.TransactionPoolTest.TransactionCleanupTest — the process-global pool (keyed by connection string) intermittently reports one extra connection (Assert.Equal Expected 2, Actual 3) due to cross-test contamination while running inside a TransactionScope. It is a test-isolation / pool-count race, not a product defect. Marked [Trait("Category", "flaky")] with an accurate root-cause comment.

Tooling:

  • Adds .github/prompts/triage-pipeline-failures.prompt.md — an access-agnostic (ADO MCP / az CLI / REST) workflow to find and classify failing tests in the SqlClient pipelines at/after a given commit, capture full xUnit output/stack traces, distinguish regression vs pre-existing, and present findings for approval before fixing or quarantining.

No product code changes; no public API changes.

Two intermittently-failing tests observed in public CI at/after bf4cd14 are quarantined, and a reusable, access-agnostic (MCP/CLI/REST) prompt for triaging pipeline test failures is added.

- SimulatedServerTests.ConnectionTests.TransientFault_RetryEnabled_ShouldSucceed_Async: transient-fault retry timing under CI load (SqlException surfaces on the retry login). Marked [Trait("Category","flaky")].

- ConnectionPoolTest.TransactionPoolTest.TransactionCleanupTest: process-global pool count race (Expected 2, Actual 3) from cross-test contamination inside a TransactionScope. Marked flaky with an accurate failure-signature comment.

- Add .github/prompts/triage-pipeline-failures.prompt.md.

Signed-off-by: Paul Medynski <31868385+paulmedynski@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 24, 2026 17:06
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Jul 24, 2026
Comment thread .github/prompts/triage-pipeline-failures.prompt.md
@paulmedynski paulmedynski added the Area\Tests Issues that are targeted to tests or test projects label Jul 24, 2026
@paulmedynski paulmedynski added this to the 7.1.0-preview3 milestone Jul 24, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Quarantines two intermittently failing tests in the SqlClient CI by marking them with the existing Category=flaky trait, and adds a reusable Copilot prompt to triage pipeline test failures at/after a specific commit SHA.

Changes:

  • Mark TransientFault_RetryEnabled_ShouldSucceed_Async (simulated server unit test) as quarantined via [Trait("Category", "flaky")] with an embedded failure signature comment.
  • Mark TransactionCleanupTest (manual pooling test) as quarantined via [Trait("Category", "flaky")] with an embedded failure signature comment.
  • Add .github/prompts/triage-pipeline-failures.prompt.md to guide ADO/GitHub pipeline failure investigation and classification workflows.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/Microsoft.Data.SqlClient/tests/UnitTests/SimulatedServerTests/ConnectionTests.cs Quarantines an intermittently failing transient-retry simulated-server test.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionPoolTest/TransactionPoolTest.cs Quarantines an intermittently failing transaction-pool cleanup test.
.github/prompts/triage-pipeline-failures.prompt.md Adds a triage workflow prompt for classifying pipeline test failures and deciding fix vs quarantine.
Comments suppressed due to low confidence (1)

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionPoolTest/TransactionPoolTest.cs:93

  • Quarantining this test as [Trait("Category", "flaky")] reduces regular CI coverage, but the described failure mode (pool ConnectionCount off by 1) can be made deterministic by isolating the pool key per invocation. Since pools are process-global and keyed by connection string, appending a unique ApplicationName (via SqlConnectionStringBuilder) prevents cross-test contamination without quarantining the test.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [ClassData(typeof(ConnectionPoolConnectionStringProvider))]
        public static void TransactionCleanupTest(string connectionString)
        {

Comment thread .github/prompts/triage-pipeline-failures.prompt.md Outdated
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.70%. Comparing base (d040700) to head (b8bc2d3).
⚠️ Report is 4 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (d040700) and HEAD (b8bc2d3). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (d040700) HEAD (b8bc2d3)
CI-SqlClient 1 0
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4476      +/-   ##
==========================================
- Coverage   70.88%   62.70%   -8.19%     
==========================================
  Files         288      283       -5     
  Lines       43935    66821   +22886     
==========================================
+ Hits        31142    41898   +10756     
- Misses      12793    24923   +12130     
Flag Coverage Δ
CI-SqlClient ?
PR-SqlClient-Project 62.70% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Fix prompt tool identifier 'codebase/search' -> 'search/codebase' in triage-pipeline-failures.prompt.md, matching the documented identifier in generate-prompt.prompt.md.

- Quarantine flaky ConnectionFailoverTests.NetworkError_WithUserProvidedPartner_RetryDisabled_ShouldConnectToFailoverPartner with [Trait("Category", "flaky")]. It fails under CI load only: the 5s ConnectTimeout can be exhausted by the failover connection itself on slow agents. Captured the xUnit output and stack trace in a comment for future triage.
Copilot AI review requested due to automatic review settings July 24, 2026 19:34
@paulmedynski paulmedynski moved this from To triage to In progress in SqlClient Board Jul 24, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area\Tests Issues that are targeted to tests or test projects

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

2 participants