FIX: Flaky tests in CI and mark stress tests

[bewithgaurav]

Work Item / Issue Reference

AB#45424

GitHub Issue:

---

Summary

Fix all known flaky test failures in PR validation. Audited 30 recent builds — 12 had failures, all from 2 test files on macOS or ARM64 QEMU. Also cap macOS benchmark step to prevent job timeouts.

---

Audit (30 PR validation builds)

test_022 — GIL heartbeat (8/30 builds, macOS only)

Test	Hits	Cause
test_query_does_not_block_other_python_threads	5/30	Heartbeat tick threshold too aggressive for CI CPU contention
test_commit_does_not_block_other_python_threads	3/30	Same

test_013 — shutdown handle cleanup (7/30 builds, macOS + QEMU)

Test	Hits	Cause
test_dbc_handle_cleanup_at_shutdown	2/30	macOS timeout + QEMU SEGV
test_force_gc_finalization_order_issue	2/30	macOS timeout
test_aggressive_dbc_segfault_reproduction	1/30	macOS timeout
test_cleanup_connections_weakset_modification_during_iteration	1/30	macOS timeout
test_weakref_cleanup_at_shutdown	1/30	QEMU SEGV
test_cleanup_connections_scenarios	1/30	QEMU
test_exception_during_query_with_shutdown	1/30	QEMU SEGV (found during this PR)

macOS benchmark timeout (2/30 builds)

Benchmark step had no timeout. On macOS, it takes 10–28 min (vs 3 min Linux, 5 min Windows). Slow runs eat the 60 min job budget and kill the entire job.

---

Changes

tests/test_022_concurrent_query_gil_release.py

• Mark both heartbeat tests with @pytest.mark.stress
• Excluded from PR validation by pytest.ini addopts; runs in nightly stress pipeline

tests/test_013_SqlHandle_free_shutdown.py

• Bump subprocess timeout from 5s/3s to 15s on all ODBC shutdown tests (fixes macOS TimeoutExpired)
• Skip entire TestHandleFreeShutdown class on QEMU (verified 0/400 on native ARM64 — all subprocess shutdown tests can SEGV under QEMU user-mode emulation)

tests/conftest.py

• Add shared is_qemu_emulated() helper and QEMU flag for reuse across test files

eng/pipelines/pr-validation-pipeline.yml

• Add timeoutInMinutes: 20 on macOS benchmark step with continueOnError: true
• Slow benchmarks are terminated gracefully without failing the build

---

What this does NOT change

• No test logic or C++ changes
• All tests run on every native platform (Linux x86, Windows, macOS, native ARM64)
• Benchmark script unchanged — macOS benchmark variance (2.7x) is a separate investigation

---

Benchmark duration analysis (13 builds)

Platform	Range	Variance
Linux Ubuntu	3m06s – 3m24s	1.1x
Windows	4m35s – 6m12s	1.3x
macOS	10m28s – 28m39s	2.7x

---

Additional findings (not in this PR)

• is_python_finalizing() in ddbc_bindings.cpp checks sys._is_finalizing which was renamed to sys.is_finalizing in Python 3.13+. Shutdown protection silently degrades on 3.13+, but atexit cleanup saves us. Separate fix needed.
• macOS benchmark variance needs deeper investigation (runner contention vs code issue).

[Copilot]

Pull request overview

This PR aims to prevent timing-sensitive/flaky tests from blocking pull requests by classifying them as stress tests and explicitly excluding stress-marked tests from the PR validation pipeline’s pytest runs.

Changes:

• Mark the two GIL-release heartbeat tests in test_022_concurrent_query_gil_release.py with @pytest.mark.stress.
• Add an explicit -m "not stress" filter to all pytest invocations in eng/pipelines/pr-validation-pipeline.yml as defense-in-depth.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
tests/test_022_concurrent_query_gil_release.py	Marks two timing-sensitive concurrency tests as stress so they’re excluded from default PR test runs.
eng/pipelines/pr-validation-pipeline.yml	Updates PR pipeline pytest commands to explicitly exclude stress tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 100%

🎯 Overall Coverage 80%

📈 Total Lines Covered: 6654 out of 8256 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.7%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.1%
mssql_python.row.py: 76.9%
mssql_python.__init__.py: 77.3%
mssql_python.pybind.connection.connection.cpp: 77.3%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.logging.py: 85.5%
mssql_python.connection.py: 85.6%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[jahnvi480]

LGTM!