Repository navigation
Engineering Health Review 2026 10 31
A pass over the test suite, CI, docs accuracy and dependencies on main at 42dab2f (fixes merged as 32c1016), 2026-10-31. It complements the Project Review (code, architecture, security) and the Product Review; findings already accepted there (for example D4, no Python lock file) are not repeated. Small fixes landed in #222 (C1 in #213, found by a parallel review the same day); the rest are proposals, ranked by how much they cost a contributor or user today.
The engineering base is healthy. The fast suite (1286 tests) passes in random order, on 4 to 8 parallel workers and on Python 3.14; main had no flaky CI failure in the last 100 workflow runs; the CLI reference matches every command and option, and the Getting Started flow runs end to end. What this pass found is mostly at the edges: CI that silently tests less than it looks like (the heavy tier and the nightly), Dependabot PRs that go red for config reasons, a CLI typo that looked like failing tests, and a connection leak that only showed as warnings.
| Check | Result |
|---|---|
pytest -m "tier_fast and not quarantined" (Python 3.13) |
1270 passed, 3 skipped, 83.6% coverage, 114 s serial on the review machine |
Same suite, pytest-randomly (3 seeds) and pytest-xdist (-n 4, -n 8, 5 runs) |
Green, except one parallel-only failure (T3, fixed). 41 s with -n 4
|
| Same suite on Python 3.14.6 | 1278 passed |
Heavy tier (tier_heavy and not tier_external) |
25 passed in 4 s |
Frontend: npm ci, typecheck, ESLint, vitest (98 tests), build |
Green; one timeout under load (T4, fixed) |
ruff, ruff format --check, mypy, docs_vault.py lint
|
Clean |
| Last 100 GitHub Actions runs |
ci: 20 green, 3 red (two Dependabot config failures, one PR that forgot updated dates, which is the lint working), 2 cancelled as superseded. No red run on main
|
CLI reference against the Typer tree (every command and --option) |
Matches; qasight lease enable is only in the Shared Resource Pools guide |
| Getting Started, by hand in a clean folder | Works; prints a confusing workers warning (U1) |
| Relative links in README, CONTRIBUTING, examples, integrations | All resolve. External links could not be checked from the review sandbox |
| TypeScript 6 and 7 against the frontend | Both compile it cleanly; both blocked by tooling peers (P1) |
npm audit |
One moderate advisory in dev-only lint tooling (P6) |
| # | Severity | Finding | Fix |
|---|---|---|---|
| C1 | High |
A CLI typo looked like failing tests. main() runs Typer with standalone_mode=False, so Click's usage errors escaped: qasight rnu, qasight run --cylce web or a missing argument printed a traceback and exited 1 (DOMAIN_FAILURE, "tests failed"), not the documented 2. A CI job with a typo went red as if tests had failed |
Fixed in #213: Click's usage message and exit 2
|
| R1 | Medium |
Database connections leaked. reset_engine_cache() cleared an lru_cache without dispose(), LRU eviction did the same, and qasight doctor / qasight config-db never closed their probe engine. The fast suite printed 110 unclosed database warnings |
A dict cache that disposes on reset, dispose() after each probe, one engine_kwargs() instead of three copies; a test checks the pooled connections are closed |
| D1 | Medium |
Dependabot PRs went red for config reasons. #211 bumped setup-python in the root action.yml only, failing the copies-match contract; #210 failed changelog required because Dependabot adds its no-changelog label just after opened; #210 also moved @types/node to 26 while CI builds on Node 22 |
Dependabot watches both action copies; changelog required skips Dependabot by author; @types/node majors are ignored and move with Node by hand; setup-python v7 in both copies |
| C2 | Medium |
The heavy tier never ran. pr-heavy.yml only runs on the e2e-heavy label or by hand (skipped in all 3 recent triggers), so its 25 tests could rot unseen. They take 4 s |
A step in the test job runs them on every PR and push |
| C3 | Low |
CI waited on lint. test and the version matrix had needs: format, so every run spent ~1 min on lint before starting the 2 min test job, and no pip cache was used |
Jobs run in parallel; cache: pip on every setup-python that installs the project |
| C4 | Low | The coverage gate was 19 points below the suite. Gate 65%, suite 84%, so coverage could fall a long way without CI noticing | Gate 80% |
| C5 | Low | Python 3.14 was untested. Released a year ago; not in the matrix or classifiers | Added to both; the suite passes |
| T3 | Low |
A parallel-only flake. The report-origin contract tests share one folder under STATIC_HISTORY_ROOT; under pytest-xdist one worker's teardown deleted the page another was reading (1 of 5 runs) |
One folder per process |
| T4 | Low |
The UI happy-path test flaked under load. 1.5 s alone; on a busy machine it passed vitest's 5 s test timeout, and on a CI runner a waitFor gave up at testing-library's 1 s default ("Unable to find role=link"), because getByRole on a full jsdom page is slow |
15 s timeout for this five-page test; asyncUtilTimeout 5 s for every test in src/test/setup.ts
|
| T5 | Low |
Warning noise. pytest tried to collect TestCaseRecord as a test class, and the sandbox JWT secret was under PyJWT's 32-byte minimum, which warned on every token |
__test__ = False; a 40-byte test secret |
| X1 | Low |
Small docs drift. Getting Started showed qasight 1.1.1; ci.yml's deploy job pointed at "the Sprint 4 publish workflows from PR #38" |
Version-free example; the comment names publish.yml, docker-publish.yml, move-major-tag.yml
|
Ranked, as written on 2026-10-31. All were taken up on 2026-11-01: see Follow-up.
| # | Severity | Finding | Proposal | Size |
|---|---|---|---|---|
| U1 | Medium |
The first run warns about something the user never set. qasight run --cycle unit from Getting Started prints pytest: workers=4 ignored, pytest-xdist is not installed; running serially.: workers defaults to 4, so every new user without pytest-xdist sees it on their first run |
Warn only when workers is set in qasight.yaml (carry "set explicitly" on the stage), or log the default case at debug level. Touches config/sections.py, config/schema.py, frameworks/pytest_adapter.py and the Command Reference |
S |
| P1 | Medium |
TypeScript 6 and 7 are blocked by two packages, not one. The project review names typescript-eslint (peer <6.1). openapi-typescript 7.13, the latest, also has a typescript ^5.x peer, so npm ci refuses TypeScript 6 (seen in #222). TypeScript 7 already compiles the UI with no errors |
Keep #106 open as the tracker, or close it with a note and let Dependabot reopen it; recheck both packages' peers monthly. No code change is waiting on us | — |
| C6 | Medium |
Per-file coverage is wrong for modules the pytest plugin imports early. The pytest11 entry point loads qasight.pytest_plugin before pytest-cov starts, so config/schema.py, engine/events.py and engine/result.py show 0% and pytest_plugin.py 13%; under coverage run -m pytest they are 98%, 100%, 100% and 36%. The total is about the same, but the per-file table points at the wrong gaps |
Run the CI suite as coverage run -m pytest (with the gate in coverage report --fail-under), or start coverage earlier with COV_CORE_SOURCE. Then the real gap, pytest_plugin.py at 36%, is visible |
S |
| C7 | Low |
The nightly is green but runs nothing. nightly-external needs TESTO_E2E_* secrets; none are set, so all 5 recent nights were a passing no-op |
Taltal's call: add a throwaway GitHub token and owner so it really runs, or drop the schedule so a green nightly doesn't suggest coverage | — |
| C8 | Low |
The fast suite could run in parallel. 114 s serial, 41 s with -n 4 on the review machine, and green in 5 parallel runs once T3 was fixed |
Add pytest-xdist to the dev extra and -n auto in CI. Watch the first week for shared-state flakes |
S |
| D2 | Low |
Container base images don't get update PRs. All three Dockerfiles default to python:3.11-slim-bookworm (Debian 12, now oldstable), and Dependabot has no docker ecosystem |
Add the docker ecosystem; move the defaults to a current Debian and Python in one PR, checked by the image scan and start jobs |
S |
| D3 | Low |
Starlette is deprecating httpx in its test client. Every API test prints install httpx2 instead; a future Starlette will drop it |
Swap the dev dependency when FastAPI's test client moves; nothing to do yet |
— |
| D4 | Low |
One moderate npm audit advisory. minimatch below 11.0.1 (ReDoS) under ESLint and typescript-eslint. Dev tooling only, nothing ships to users. An overrides entry breaks their peer ranges |
Wait for ESLint and typescript-eslint to move to minimatch 11 |
— |
| D6 | Low |
The pre-commit hook and CI run different ruff versions. .pre-commit-config.yaml pins ruff v0.15.0; CI installs ruff>=0.15.0, today 0.17.3. They agree on the repo now, but ruff changes its format style yearly, and then a hook-formatted commit fails ruff format --check in CI. Dependabot doesn't bump pre-commit revs |
Pin ruff to one version in the dev extra and the hook, and bump both together (or run pre-commit autoupdate in the weekly deps PR) |
XS |
| D5 | Low |
Two Postgres versions. The root docker-compose.yml uses postgres:15-alpine, the team server compose file postgres:16-alpine
|
Use one version in both | XS |
Taltal asked for all of the proposals. What each one became, in one pull request:
| # | Outcome |
|---|---|
| U1 | Done. A stage carries workers_explicit; on the built-in default a pytest stage without pytest-xdist runs serially without a warning. workers on the stage, in defaults: or --workers still warns |
| C6 | Done, by a different fix than proposed: import qasight no longer loads config and engine eagerly (they resolve on first use), so the pytest11 plugin imports nothing measurable before pytest-cov starts. config/schema.py now reports 99%, engine/events.py and engine/result.py 100%. pytest_plugin.py itself still loads before coverage starts and shows 14%; its function bodies are measured when they run |
| C7 | Done without a token: the nightly job runs only when the repository variable TESTO_E2E_ENABLED is true, so until Taltal adds the secrets and the variable it shows as skipped, not green |
| C8 | Done. pytest-xdist is in the dev extra and CI runs the fast suite with -n auto (1617 tests in 66 s on 4 cores, coverage 86.9%) |
| D2 | Done. The three Dockerfiles use python:3.13-slim-trixie (Debian 13) as literal FROM lines Dependabot can read, the server's UI stage node:22-trixie-slim; Dependabot has docker and docker-compose ecosystems (Postgres majors stay manual) |
| D5 | Done. Both compose files use postgres:16-alpine; an old local volume needs a dump and restore or docker compose down -v
|
| D6 | Done. ruff is pinned (ruff==0.17.3) in the dev extra and the hook; tests/contract/ci/test_ruff_pin.py fails when they differ |
| P1 | Nothing to change yet, rechecked: openapi-typescript 7.13.0 still has peer typescript ^5.x, typescript-eslint 8.72.1 <6.1.0; TypeScript is at 7.0.2. #106 stays open as the tracker |
| D3 | Nothing to change yet: FastAPI's test client still uses httpx
|
| D4 | Nothing to change yet: the same minimatch advisory, now listed under 13 dev-only packages (ESLint, typescript-eslint and openapi-typescript chains); none ship to users |
- Project Review: code, architecture and security scorecard
- QA Strategies: test tiers and which workflow runs which
- CI-CD Pipeline Setup: the workflows
- Technical Debt Tracker: where accepted items live long term
Generated from docs/Specs & ADRs/Engineering Health Review - 2026-10-31.md. Edit it there; changes made in the wiki are overwritten.
Architecture
CLI Commands
Processes & Guides
- CI-CD Pipeline Setup
- Closed Network Install
- E2E Harness Operations Guide
- GitLab Pages Demo
- Publishing Docker Images
- Publishing to Artifactory
- Publishing to PyPI
- Setup Snippets
- Shared Resource Pools
- Stability and Deprecation Policy
- Team Server
- Upload Without Python
Prompts & Snippets
Roadmap & Strategy
- Feature Gaps and Quick Wins - 2026-10-31
- New User Journey Audit - 2026-11-01
- Paid Features to Offer Free - 2026-10-31
- Product Roadmap
Specs & ADRs
- Adoption Blockers - 2026-11-12
- Affected Test Selection - 2026-10-30
- Alerts Where Developers Look - 2026-10-22
- Architecture Consolidation - 2026-10-06
- Architecture Review - 2026-10-31
- Automation and Unification - 2026-11-12
- BehaveX Reports - 2026-10-21
- Better Flaky Signal - 2026-11-01
- CLI-Only Bug Hunt - 2026-11-15
- Closed Network Install - 2026-11-15
- Closed-Network Monorepo Rollout - 2026-11-20
- Command Adapter and JUnit Import - 2026-09-25
- Config JSON Schema - 2026-10-22
- Crash Detection - 2026-10-31
- Delta Comparison Policy
- Docs Site - 2026-10-19
- Docs Vault and Wiki Sync - 2026-10-07
- Easy Setup - 2026-11-01
- Engine Fixes - 2026-11-01
- Engineering Health Review - 2026-10-31
- Exporting Results - 2026-11-01
- External Audit Tools - 2026-11-11
- Flaky Test Issues - 2026-11-01
- Flaky Test Quarantine - 2026-10-22
- Focus Review - 2026-11-20
- Free Copies of Paid Features - 2026-11-01
- GitHub Action PR Comment - 2026-10-19
- Go Uploader - 2026-11-20
- Human-Run Resource Provider - 2026-10-30
- Importer and Integration Bug Hunt - 2026-10-31
- Ingest and Team Server - 2026-10-22
- Interview Readiness Fixes - 2026-10-07
- Library Packaging
- Lifecycle Fixes - 2026-11-02
- MCP Server - 2026-11-01
- Menu by Question - 2026-11-12
- Merge and Automate - 2026-11-20
- New User Journey Automation - 2026-11-01
- Open Source Features - 2026-10-19
- OpenTelemetry Export - 2026-10-30
- Phase 3 Delta Comparison Engine Plan
- Phase 3 Unified Dashboard Plan
- Phase 4 BYOK and Failure Analysis
- Phase 5 UI Redesign - Cycles-First Navigation
- Playwright Adapter - 2026-10-19
- Playwright JSON Import - 2026-10-22
- Plug and Play - 2026-11-12
- Plugin API - 2026-10-19
- Product Roadmap Work - 2026-10-25
- Product to Ten - 2026-10-25
- Project Health - 2026-10-26
- QASight Logo - 2026-11-05
- Qualflare Follow-ups - 2026-10-26
- Quarantine From the UI - 2026-11-01
- Specs & ADRs
- Release Gate - 2026-11-12
- Release Scorecard - 2026-11-12
- Rename to QASight - 2026-11-04
- Repository Pattern - Database-Agnostic Refactor
- Resource Lifecycle Hooks - 2026-10-26
- Resource Pools and Leases - 2026-10-26
- Run Deletion and Stop - 2026-11-02
- Runner Coverage - 2026-10-26
- Scale Runs and Stages - 2026-10-24
- Scorecard on MegaLinter - 2026-11-12
- Security and Install Review - 2026-10-31
- Standards Compliance - 2026-11-12
- Switchable Team Features - 2026-10-24
- Test Delivery Metrics - 2026-11-02
- Test Labels, Environments and Releases - 2026-11-02
- Test Source in AI Summaries - 2026-11-01
- Throwaway Stage Containers - 2026-10-24
- Tidy Cards and Demo Examples - 2026-11-05
- UI Design Review - 2026-10-31
- UI Polish - 2026-11-08
- UI Restyle - 2026-10-19
- UI Review Fixes - 2026-11-01
Testing Workflows
Vault