Technical Debt Tracker

Technical Debt Tracker

Index

Open backlog for the qasight codebase. Items come from code reading and known contract gaps, not only inline TODO markers (there are none in qasight/, qasight_api/ or tests/).

Related: Deep Dive - Execution Logic, Troubleshooting and Error Codes, Architecture Overview.

---

Open

1. Signal deaths classify as test failures

Timeouts normalize to return code 124 (exit 3) and engine exceptions to exit 4, but other signal deaths (e.g. SIGKILL, rc 137) still classify as exit 1. This is locked as a known misclassification in tests/contract/qasight/test_exit_code_contract.py until a signal-aware classifier lands.

2. Reporter failures don't fail the run

qasight/reporting/reporters/factory.py catches per-reporter exceptions, so a configured integration can be skipped silently after a green run. Fix: an opt-in reporters_required: true that maps a reporter failure to exit 3, plus a reporter_failed NDJSON event.

3. Log reader join timeout

executor.py calls reader.join(timeout=2.0) after the subprocess exits, so a very large final stdout burst could be cut short. Fix: drain until EOF; add an integration test with a large burst.

Known and intentional

• Per-stage Allure wipe: executor.py removes the stage's allure-results/ before each run to isolate stages. Retries must use separate stage names.
• LogBuffer.on_chunk contains renderer errors (engine/log_buffer.py) so a broken renderer cannot crash a run; the first failure per stage is logged as a warning.
• Boundary catch-alls: the except Exception sites left in qasight/ and qasight_api/ are deliberate boundaries (orchestrator, reporter factory, persistence backends, API routes, connection checks, third-party report generators). Each logs with exc_info and has a one-line comment saying why it is broad.
• Orchestrator catch-all: run_plan() turns an exception from run_stage() into an internal-failure stage result, which classifies the plan as exit 4 rather than crashing.

Resolved

• Sequential-only orchestrator: a cycle can set max_parallel_stages (see Scale Runs and Stages - 2026-10-24); the API caps concurrent runs with TESTO_MAX_CONCURRENT_RUNS.

• Exit-code contract single-sourced in engine/exit_codes.py; timeouts → 124 → exit 3, engine errors → exit 4.

• The second, Docker-based execution stack (HeadlessEngineService, runners.py, pluggy plugins, /api/v1/executions, Streamlit UI) was removed in v1.1; every run goes through CycleRunService and the engine.

• --ci forces a synchronous report archive; a required archive failure exits 3.

• qasight/persistence/ (JsonBackend, DbBackend, composite_backend()) replaced the persistence stub.

• MinIO, the pre-v1.1 snapshot reads and the Docker-stack record shape in history/ were removed in v1.1.

• mypy qasight qasight_api reports 0 errors and blocks CI, as does ruff format --check.

• Silent-error audit: I/O paths (ReportPortal, Extent, metrics and Allure readers, doctor checks, persistence) catch specific exceptions (OSError, ValueError, requests.RequestException, jinja2.TemplateError, SQLAlchemyError) and log with exc_info instead of passing silently.

• BehaveX runs without --formatter, so it writes its own HTML report again; Allure results come from its report.json after the stage. Export and HTML failures are logged, appended to run.log and set StageResult.error (exit code unchanged), so the stage panel and stage_finished NDJSON show them.

• A failed Git trigger evaluation logs a warning and prefixes the snapshot fallback's reason with the Git error, so the trigger panel and cycle_trigger NDJSON event show it.

Refreshing this note

On each release or large refactor, re-run rg 'TODO|FIXME|HACK' qasight qasight_api tests, look for new except Exception sites in qasight/engine/, and move fixed items to Resolved.

---

Generated from docs/Testing Workflows/Technical Debt Tracker.md. Edit it there; changes made in the wiki are overwritten.