Skip to content
Alex Miarsky edited this page Nov 10, 2026 · 1 revision

Application test strategy

The application client uses executable contracts split by failure type, so a regression reports the broken boundary rather than only a later visual symptom.

Test layers

Core and server procedure contracts

tests/test_contracts.py discovers every tq.* procedure passed to execute_procedure_serverside by thlib/tactic_classes.py, generates its server script, and compiles it. This catches stale source offsets, malformed generated Python, missing function names, and accidental syntax fragments before XML-RPC.

The same module enforces:

  • code-based identity for task, note, message, snapshot, and activity procedures; deprecated id links may be read only by explicit legacy link parsing code;
  • stable result keys from the batched server-update, task-page, and user-profile procedures (tests/contracts/data_shapes.json);
  • native SObject, Task, Snapshot, and File behavior, including the native file repository/path/preview API;
  • reviewed metadata namespaces (tests/contracts/metadata_schemas.json);
  • the public top-level functions, classes, and public methods of tactic_classes.py, tactic_query.py, and environment.py.

The public API baseline is generated by tests/support/build_contract_manifest.py. Regenerate it only after an intentional API compatibility review. Adding optional parameters is compatible; deleting or reordering established positional parameters is not.

Controller and worker scenarios

tests/test_controller_scenarios.py uses deterministic deferred workers and in-memory settings. It covers selection-to-request-to-store flow, duplicate launch rejection, stale results after selection changes, cancellation and late callbacks during shutdown, persisted window state, and cache hit/invalidate/refresh behavior.

ServerUpdateService now assigns a generation to every worker. stop() and project changes invalidate that generation and cancel the worker, so a queued callback can never advance cursors or mutate models for a dead session.

QML smoke harness

tests/test_qml_smoke_harness.py launches the real thlib/ui/application.py composition root with the offscreen renderer. It uses a temporary Handler settings directory and TACTIC_QML_SMOKE_OFFLINE=1, therefore it neither reads user window state nor starts TACTIC bootstrap, Handler Server, or polling.

Every registered standalone window is opened and closed twice. The harness fails on QML startup errors, warnings, missing properties/handlers, binding loops, ReferenceErrors, required-property failures, or a visible window below its declared minimum. Finally it destroys the QML engine and executes the normal ordered shutdown; the subprocess timeout detects a timer, connection, worker, or QObject lifecycle hang.

Run it directly with:

python -m unittest tests.test_qml_smoke_harness -v

Before running the client and extracted server-procedure tests, install requirements-dev.txt. On Windows it includes the IANA time-zone data used by the procedure fixtures; those fixtures run in Python without the TS server's runtime dependencies.

For application startup and its controller/QML dependency wiring, run:

python -m unittest tests.test_application_runtime tests.test_nested_columns tests.test_administration_menu_qml tests.test_qml_smoke_harness
python tests/run_entrypoint_smoke.py

The startup regression constructs the actual application bundles and publishes their real bindings, rejects missing values and duplicate names, and verifies that embedded tables share the application's task and column owners. Do not manually add missing owners to a fixture to make that check pass. Menu checks run after loading the real QML root and require fresh authorization after sign-out. The context-property budget remains a separate architecture guard; a passing source inventory alone does not establish a working launch.

Request budgets

thlib/ui/request_metrics.py provides process-wide, thread-safe counters for calls, failures, cumulative duration, response bytes, and cache hits/misses. The existing TACTIC server diagnostic wrapper records every outer XML-RPC call; the unified messages/activity/presence request records one logical server.poll; the skey preview resolver records cache reuse.

Request budgets live in tests/contracts/request_budgets.json. Architecture debt budgets live in tests/contracts/ui_architecture_debt.json; they cap global QML bindings, env_inst consumers, the ApplicationController/mixin method surface, the giant item-action dispatcher, oversized source files, and qmllint warnings. A cleanup lowers the stored maximum. Raising one requires an explicit architecture decision rather than silently accepting a new baseline. Current hard request invariants are:

  • reopening a registered window causes zero XML-RPC calls by itself;
  • one update batch is one logical poll;
  • repeated skey rendering uses the cache and does not add a miss.

Duration and payload size are recorded for profiling but are not compared with a machine-dependent wall-clock threshold. A scenario should instead receive a stable call/byte ceiling once its fixture response is deterministic.

Source-coupled test debt

tests/test_source_assertion_debt.py measures tests that read implementation text with read_text() or inspect.getsource() and then assert against that text. The per-file and total ceilings live in tests/contracts/source_assertion_debt.json. Existing architecture, localization, and generated-procedure contracts remain valid uses, but the baseline cannot grow. Behavior changes should use the lowest real boundary available: Python objects, QML component creation and input, model wiring, filesystem, or transport. When a source assertion is replaced by a behavior test, lower the stored ceiling in the same change.

QML render budgets

tests/contracts/qml_render_budgets.json limits the QObject and QQuickItem footprint of mass-created QML delegates. The runtime test instantiates the real component and verifies that idle ripple effects and drag overlays are not present. Construction time remains a diagnostic because it depends on the host and Qt shader cache; deterministic object counts prevent the same regression without a flaky millisecond threshold. The complete audit and lifecycle rules are documented in QML rendering and lifecycle performance.

Commands

Fast contract suite:

python -m unittest tests.test_contracts tests.test_controller_scenarios `
  tests.test_request_budgets tests.test_tactic_test_project `
  tests.test_server_updates -v

Full suite (recommended for Qt/QML isolation):

python tests/run_isolated_suite.py

Each module runs in a fresh process. Qt Quick, font, clipboard, socket, and QML engine globals therefore cannot leak into the next module. Plain unittest discover remains useful for non-QML subsets, but one long process is not a reliable lifecycle boundary for the complete PySide6 suite.

Editor input/layout fixtures select QSG_RENDER_LOOP=basic before importing the application entry point. Otherwise its production threaded default leaks into the offscreen software fixture and can deadlock synchronous input or deferred window deletion. This does not change production rendering; rendering benchmarks may still explicitly select their loop. View-mode workflow tests use keyboard selection and await a rendered frame between changes that reflow the focused view.

tests/test_editor_windows_lifecycle.py runs the complete editor class in a fresh offscreen process with a 45-second timeout, covering repeated creation, real input, and cleanup without depending on another test's Qt initialization.

On Windows, the sidebar and preview saved-search dropdown tests use tests.qt_application.run_native_windows_test to run their real mouse/popup scenarios in separate windows QPA processes with a 30-second timeout. The offscreen platform can close the first native Popup.Window during its focus handoff, making this check depend on which popup test ran first. Other sidebar layout checks remain headless; the native scenario must also pass on its own:

python -m unittest tests.test_sidebar_editor_qml.SidebarEditorQmlTests.test_preset_dropdown_keeps_rows_inside_viewport_and_reserves_scrollbar -v
python -m unittest tests.test_editor_windows_lifecycle -v

tests.test_qml_lint runs the pyside6-qmllint shipped with the pinned PySide6 toolchain over the complete QML tree. Property overrides and layout-positioning warnings have a zero budget. Existing dynamically typed missing-property, unqualified, and unused-import warnings are capped in tests/contracts/ui_architecture_debt.json; the cap may decrease but must not be raised to make a regression green. Inline suppression is limited to a named warning and documents the verified runtime ownership that makes the diagnostic inapplicable.

Release gate

Install the pinned development tools, then run the strict merge entry point:

python -m pip install -r requirements-dev.txt
python tests/run_release_gate.py

It adds mandatory Ruff and qmllint, worktree/whitespace checks, Python compilation, and foreign-working-directory launcher smoke around the isolated suite. qmllint is an explicit gate step and is excluded from the later isolated run so the complete QML tree is not linted twice. It intentionally does not turn live mutation, repository, Windows tray, Maya or visual acceptance into fake offline tests. Those required checks are declared in tests/contracts/release_acceptance.json and documented in docs/release_readiness.md.

Clone this wiki locally