Developer Testing

Developer Testing

This page collects test guidance for contributors: the main test suites, reproducible fixtures, the documentation checks against this Wiki, and the synthetic dashboard trend benchmark. Start with Contributing for the development setup.

Test suites

Suite	Command	Notes
Python	python -m pytest tests/ -n auto --ignore=tests/e2e -m "not wiki"	Analyzer, drivers, collectors, storage, web, config, MQTT, i18n, PDF generation. -n auto uses pytest-xdist; drop it to debug serially.
JavaScript	npm ci && npm test	Node 22's built-in test runner for pure browser modules such as correlation-data.js, event-log-data.js, and empty-state.js, plus render and contract checks for scripts such as events.js, the settings form, and the module views.
Browser (E2E)	TZ=UTC python -m pytest -q tests/e2e --tb=short	Needs pytest-playwright==0.7.2 playwright==1.58.0 and python -m playwright install chromium. CI splits the full run into six shards.
Mobile quality gate	TZ=UTC python -m pytest -q tests/e2e/test_mobile_quality_gate.py --tb=short	Runs in CI when browser-facing files change.
Wiki documentation checks	see below	Needs a checkout of this Wiki.

CI routes each job by the paths a change touches, and a summary job fails closed when a routed job did not succeed.

• E2E shards: tests/e2e/shards.json assigns every E2E file to exactly one shard; python scripts/e2e_shards.py validate fails when a new test file is missing from it. CI validates only the latest attempt of each shard and checks the summed CPU time against a per-test budget from a measured baseline, normalized for the runner's speed.
• Browser installs: CI restores Chromium and its system packages from the Actions cache and installs them with scripts/install_playwright_chromium.sh, which only asks an Ubuntu mirror when the cache is cold. Every test job has a time limit.
• Workflow lint: workflow files are checked with actionlint.
• Visual review: for pull requests that touch the interface, scripts/visual_review.py renders every navigation view and settings section of the base branch and the pull request from the same demo data, in dark and light mode at desktop and phone size, and reports pixel and style differences in the job summary with screenshots as an artifact. It is informational and never fails the job.
• Template rules: tests/test_dashboard_templates_inline.py rejects inline event handlers and inline styles other than custom properties in every template, and tests/test_css_ownership.py keeps shared components in app/static/css/components.css and flags undefined theme tokens.

---

DOCSIS case fixtures

tests/fixtures/docsis_cases/ contains public-safe DOCSIS evidence fixtures used by tests/test_docsis_case_fixtures.py.

Fixtures are synthetic and must not contain private IP addresses, MAC addresses, serial numbers, provider account data, cookies, tokens, passwords, or location-specific hints.

Run the focused replay suite with:

python -m pytest tests/test_docsis_case_fixtures.py

Each JSON case may define:

• raw: redacted raw DOCSIS input accepted by app.analyzer.analyze().
• previous_raw: optional prior raw input for event/baseline replay.
• postprocess: optional collector-side post-processing to apply before assertions.
• checklist: optional evidence-checklist input for non-DOCSIS capability cases.
• expect: golden analyzer, event, or checklist expectations.

The bundled cases cover a counter-reset baseline, Generic Router "not applicable" evidence, null gaps, parse errors, a stable downstream with a degraded upstream, unsupported counters, and low-QAM DOCSIS 3.1 upstream.

Driver fixtures

Driver tests replay captured or synthetic modem payloads through the driver and its format profile. When you add a fixture:

• Keep only the fields the parser needs. Remove authentication material, session cookies, device identifiers, subscriber addresses, and wireless configuration.
• Do not commit raw HAR files or frontend source; describe what the fixture represents instead.
• Make the parser fail safely on missing, duplicated, or reordered rows, and test those cases explicitly.
• State what the fixture proves. Passing tests establish parser behavior, not universal firmware compatibility.

See Adding Modem Support for the driver contract.

PYUR FAST3896-15 channel fixture

tests/fixtures/pyur_fast3896/connection.json contains only allowlisted DOCSIS numeric/unit/modulation fields from an examined FAST3896-15_PYUR-RDK_83.2.4 connection response: 20 downstream 256-QAM channels, one OFDM channel, five upstream QAM channels, and 21 separate error-counter rows. It contains no authentication, session, device identifiers, subscriber addresses, or wireless configuration.

id joins counter rows to downstream rows; ChannelID is the channel identity. Tests permute the counter order and exercise missing and duplicate join IDs. All auth/session/device test inputs elsewhere are synthetic. The original HAR and frontend source are intentionally not included. Automated tests alone do not establish universal firmware compatibility. In #865, the reporter confirmed successful login and data collection with image sha-208d8b4, followed by nominal operation after one day. Relevant tests: tests/drivers/test_pyur_format.py and tests/test_pyur_fast3896_driver.py. See PYUR FAST3896-15.

---

Documentation checks against the Wiki

User and developer documentation lives in this Wiki; the repository keeps only its core files (README, security, contributing, support, code of conduct, code signing, trademarks, data contract, the pull request template, and the Windows packaging notes). tests/test_wiki_docs.py checks the repository against a real checkout of this Wiki:

• every github.com/itsDNNS/docsight/wiki/... link in the README, the core documents, docs/index.html, the issue templates, the application code, and the benchmark script points to an existing page and heading;
• the Desktop Preview link in the app points to the Windows Desktop Preview page;
• pages that replaced repository documents keep their essential content, for example the out-of-scope list of the Feature Matrix and the screenshot safety checklist of the Proof Pack;
• public positioning pages contain no private addresses or localhost values;
• repository files that Wiki pages embed or link (screenshots, the sample report, code paths) exist on the main branch.

Run them locally against a Wiki checkout:

git clone https://github.com/itsDNNS/docsight.wiki.git ../docsight.wiki
DOCSIGHT_WIKI_DIR=../docsight.wiki python -m pytest tests/test_wiki_docs.py

Without DOCSIGHT_WIKI_DIR, the tests are skipped locally with a hint. In CI they fail instead, so they can never be skipped silently. The docs job of the test workflow clones the published Wiki and runs them when documentation, the issue templates, app/web.py, the benchmark script, or the test itself change, and on the weekly scheduled run. The other Python lanes deselect them with -m "not wiki".

When you rename a Wiki page or heading, update incoming links in the repository in the same change, or keep the old page as a short pointer.

---

Synthetic dashboard trend benchmark

scripts/benchmark_dashboard_trends.py compares the dashboard trend data path between source trees.

Run the script from the changed checkout with its Python environment, passing any baseline and changed source directories. It imports each source in a fresh subprocess and runs that source's production create_app, runtime, storage, routes, templates, and frontend on a real loopback HTTP server with four Waitress threads. Install the repository's hashed Python requirements plus pytest-playwright==0.7.2 playwright==1.58.0, then install Chromium with python -m playwright install chromium.

.venv/bin/python scripts/benchmark_dashboard_trends.py \
  --repo /path/to/baseline --repo /path/to/changed \
  --seed-dir /tmp/docsight-signal-synthetic \
  --output /tmp/dashboard-trends-comparison.json \
  --runs 5 --browser-runs 3 --parallel 4 --modules both

Supply any baseline with --repo; no fixed commit is assumed. The seed directory is created only when absent. Existing inputs require the synthetic marker and matching database SHA-256 hashes. Never point it at a user data directory. Every measurement copies the same input databases into a new temporary directory, including fresh config/session state. No collectors are constructed or started; update checking is disabled, integration settings are not inherited, community module search paths are empty, and outbound socket connections are blocked. Browser requests outside the loopback server are blocked and reported. API data is never mocked or replaced.

The default seed has 92 days at five-minute intervals (26,496 snapshots), exactly 288 visible snapshots, eight recent speed tests, and 34,560 Connection Monitor samples for two synthetic targets. Shipped synthetic channel definitions pass through the real analyzer and cumulative-baseline builder, producing realistic signal_families, error_baseline, and error_counter_coverage objects. Their sizes and database hashes are recorded. --days changes the history length for scaling probes. Fixed rolling-window clocks in app.tz, the trend blueprint, and the browser keep the identical 288 snapshots visible even in later comparisons. Performance clocks are real. The Connection Monitor summary uses real time; its rolling trend window uses the fixed API clock.

--modules both measures enabled built-ins and an app without module registration or the optional Connection Monitor database. The shared snapshot database still contains speedtest tables in both cases: this preserves the generic API's existing behavior even when no speedtest module card is registered.

Each stage records an ordinary first run separately from the warm repetitions. This is not a controlled cold OS/SQLite-cache benchmark. Warm medians, minima, maxima, and standard deviations accompany raw runs. Measurements include:

• Storage calls, HTTP API times, decoded payload bytes, and row counts.
• Concurrent dashboard API cycles: a baseline's actual legacy requests run in parallel; updated cycles request signals and then the retained legacy API. All requests contend for the same four Waitress threads. Raw critical, legacy, and total times remain separate.
• Real Chromium full-page loads of each repository's own frontend, DOM/load completion, completion of all resource requests, request counts, decoded and transferred bytes, trend request start/TTFB/duration, and payload.
• First actual rendering of the Home signal trend curve: a probe observes real uPlot draws and checks opaque signal-colored pixels inside the plotting area, requiring at least two real data values. Blank canvases, axes, and translucent fills do not qualify. The next animation-frame callback records the completed draw's paint opportunity; this is a browser rendering milestone, not physical display scanout. The probe also reports curve point/pixel counts and active charts.
• Parallel real browser navigations with the frontend's actual requests, including legacy work, against the same server.

Chromium uses a 1440×1000 viewport, UTC, blocked service workers, and a disabled HTTP cache (Playwright request interception). There is no proxy or compression. The report records Python, platform, and Chromium versions. Use an otherwise idle machine for comparison, and do not run tests at the same time as authoritative measurements. A local smoke run validates the harness, not production latency.

Failures (including no detected curve) fail the command; inspect page errors and blocked outbound requests before interpreting timings. There are no hardware timing assertions in CI. Signal API < 100 ms warm median / < 50 kB at 288 points and query-to-curve < 1 s are evaluation targets, not asserted results. The retained legacy request is still expensive and its total/parallel overhead is not fixed by the compact route; see Dashboard signal data path.