Skip to content

Architecture

Jump to bottom
Dennis Braun edited this page Oct 8, 2026 · 4 revisions

Architecture

This page describes the technical architecture of DOCSight. For the data ownership and sharing boundaries, see the data contract in the repository.

Overview

DOCSight is built around a modular collector pattern that separates data collection, analysis, storage, and presentation into independent, testable components. app.signal_health_view projects explicit analysis, threshold, and tariff inputs into Home presentation; app.line_status and app.channel_matrix build the Home line status and the Channels status matrix from the analyzed channel lists.

Reverse-proxy mount contract

DOCSight implements a generic proxy-stripped mount contract in app/base_path.py. An operator can provide an explicit BASE_PATH, or can opt into an exact trusted X-Forwarded-Prefix hop count with REVERSE_PROXY_PREFIX. DOCSight validates all selected sources, sets the request SCRIPT_NAME, and leaves the already-stripped upstream PATH_INFO unchanged. Browser URLs, cookies, the manifest, and the service worker then remain scoped to that effective same-origin mount. Operator-facing setup is described in Reverse Proxy.

This contract is independent of Home Assistant. DOCSight does not call the Supervisor API, query Supervisor app information, or consume Home Assistant identity or authentication headers. A Home Assistant wrapper may query Supervisor app information and set an explicit BASE_PATH before starting DOCSight, but the wrapper owns that platform integration. DOCSight continues to use its own authentication and generic reverse-proxy contract.

Application factory

app.app_factory.create_app() is the single production construction path for the Flask application. Importing app.web defines the core HTTP surface and its accessor facade, but does not construct an application. Factory creation first resolves one immutable registration plan containing core routes, core blueprints, built-in modules, and enabled community modules. Blueprint callbacks are preflighted on an isolated application, and URL/config ownership collisions are validated before the target application or process-wide module catalogs are changed. The validated plan then applies its HTTP, catalog, and template contributions once, followed by base-path handling and reverse-proxy handling as the outer WSGI layer.

app.registration is the only productive Flask registrar. It owns blueprint and direct-rule application, endpoint/blueprint/route-method collision checks, and the canonical registration manifest. The manifest contains stable public identifiers only; its SHA-256 fingerprint is logged at startup and stored in the application extensions. Filesystem locations, callables, and configuration or secret names and values are excluded. app.module_registry owns manifest discovery, app.module_config_registry owns configuration-schema preflight, app.module_contributions owns contribution-specific resolution and preflight, and app.module_loader orchestrates ownership and rejection policy behind its compatible public facade.

Runtime ownership

Internal routes and modules use app.runtime.current_runtime() to access the DocsightRuntime at app.extensions["docsight"], which owns per-application configuration, storage, auth state, rate limiter, update checker, module loader, collectors, derived storage, and lock-protected state. get_state() retains its lock-protected snapshot. Collectors receive that runtime through existing web= parameters; its collector-thread duck contract provides update_state(), clear_speedtest_latest(), get_state(), get_module_loader(), and the read-only _state snapshot without requiring a Flask request or application context.

  • app.web_locale owns request and setup language.
  • app.tz owns pure date validation and timestamp localization with explicit time zone input.
  • app.theme_registry owns the built-in theme collection and active-theme resolution; it loads the built-in themes through app.theme_catalog from the validated app/catalogs/builtin_themes.json. app.theme_contrast checks theme text and status color contrast.
  • app.version owns version lookup.
  • app.web_auth owns authentication policy and bootstrap; lower-level owners never import app.web.

app.web retains HTTP, Jinja, setup, settings, glossary, and desktop adapters plus community compatibility: get_storage, get_config_manager, get_collectors, get_modem_collector, get_module_loader, get_on_config_changed, get_state, get_last_manual_poll, set_last_manual_poll, update_state, clear_speedtest_latest, reset_modem_state, APP_VERSION, and require_auth. Existing community imports remain supported.

Module schema registries in app.config, analyzer threshold selection, translation catalogs, the built-in driver catalog, theme registries, and dynamic Python imports are process-wide. Community driver registrations and hints belong to each application runtime. The registrar applies resolved driver contributions only after complete preflight; HTTP routes and polling use that runtime registry. Per-application configuration values, storage paths, signing and authentication state, rate-limit buckets, update caches, module loader, templates, collectors, and derived storages remain isolated.

System architecture

┌────────────────────────────────────────────────────────────────────┐
│                         Main process (app/main.py)                  │
│                                                                      │
│   discover_collectors()                                              │
│     core:    Modem │ Demo │ Segment utilization (FRITZ!Box)          │
│     modules: Speedtest │ BQM │ BNetzA watcher │ Backup │ Weather │   │
│              Connection Monitor │ community collectors               │
│                                   │                                  │
│                                   ▼                                  │
│                      ┌─────────────────────────┐                     │
│                      │  Polling loop (1s tick) │                     │
│                      │  ThreadPoolExecutor:    │                     │
│                      │  submit due collectors  │                     │
│                      │  that are not in flight │                     │
│                      └────────────┬────────────┘                     │
└───────────────────────────────────┼──────────────────────────────────┘
                                    │
         ┌──────────────────────────┴──────────────────────────┐
         ▼                                                      ▼
  ┌─────────────┐                                      ┌──────────────┐
  │  Analyzer   │                                      │    Event     │
  │ DOCSIS data │                                      │   Detector   │
  │ → health    │                                      │   anomaly    │
  │ assessment  │                                      │   detection  │
  └──────┬──────┘                                      └──────┬───────┘
         └────────────────────┬───────────────────────────────┘
                              ▼
                  ┌────────────────────────┐
                  │  SQLite storage        │
                  │  snapshots, events,    │
                  │  speedtest cache,      │
                  │  journal, modules      │
                  └───────────┬────────────┘
         ┌────────────────────┼────────────────────┬───────────────┐
         ▼                    ▼                    ▼               ▼
  ┌───────────┐      ┌──────────────┐    ┌───────────────┐  ┌─────────────┐
  │   MQTT    │      │  Flask web   │    │  PDF reports  │  │ Notifier    │
  │ publisher │      │  UI + REST   │    │  (fpdf2)      │  │ webhook,    │
  │ (Home     │      │  API, PWA    │    │  complaint    │  │ Apprise,    │
  │ Assistant)│      │              │    │  letters      │  │ Web Push    │
  └───────────┘      └──────────────┘    └───────────────┘  └─────────────┘

Collector pattern

Base collector class

All data collectors inherit from app/collectors/base.py:

class Collector:
    """Base class for data collectors."""

    MAX_PENALTY_SECONDS = 3600  # 1 hour max backoff
    PENALTY_RESET_HOURS = 24    # auto-reset after 24h idle

    def __init__(self, poll_interval_seconds: int): ...

    @property
    def name(self) -> str:
        """Unique identifier; subclasses must override."""

    def collect(self) -> CollectorResult:
        """Run one collection cycle; subclasses must override."""

    def is_enabled(self) -> bool: ...   # defaults to True
    def should_poll(self) -> bool:
        """True if the poll interval plus any penalty has elapsed."""
    def record_success(self): ...   # reset penalty counter
    def record_skip(self): ...      # advance the timestamp without a failure
    def record_failure(self): ...   # increment penalty counter
    def get_status(self) -> CollectorStatus: ...

Parallel execution

Collectors run in parallel through concurrent.futures.ThreadPoolExecutor with one worker per collector. A blocking external call, for example a Speedtest Tracker timeout, never delays the local modem poll. The loop never submits a collector that is still in flight and logs a warning once when a run takes longer than 120 seconds.

# app/main.py (simplified)
executor = ThreadPoolExecutor(max_workers=len(collectors), thread_name_prefix="collector")
while not stop_event.is_set():
    for collector in collectors:
        if already_in_flight(collector) or not collector.is_enabled() or not collector.should_poll():
            continue
        in_flight[executor.submit(_run_collector, collector)] = collector
    process_finished_runs(in_flight)   # record_success() or record_failure()
    stop_event.wait(1)

_run_collector() holds the collector's _collect_lock without blocking, so a manual poll and an automatic poll never run collect() at the same time.

Thread safety

Lock Location Protects
_lock Collector base Scheduling state (_last_poll, _consecutive_failures)
_collect_lock Collector base Prevents concurrent collect() (manual poll vs. automatic poll)
RuntimeState._lock runtime.py Per-application dashboard state (written by collectors, read by Flask)
LoginRateLimiter._lock runtime.py Per-application login failure buckets
UpdateChecker._lock runtime.py Per-application update-check state
_lock EventDetector Previous snapshot comparison (_prev)

SQLite uses WAL mode (PRAGMA journal_mode=WAL) for concurrent reads during writes.

Fail-safe mechanism

Failure #1:   30s  penalty
Failure #2:   60s  penalty
Failure #3:  120s  penalty
Failure #4:  240s  penalty
Failure #5:  480s  penalty
Failure #6:  960s  penalty
Failure #7: 1920s  penalty
Failure #8: 3600s  penalty (cap reached)
Failure #9: 3600s  (stays at cap)
...
After 24h idle: auto-reset to 0

This prevents hammering external services during outages, with automatic recovery.

Implemented collectors

Collector Location Default interval Enabled when
Modem app/collectors/modem.py poll_interval (900 s) A modem is configured
Demo app/collectors/demo.py poll_interval (900 s) DEMO_MODE=true or the first-run demo action; replaces all other collectors
Segment utilization app/collectors/segment_utilization.py 300 s modem_type is fritzbox and segment utilization is enabled
Speedtest app/modules/speedtest/collector.py 300 s Speedtest Tracker is configured
BQM app/modules/bqm/collector.py 86400 s, gated by bqm_collect_time plus a spread offset A BQM share URL is configured
BNetzA watcher app/modules/bnetz/collector.py 300 s BNETZ_WATCH_ENABLED=true
Backup app/modules/backup/collector.py backup_interval_hours (24 h) Scheduled backups are enabled with a backup path
Weather app/modules/weather/collector.py 3600 s Coordinates are configured
Connection Monitor app/modules/connection_monitor/collector.py 1 s loop; each target is probed at the configured probe interval The Connection Monitor is enabled

Core collectors are created in app/collectors/__init__.py. Module collectors are contributed through the module manifest ("collector": "collector.py:ClassName") and are constructed with config_mgr, storage, and web keyword arguments. Community module collectors receive a read-only configuration proxy that hides core secrets and other modules' secrets.

ModemCollector

Fetches DOCSIS channel data from the cable modem or router through a pluggable driver (see Driver architecture).

Driver.get_docsis_data()
  → analyzer.analyze()
    → event_detector.check()
      → storage.save_snapshot()
        → mqtt_pub.publish_data()
          → runtime.update_state()

DemoCollector

Generates realistic DOCSIS data for evaluation without a real modem, from app/fixtures/demo_channels.json with per-poll random variation, and runs it through the real analyzer and event detector.

  • 25 downstream channels (24 × DOCSIS 3.0 SC-QAM + 1 × DOCSIS 3.1 OFDM) and 5 upstream channels (4 × SC-QAM + 1 × OFDMA)
  • Per-poll variation: ±0.3 dBmV power, ±0.5 dB SNR, slowly accumulating errors
  • Seeds 270 days of history at 15-minute intervals, plus events (derived by the event detector from that history, plus three device events), 12 journal entries in 3 incident groups, speed tests, 30 days of BQM graphs, 9 BNetzA campaigns, 270 days of weather, 7 days of Connection Monitor samples and traceroutes, and 7 days of one-minute FRITZ!Box segment utilization samples
  • All demo rows are marked with is_demo=1 for clean separation from user data, and purge-before-seed prevents duplicates after a rebuild
  • Device info: "Demo Router", connection 250/40 Mbit/s cable
  • Demo discovery selects only DemoCollector; optional module collectors are not started while Demo Mode is active

Users can switch from demo to live mode through Settings or POST /api/demo/migrate. This purges all is_demo=1 rows while preserving user-created entries, disables demo mode, and restarts polling for real modem data. See Demo Mode.

BQMCollector

Downloads ThinkBroadband BQM evidence for the BQM view.

  • Primary output: CSV Yesterday data parsed into bqm_data for interactive uPlot charts
  • Legacy output: PNG graph images saved to bqm_graphs when a PNG share URL is configured
  • Saving a new CSV share URL triggers an immediate authenticated initial fetch through POST /api/bqm/fetch-now.
  • Daily polling records collection metadata in bqm_meta (last_success_target_date, mode, rows, timestamp), so a restart after a successful collection does not fetch the same target date again.
  • DOCSight does not silently invent multi-day backfill. Longer gaps should be filled with the BQM CSV bulk import.
  • The UI labels cached PNG fallback separately from live refresh, so cached evidence is not presented as live freshness.

BNetzA watcher

Auto-imports BNetzA measurement protocols from a watched directory (BNETZ_WATCH_DIR, default /data/bnetz).

Scan watch_dir for new .pdf/.csv files (not in .imported marker)
  -> PDF: parse_bnetz_pdf(bytes) -> storage.save_bnetz_measurement(parsed, pdf_bytes, source="watcher")
  -> CSV: parse_bnetz_csv(content) -> storage.save_bnetz_measurement(parsed, None, source="csv_import")
    -> Move processed files to processed/ subdirectory
      -> Append filenames to .imported marker

PDF parsing uses app/modules/bnetz/parser.py (official BNetzA Messprotokoll format); CSV parsing uses app/modules/bnetz/csv_parser.py (semicolon-separated, German locale numbers). Partial failures continue with the remaining files. See Example Compose Stacks for sidecar examples.

BackupCollector

Creates scheduled backups of all DOCSight data.

create_backup_to_file(data_dir, dest_dir)
  -> VACUUM INTO for an atomic, consistent DB copy
    -> Strip demo data (is_demo=1) from the copy
      -> Pack into .tar.gz with backup_meta.json
        -> cleanup_old_backups(dest_dir, keep=retention)

Core logic lives in app/modules/backup/backup.py without a Flask dependency. See Backup & Restore.

Notifier

app/notifier.py routes event notifications to external channels. The event detector decides what happened; the notifier applies severity filtering, per-event toggles, and cooldowns and then delivers through WebhookChannel (direct webhook, Discord formatting), AppriseChannel (optional Apprise API sidecar), and WebPushChannel (subscribed browsers). See Notifications.

Driver architecture

Modem drivers live in app/drivers/ and implement the ModemDriver base class. A driver owns device orchestration: network requests, endpoint and firmware selection, authentication, sessions, retries, TLS, and cryptography. It fetches a raw modem payload and delegates normalization to one of the pure, explicit profiles in app/drivers/formats/.

The dependency direction is one-way:

collector -> concrete driver -> named format profile -> parser primitives/types
                              -> transport helpers (driver code only)

Format modules never depend on a concrete driver, session, request client, Flask, clocks, randomness, or cryptography. They return an immutable ParseResult(value, diagnostics). Diagnostics contain only the finite safe fields family, profile, code, direction, row, index, and field. Public driver methods unwrap the result and retain the established DocsisData/channel-list contracts.

app/drivers/format_compat.py is a finite compatibility boundary for legacy warning messages. app/drivers/arris_html.py remains an import-compatible shim for the established bonded 8/7-column parser. Existing private parser seams that are covered by integrations remain one-statement delegations; format grammar is not implemented in concrete drivers.

The base interface is:

class ModemDriver(ABC):
    def __init__(self, url: str, user: str, password: str): ...

    @abstractmethod
    def login(self) -> None: ...

    @abstractmethod
    def get_docsis_data(self) -> dict: ...

    @abstractmethod
    def get_device_info(self) -> dict: ...

    @abstractmethod
    def get_connection_info(self) -> dict: ...

Driver-to-profile matrix

Every registered concrete class exposes a non-empty immutable FORMAT_FAMILIES tuple. Registry aliases appear together in the first column. There are 23 keys, 22 concrete classes, and 24 explicit profiles.

Registry key(s) Concrete class Format profile(s) Cohesive module / entrypoint
cgm4981 CGM4981Driver cgm4981_columnar_html html_columnar.parse_cgm4981_columnar_html
ch7465, ch7465_play CH7465Driver ch7465_xml xml_payloads.parse_ch7465_xml
cm1000 CM1000Driver cm1000_html_table, cm1000_javascript html_rows.parse_cm1000_html_table; javascript.parse_cm1000_javascript
cm3000 CM3000Driver cm3000_javascript javascript.parse_cm3000_javascript
cm3500 CM3500Driver cm3500_html html_rows.parse_cm3500_html
cm8200 CM8200Driver arris_html html_rows.parse_arris_html
f3896lg F3896LGDriver f3896lg_rest_json sagemcom.parse_f3896lg_rest_json
fritzbox FritzBoxDriver fritzbox_data_lua fritzbox.parse_fritzbox_data_lua
generic GenericDriver generic_no_docsis boundaries.parse_generic_no_docsis
hitron HitronDriver hitron_coda56_json hitron.parse_hitron_coda56_json
hitron_coda_4680 HitronCoda4680Driver hitron_coda4680_json hitron.parse_hitron_coda4680_json
pyur_fast3896 (main only) PyurFast3896Driver pyur_api_v1 pyur.parse_pyur_api_v1
sagemcom SagemcomDriver sagemcom_xmo_json sagemcom.parse_sagemcom_xmo_json
sb6141 SB6141Driver sb6141_transposed_html html_transposed.parse_sb6141_transposed_html
sb6183 SB6183Driver sb6183_html html_rows.parse_sb6183_html
sb6190 SB6190Driver sb6190_html html_rows.parse_sb6190_html
sb8200_cbn SB8200CBNDriver sb8200_cbn_xml xml_payloads.parse_sb8200_cbn_xml
sercom_dm1000 SercomDM1000Driver sercom_dm1000_json sercom.parse_sercom_dm1000_json
surfboard SurfboardDriver arris_html, surfboard_hnap html_rows.parse_arris_html; surfboard.parse_surfboard_hnap
tc4400 TC4400Driver tc4400_html html_rows.parse_tc4400_html
ultrahub7 UltraHub7Driver ultrahub7_json vodafone.parse_ultrahub7_json
vodafone_station VodafoneStationDriver vodafone_station_cga_json, vodafone_station_tg_embedded_json vodafone.parse_vodafone_station_cga_json; vodafone.parse_vodafone_station_tg_embedded_json

Driver registry

Drivers are loaded by name through the registry in app/drivers/registry.py. The registry maps type strings to fully qualified class paths for lazy importing. ch7465 and ch7465_play intentionally resolve to the same class; the registry applies the Play firmware selection without creating a second parser profile.

Each built-in registration carries UI hints such as the default URL, default user, and whether a username or credentials are required. Hints also carry a manufacturer and an optional region; get_setup_catalog() uses them to group the setup wizard's searchable modem list by manufacturer. The generic router driver is offered by the connection-type step instead of the modem list. Community drivers cannot declare manufacturer or region hints, so they appear under Other unless they override a built-in key without hints of their own. See Driver Modules.

Experimental PYUR FAST3896-15

pyur_fast3896 owns the /api/v1 session and form login; formats.pyur normalizes connection data and allowlists device model, running firmware (falling back to main firmware), and uptime. This is a separate protocol from Sagemcom XMO and Liberty Global REST. It was developed from a capture; the reporter has since confirmed successful login, data collection, and nominal operation after one day with image sha-208d8b4 in #865. See setup and limitations.

The examined frontend computes e = SHA512-crypt(password, salt), then f = SHA512(username + ":" + nonce + ":" + e[3:]) and auth_key = SHA512(f + ":0:" + cnonce) using hex SHA512 digests. Removing $6$ preserves salt$digest. pyur_auth implements only the fixed 5000-round SHA512-crypt algorithm using hashlib, with no extra dependencies, stdlib crypt, libc calls, or executable tools. It is checked against the published SHA-crypt vector and hardcoded synthetic vectors independently generated with Passlib's builtin backend. The actual firmware helper was not captured; these tests establish the standard algorithm, not hardware compatibility.

The driver accepts raw salts of 1–16 characters ([./A-Za-z0-9]), bounds the numeric nonce to 32 characters, and generates a 19-digit zero-padded random client nonce. Salt settings cannot select expensive rounds. Password input is limited to 1024 UTF-8 bytes. Login verifies /authenticated after both empty 201 responses. Cookies persist within one requests.Session; each request selects the current matching CSRF cookie. Requests disable redirects and ambient netrc/proxy settings, use 5-second connect / 15-second read timeouts, and limit JSON bodies to 1 MiB. A data request receiving 401 can reauthenticate and retry once; other HTTP failures and invalid JSON fail without relogin. Exceptions contain fixed diagnostics, never response text or auth material.

The connection profile requires one object with downstream/upstream arrays. It uses explicit frequency units, finite measurements, and exact integer counters without 32-bit wrapping. Separate error rows join on unique id values on both sides; channel identity comes from ChannelID. Missing or ambiguous counters stay null. No LockStatus, QAM order, OFDM profile, multiplex mode, or service rate is inferred from the capture. Optional symbol rate appears only when supplied. /docsis-info/network_parameters is not queried because it includes subscriber addresses and is unnecessary here. The test fixture is described in Developer Testing.

Vodafone Station auto-detection

The Vodafone Station driver supports two hardware variants with different auth flows:

  • CGA (CGA6444VF / CGA4322DE): double PBKDF2-SHA256 + JSON REST API
  • TG (TG3442DE): AES-CCM encrypted credentials + HTML/AJAX endpoints

The variant is auto-detected on first login: CGA is tried first, then TG on failure.

Modules, themes, and settings

Extension module state

Installed non-theme modules are discovered by ModuleLoader and persisted through the comma-separated disabled_modules config key. A module switch in the Settings Extensions panel saves at once: it queues an instant save that sends only the changed module records through POST /api/modules/batch and leaves other pending form edits unsaved. Built-in themes are registered from the theme catalog (app/catalogs/builtin_themes.json, validated by app/theme_catalog.py with schema version 1). Community theme data is canonicalized on load: a deprecated alias such as --text-primary is carried over to its canonical token unless the theme sets that token too. Installed community themes still use their manifest and theme.json package format and remain on the dedicated theme APIs because preview and active-theme handling are separate flows. See Themes.

Module manifests can declare module-owned config defaults through the top-level config object. Normal module config remains plain local configuration. A community manifest may opt specific declared string-default keys into write-only secret handling with config_secrets; the value must be a list of unique strings, every listed key must exist in that manifest's config, and each corresponding default must itself be a string so encrypted values never enter scalar coercion paths. The pure-stdlib validator in app/manifest_contract.py is the authoritative manifest capability contract used by the runtime loader and can also validate external catalogs without importing Flask.

Before any enabled module loads, ModuleLoader resolves secret ownership from every discovered community manifest, including disabled modules, as fail-closed safety metadata. Core secret, hash-backed, private, and core configuration keys cannot be claimed. A secret key must also be exclusive to its declaring module's config; claims that overlap another module's plain or secret config fail every affected module and grant no owner. Normal community configuration keys also require one unambiguous owner. No ownership or classification is applied until the complete registration plan validates. The installer performs the same ownership evaluation against installed modules and rejects a conflicting package before it can be persisted. Valid module secrets are encrypted by ConfigManager, masked in Settings responses, and exposed through the community config proxy only to their owning module; other module secrets and all core protected values are removed.

Module source order is stable: the explicit built-in directory registry, the built-in threshold and theme registries, then configured community search paths and lexicographically sorted directories within each path. Duplicate community IDs retain the first source for manifest-v1 compatibility and skip later sources deterministically. Disabled and rejected modules remain visible as metadata but apply no routes, static mounts, templates, config defaults, i18n, active themes, thresholds, collectors, or publishers. As a metadata-only exception, a disabled community theme's declared theme.json is resolved and validated through the same safe contribution path so Appearance can render its preview. Invalid or unsafe preview data rejects that module, and preview data never creates a registration-plan contribution. A built-in planning failure aborts application construction; a community failure rejects only that module.

Threshold profiles are mutually exclusive. The bundled VF/KD profile is registered from the static analyzer profile registry in app/threshold_profiles.py; installed community threshold profiles still use the module manifest thresholds contribution. The UI renders threshold profiles as a single radio group, and the batch API validates the invariant server-side so exactly one threshold profile remains active after save.

Settings scripts

Settings scripts under app/static/js/settings/ have explicit component owners: navigation, search, tokens, connections, notifications, backups, themes, smart capture, and the module catalog. settings.js composes these owners and exports the handlers used by templates and module scripts. Bootstrap validation and shared utilities load first; versioned component assets load before the entry point and remain covered by the service worker's runtime cache.

form-state.js is the pure comparison owner. Records use tuple identities that distinguish hidden inputs, checkboxes, and repeated fields. One baseline supplies both full and manual projections. Secret records contain edit versions only, including initially unsaved secrets. form.js owns DOM capture, the separate API payload adapter, and the FIFO save queue. Manual submits save the entire form, captured when each queued job starts. Instant controls (module switches, controls marked data-instant such as the Extensions feature toggles and the font choice, and the dark mode switch) save only their own changed records and acknowledge exactly those, so manual edits stay pending. Config and module requests acknowledge their sent records independently: a failed module batch leaves config confirmed and modules pending for retry. Later edits remain dirty, including secret reedits; confirmed secrets are cleared only if their edit version still matches. Instant saves stay silent, with a retry footer on failure. Language/timezone reloads require confirmed values, no queued saves, and no unsaved changes when the reload timer runs. Component callbacks use this single save owner and unsaved guard; no component maintains another form baseline.

Settings receives the active module-secret and saved-secret key sets from the server, so installed templates from before the explicit field-marker contract still preserve masked values safely during a coordinated catalog rollout. Current templates also represent saved secrets locally with empty password inputs plus explicit data-config-secret and data-saved-secret metadata. Untouched saved fields submit the standard mask, which ConfigManager treats as preserve-existing, while edited fields submit the replacement value. Saved plaintext is never rendered into HTML.

Integer settings are validated when saved: ConfigManager.save() rejects values such as 1500.5 for integer keys with <key> must be a whole number, and the config API returns that as HTTP 400. Previously stored values that cannot be read as integers fall back to the key's default and are logged once without the key name.

Built-in manifests may separately mark declared config keys with the built-in-only configPrivate field. Those values are encrypted at rest but remain displayable and editable private metadata, so configPrivate is not a password/secret mechanism and its semantics are unchanged.

German outage-compensation assistant

docsight.de_tkg_compensation is a default-enabled built-in analysis module. It has no collector, publisher, or remote service. Its rules.py, rules_data.py, and letter.py layers are deterministic and Flask-free; authenticated routes adapt those layers to one module-owned claim table. The module calculates only user-confirmed local complete-outage calendar days. Outage dates are entered by the user; the assistant does not infer service outages from ping timeouts or open journal incidents.

For § 58(3) day counting, the fault-report receipt date is day 0. Calculation can first include the third local calendar day after receipt (day index 3), and only when the user confirms that entire calendar day as a complete outage day.

Missed service or installation appointments under § 58(4) are an independent claim basis and do not require an outage window or outage assertions. Manually entered timestamps use DOCSight's configured timezone. Fault-report receipt, restoration, and each complete-outage day remain explicit user facts.

Reports, Evidence Journey, and Incident Journal provide optional evidence exports. Links are offered only when the corresponding module is enabled. The manual claim, calculation, German text letter, copy action, and .txt export remain available when every support module is disabled. Reports longer than 90 days are linked as deterministic report-sized chunks; this technical PDF boundary never limits or truncates the claim calculation.

The four-step flow ends with the editable letter and its copy/download actions. The authenticated context endpoint supplies the configured local date and optional report customer defaults. Saved drafts and their existing status API remain supported; the wizard does not track whether a claim has been sent.

The ruleset ships with the application release and records jurisdiction, version, review date, and source URLs. Percentage results use integer-cent outputs and documented Decimal ROUND_HALF_UP sub-cent rounding. That rounding mode is a product calculation rule, not represented as statutory wording. Persisted rule versions are resolved through the ruleset registry; unsupported versions fail explicitly. Upgrade migration to a current rules version clears any previously generated letter atomically. See German TKG Compensation.

Data flow

Modem data collection

┌──────────────────────────────────────────────────────────────┐
│ 1. Polling loop (main.py)                                    │
│    modem collector is due → submitted to the executor        │
└────────────────────┬─────────────────────────────────────────┘
                     ▼
┌──────────────────────────────────────────────────────────────┐
│ 2. ModemCollector (collectors/modem.py)                      │
│    driver.login()                                            │
│    data = driver.get_docsis_data()                           │
│    analysis = analyzer.analyze(data)                         │
└────────────────────┬─────────────────────────────────────────┘
                     ▼
┌──────────────────────────────────────────────────────────────┐
│ 3. Analyzer (analyzer.py)                                    │
│    • Load active thresholds from the analyzer profile registry│
│    • Parse DS/US channels and channel families               │
│    • Assess power, SNR/MER, errors, modulation per channel   │
│    • Estimate SC-QAM gross channel capacity where supported  │
│    • Aggregate to overall health                             │
│      (good / tolerated / marginal / critical)                │
└────────────────────┬─────────────────────────────────────────┘
                     ▼
┌──────────────────────────────────────────────────────────────┐
│ 4. Event detector (event_detector.py)                        │
│    • Compare to the previous snapshot                        │
│    • Detect power shifts, SNR drops, modulation changes,     │
│      error spikes, channel changes, modem restarts           │
│    • Generate event records with severity                    │
└────────────────────┬─────────────────────────────────────────┘
                     ├──────────────┬──────────────┬────────────┐
                     ▼              ▼              ▼            ▼
             Storage (snapshot,  MQTT publish   Runtime state  Notifier
             events)             (HA)           (web)

Manual refresh

User clicks refresh
    │
    ▼
POST /api/poll
    ├─ Rate limit check (10s cooldown)
    ├─ Acquire _collect_lock (non-blocking)
    │   └─ If busy → 429 "Poll already in progress"
    ▼
result = modem_collector.collect()   (same flow as automatic polling)
    ▼
JSON { success: true, analysis: {...} }

Manual refresh uses the same collector as automatic polling, so the fail-safe behavior stays consistent.

Period aggregation layer

Stored DOCSIS snapshots are reduced through app.aggregation.aggregate_snapshot_period(). The entry point is pure: callers supply inclusive UTC bounds and an independent threshold context, and receive a deterministically ordered, JSON-serializable aggregate with coverage, provenance-aware diagnostics, averages, totals, worst values, and channel drivers. Equal timestamps use a stable content digest as a tiebreaker while preserving duplicate multiplicity.

Report PDFs, incident reports, complaint text, before/after comparison, and the evidence checklist share this contract. Storage queries remain in their route adapters. The evidence checklist loads snapshots once and requests its remaining timeline sources without modem data; BQM and Connection Monitor remain explicit external evidence adapters.

Threshold resolution remains analyzer-owned. The analyzer exposes copied plain threshold data and pure resolution functions; the aggregation layer may call only those resolution functions, and the analyzer has no dependency on aggregation.

Transition detection, latest-analysis Home displays, local-calendar modulation analysis, storage trend queries, and journal incident-date windows intentionally retain their distinct scopes and semantics.

Independent observations

Correlation, the AI export, and Gaming Quality treat each source as an independent observation. A speed test is not assigned the signal health of the nearest modem snapshot, and Gaming Quality does not mix DOCSIS health into a Speedtest-based score.

Storage layer

Database: SQLite (/data/docsis_history.db) with WAL mode for concurrent access. The Connection Monitor uses its own connection_monitor.db.

Timestamp convention: All timestamp, created_at, updated_at, and last_used_at columns store UTC with a Z suffix (YYYY-MM-DDTHH:MM:SSZ). Date-only columns (date, start_date, end_date) store calendar dates (YYYY-MM-DD) without time zone conversion. Many read APIs convert UTC values to the configured time zone before responding; see Configuration.

Schema (abridged):

-- DOCSIS signal snapshots
CREATE TABLE snapshots (
    id INTEGER PRIMARY KEY,
    timestamp TEXT NOT NULL,  -- UTC with Z suffix
    summary_json TEXT,
    ds_channels_json TEXT,
    us_channels_json TEXT,
    is_demo INTEGER NOT NULL DEFAULT 0
);

-- Speed test results (cached from Speedtest Tracker)
CREATE TABLE speedtest_results (
    id INTEGER PRIMARY KEY,
    timestamp TEXT NOT NULL,
    download_mbps REAL,
    upload_mbps REAL,
    ping_ms REAL,
    ...,
    is_demo INTEGER NOT NULL DEFAULT 0
);

-- Event log (anomaly detection)
CREATE TABLE events (
    id INTEGER PRIMARY KEY,
    timestamp TEXT NOT NULL,
    severity TEXT,     -- info|warning|critical
    event_type TEXT,   -- health_change, power_change, snr_change, ...
    message TEXT,
    acknowledged INTEGER,
    is_demo INTEGER NOT NULL DEFAULT 0
);

-- Incident containers (cases)
CREATE TABLE incidents (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    description TEXT,
    status TEXT DEFAULT 'open',  -- open|resolved|escalated
    start_date TEXT,
    end_date TEXT,
    icon TEXT,
    created_at TEXT,
    updated_at TEXT,
    is_demo INTEGER NOT NULL DEFAULT 0
);

-- Journal entries
CREATE TABLE journal_entries (
    id INTEGER PRIMARY KEY,
    date TEXT,
    title TEXT,
    description TEXT,
    icon TEXT,
    incident_id INTEGER,  -- FK to incidents.id, nullable
    created_at TEXT,
    updated_at TEXT,
    is_demo INTEGER NOT NULL DEFAULT 0
);

-- FRITZ!Box cable segment utilization
CREATE TABLE segment_utilization (
    timestamp TEXT,
    ds_total REAL, us_total REAL, ds_own REAL, us_own REAL,
    is_demo INTEGER NOT NULL DEFAULT 0   -- added by migration segment-0002
);

-- User-owned German TKG claim drafts (module-owned migration)
CREATE TABLE de_tkg_claim_drafts (
    id INTEGER PRIMARY KEY,
    status TEXT NOT NULL,              -- draft|completed
    window_from TEXT, window_to TEXT,  -- UTC Z timestamps
    origin TEXT NOT NULL,              -- manual|telemetry|incident
    fault_report_received_date TEXT, fault_report_channel TEXT,
    ticket_ref TEXT, restored_date TEXT,
    monthly_fee_cents INTEGER,
    confirmed_days_json TEXT NOT NULL,
    eligibility_json TEXT NOT NULL,
    prior_credit_json TEXT NOT NULL,
    letter_text TEXT,
    rules_version TEXT NOT NULL,
    created_at TEXT NOT NULL, updated_at TEXT NOT NULL,
    is_demo INTEGER NOT NULL DEFAULT 0
);

Further tables hold journal attachments, BQM data and graphs, BNetzA measurements, weather data, API tokens, Smart Capture executions, and PWA push subscriptions. Schema changes run as ordered, idempotent migrations in app/storage/migrations.py and module-owned migration files.

Retention: Snapshot retention is configurable through history_days; the default 0 keeps history without a time limit. Journal entries, incidents, and claim drafts are user data and are not removed by history cleanup.

Web layer

Framework: Flask, served by Waitress Port: 8765 (WEB_PORT) Auth: app.web_auth owns optional password protection (scrypt/pbkdf2), login CSRF and rate limits, the persisted session policy, and Bearer token authentication; app.web orchestrates the login and logout routes.

The browser UI is a single page (app/templates/index.html) whose views are switched by hash routes such as #channels?mode=status&range=1d or #evidence?from=…&to=…. All endpoints are listed in the API Reference.

Front-end structure

  • Navigation: one DOM for both layouts: a top bar with dropdown panels on desktop and a bottom bar with sheets below 1024 px. Groups (Signal, Connection, Cases, More) follow the disclosure pattern with keyboard support. app/templates/partials/topnav.html renders it for Home and Settings, together with the command palette (partials/command_palette.html, command-palette.js) and the notice center (app.maintainer_notices.get_notice_center(), notices.js). Setup and login share the brand bar, theme variables, and theme bootstrap partials.
  • Settings: app/templates/settings.html renders the section index from partials/settings_sections.html, which the command palette reuses, and places built-in module settings into topic sections; community module settings keep their own sections.
  • Home: app.line_status.build_line_status() turns the analyzed channels into one segment per channel and a callout for the worst deviation per direction, with target bands from the same threshold resolvers the analyzer uses. line-status.js loads a channel's last 24 hours on demand.
  • Channels status matrix: app.channel_matrix.build_channel_matrix() turns stored snapshots into 48 time cells per current channel, each holding the worst rating seen in that interval, and is served by GET /api/channel-status.
  • Pure browser data modules: correlation-data.js (lane layout, reachability buckets, CSV rows), event-log-data.js (day sections, collapsed runs, acknowledgement bookkeeping), and empty-state.js (shared empty states) contain no DOM access and are tested in Node.
  • Time and language contract: browser-contracts.js formats dates and numbers in the selected UI language and places timestamps in the configured DOCSight time zone, independent of the browser's own time zone.
  • View state and time windows: view-state.js reads and writes a view's range and filters in the hash (#trends?range=7d) with history.replaceState. window-shift.js moves a range window into the past and back and carries its end as wall-clock time in DOCSight's zone, the same end parameter the trend, correlation, and Connection Monitor APIs accept. snapshot-panel.js opens the snapshot behind a chart point from GET /api/snapshots/at.
  • Shared components and delegated actions: buttons, form fields, cards, badges, segmented controls, data tables, dialogs, toasts, and the save bar are styled once in app/static/css/components.css. Templates carry no inline on…= handlers; actions.js dispatches data-action, data-change-action, data-pill-action, and related attributes to named functions.

Dashboard signal data path

GET /api/trends/signal?range=1d is an authenticated, additive API. It accepts only normalized rolling ranges (1h, 6h, 1d, 2d, 3d, 7d, 30d, 90d), defaulting to 1d. Calendar date anchors are not used. Each chronological row contains timestamp, ds_power_avg, us_power_avg, and ds_snr_avg; missing values are null. All window rows are returned, including all-null rows. Timestamps use the same configured display-time localization as /api/trends. The Home signal trend chart retains its existing client-side time and all-null row filters.

Snapshot storage restricts the timestamp window in SQL and projects these three values without decoding full historical summaries in Python. The generic /api/trends contract and its 92-day error-counter unwrap anchors are unchanged.

signal-series.js shares one pending/completed request between the Home trend chart and signal sparklines per dashboard generation. Invalidating advances the generation; failed requests can retry and stale completions cannot populate the new cache. HTML refreshes also reject older responses. Refresh moves the actual chart node into the replacement view, retaining its canvas bitmap until valid data arrives. Replacement and theme rendering clean up the old chart, observer, and tooltip. Theme changes use rendered data without fetching or invalidating. Existing public refreshHeroChart() and refreshSparklines() calls request a fresh generation, coalescing calls in the same JavaScript turn.

sparklines.js still issues one separate generic /api/trends?range=1d request per generation, scheduled after signal rendering gets a paint opportunity (also after a signal failure), for error, module, and speed sparklines. It restores cached sparse rows after a DOM swap and replaces them when current results arrive. This request still incurs historical decoding and module work; it remains part of the full-dashboard and parallel measurements of the dashboard trend benchmark.

Configuration

File: /data/config.json (JSON, file mode 0600) Secrets: modem, MQTT, Speedtest Tracker, Apprise, VAPID, and module-declared secrets are encrypted at rest with a key in /data/.config_key; the admin password is stored only as a hash.

{
  "modem_type": "fritzbox",
  "modem_url": "http://192.168.178.1",
  "modem_user": "user",
  "modem_password": "<encrypted>",
  "poll_interval": 900,
  "history_days": 0,
  "mqtt_host": "localhost",
  "speedtest_tracker_url": "http://...",
  "speedtest_tls_insecure": false
}

Override: environment variables take precedence over config.json. See Configuration.

Demo Mode: set DEMO_MODE=true to run without a real modem. The DemoCollector replaces the ModemCollector and generates nine months of realistic simulated data. All demo-seeded rows are tagged with is_demo=1 so they can be cleanly purged when switching to live mode through POST /api/demo/migrate.

Extending DOCSight

New data sources should normally be built as modules instead of core code. A module collector is a Collector subclass that the manifest names in contributes.collector:

# my_module/collector.py
from app.collectors.base import Collector, CollectorResult


class MyCollector(Collector):
    name = "my_source"

    def __init__(self, config_mgr, storage, web, poll_interval=3600, **kwargs):
        super().__init__(poll_interval)
        self._config = config_mgr
        self._storage = storage

    def is_enabled(self) -> bool:
        return bool(self._config.get("my_source_url"))

    def collect(self) -> CollectorResult:
        try:
            data = self._fetch_data()
            return CollectorResult(source=self.name, data=data)
        except Exception as e:
            return CollectorResult(source=self.name, success=False, error=str(e))
{
  "contributes": {
    "collector": "collector.py:MyCollector",
    "settings": "templates/my_settings.html"
  },
  "config": {
    "my_source_url": ""
  }
}

The collector then polls at its interval, uses the fail-safe backoff, and reports its health through GET /api/collectors/status. Add tests next to the module or under tests/. Modem support uses driver modules instead; see Adding Modem Support.

Security

Secret storage:

  • Admin password: scrypt/pbkdf2 hash
  • Modem, MQTT, Speedtest Tracker, Apprise, and VAPID secrets: encrypted at rest
  • Config file and key files: owner-only file mode

Session management:

  • Flask sessions with HTTPOnly cookies
  • Rolling 30-day lifetime by default (SESSION_LIFETIME_DAYS, bounded to 1–365 days)
  • SameSite=Lax policy; reverse-proxy deployments retain Secure HTTPS-only cookies
  • Password-bound session markers invalidate existing logins when the effective admin password changes
  • Persistent session key in /data/.session_key
  • Keyed effective-auth state in /data/.auth_state detects changes across restarts

Rate limiting:

  • Login: 5 attempts per 15 minutes per IP
  • Manual poll: 10-second cooldown

Headers: HSTS, X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy, and a Content Security Policy.

Community code: community modules and drivers execute trusted Python in the DOCSight process; contribution rules are not a sandbox. Theme tokens are rendered only as well-formed CSS custom properties, and values that could close the declaration, rule, or style element are dropped.

See the repository's security policy for supported versions and vulnerability reporting.

Testing

Framework: pytest for Python, Node's test runner for pure browser modules, and Playwright for browser end-to-end tests.

python -m pytest tests/ -n auto --ignore=tests/e2e
npm test

Test categories include analyzer threshold logic, driver format profiles and fixtures, collectors and fail-safe behavior, storage and migrations, web APIs and auth, event detection, config validation, BNetzA parsing, demo mode, backup and restore, notifier delivery, i18n completeness, and static contracts. CI routes jobs by changed paths. See Developer Testing and Contributing.

Performance

  • Memory: about 50–100 MB typical, depending on history size
  • CPU: below 1% on average, with spikes during polling
  • Disk: about 1–5 MB per day, depending on the poll interval and enabled collectors
  • Network: modem queries plus the optional external APIs you configure

Speedtest results are cached locally, SQLite queries use indexes and SQL-side window filtering, and charts use uPlot.

Deployment

Recommended: Docker Compose

services:
  docsight:
    image: ghcr.io/itsdnns/docsight:stable
    container_name: docsight
    restart: unless-stopped
    ports:
      - "8765:8765"
    volumes:
      - docsight_data:/data
      - docsight_backup:/backup  # Optional: for scheduled backups
    environment:
      - TZ=Europe/Berlin

volumes:
  docsight_data:
  docsight_backup:

All data lives in the /data volume. Image tags are explained in Installation.

Troubleshooting

Check collector status:

curl http://localhost:8765/api/collectors/status | jq .

With an admin password set, add an API token as -H "Authorization: Bearer <token>".

Check logs:

docker logs docsight

Common issues:

  1. Modem collector failing: check the modem URL and credentials, verify the modem is reachable from the DOCSight network, and check /api/collectors/status for the penalty state.
  2. Speedtest not updating: verify the Speedtest Tracker URL and token, and check /api/collectors/status for errors.
  3. High penalty on a collector: fix the underlying issue (credentials, network). The penalty auto-resets after 24 hours idle; restarting the container resets it immediately.

The passive doctor summarizes local runtime, config, storage, and database state without contacting external services.

Related

Home | Quick Start | Configuration | API Reference | GitHub

Home

Getting Started

Monitoring

Documentation & Evidence

Integrations

System

Community

Reference

Development

Clone this wiki locally