Skip to content

Architecture

Jump to bottom
l0rdg3x edited this page Jun 16, 2026 · 7 revisions

Architecture

A component-level overview of OPNGMS for operators and contributors: the request path, the background worker, the optional log lake, the multi-tenant security model, and the technology stack — all grounded in the actual Compose files and backend/app/ code.

For deployment mechanics see Installation; for every environment variable see Configuration; for the security model in depth see Security.

Fleet overview

Contents

Component diagram

OPNGMS is a single-origin SPA + API, a background worker, and the two datastores they share. The frontend nginx serves the React bundle and reverse-proxies /api to the API container, so the browser only ever talks to one origin. The worker is the only component that reaches out to managed OPNsense appliances; the API never calls a firewall directly on the request path. The log lake is an opt-in overlay where managed devices push syslog over mTLS, independent of the API-pull telemetry path.

                         ┌─────────────────────────────────────────────────────────┐
                         │                      Docker host                         │
   Browser (SPA)         │                                                          │
        │   HTTPS         │   ┌──────────┐  /api    ┌──────────┐                     │
        ▼ ───────────────┼──▶│ frontend │ ───────▶ │   api    │ ──┐ asyncpg         │
  TLS terminator         │   │ (nginx)  │  proxy   │ (uvicorn │   │  (opngms_app,    │
  (your proxy / built-in │   │ SPA +    │          │  FastAPI)│   │   RLS-enforced)  │
   nginx/Caddy/Traefik;  │   │ /api     │          └────┬─────┘   ▼                  │
   sets X-Forwarded-Proto)│  │ proxy)   │               │      ┌──────┐              │
        ▲ ───────────────┼──┘          │               │      │  db  │ TimescaleDB  │
        │   HTTP(S)       │             enqueue jobs ───┼────▶ │(PG16 │  + RLS       │
        │                 │                  ▼          │      │ +RLS)│              │
        │                 │            ┌──────────┐     │      └──────┘              │
        │                 │            │  redis   │ ◀───┘         ▲                  │
        │                 │            │  (arq    │               │ asyncpg (owner,  │
        │                 │            │  queue)  │               │  RLS-exempt)     │
        │                 │            └────┬─────┘               │                  │
        │                 │                 │ poll / apply / report                  │
        │                 │            ┌────▼─────┐ ──────────────┘                  │
        │                 │            │  worker  │                                  │
        │                 │            │  (arq)   │ ── HTTPS (API key/secret) ──┐    │
        │                 │            └──────────┘                             │    │
        │                 └──────────────────────────────────────────────────┼────┘
        │                                                                      ▼
        │                                                      ┌───────────────────────────┐
        │                                                      │  Managed OPNsense fleet   │
        │                                                      │  (per-tenant firewalls)   │
        │                                                      └─────────────┬─────────────┘
        │                                                                    │ syslog/RFC 5424
        │             ── optional log lake overlay ──                        │ over mTLS :6514
        │                                                                    ▼
        │                                              ┌──────────┐    ┌──────────────┐
        └───── Logs page query (api → OpenSearch) ────▶│OpenSearch│◀───│  syslog-ng   │
                                                       │ (index   │    │ (mTLS recv,  │
                                                       │ per day) │    │ tenant from  │
                                                       └──────────┘    │ cert O/CN)   │
                                                                       └──────────────┘

Note: The api container's port 8000 is not published to the host — only the Compose network (the trusted nginx) can reach it. Likewise OpenSearch (port 9200) is internal-only; the trust boundary for the log lake is the mTLS device → syslog-ng:6514 hop.

Services

The core stack is six services (docker-compose.prod.yml). The optional log lake adds three more (docker-compose.logs.yml, or bundled into docker-compose.full.yml). Every service also receives TZ for log timestamps (data is always stored in UTC).

Service Image / base Responsibility Key dependencies
db timescale/timescaledb:2.17.2-pg16 Primary datastore: tenants, users, devices, metrics (Timescale hypertables), events, config snapshots/changes, reports, schedules. Enforces Row-Level Security. — (has healthcheck pg_isready)
redis redis:7 Job queue broker for arq — the worker's task queue and cron scheduler. — (has healthcheck redis-cli ping)
migrate ghcr.io/l0rdg3x/opngms-backend One-shot: runs alembic upgrade head as the DB owner, creating the schema, the opngms_app role, RLS policies, and grants. Exits 0 on success. db healthy
api ghcr.io/l0rdg3x/opngms-backend uvicorn app.main:app — the FastAPI HTTP API. Connects as the non-superuser opngms_app role, so RLS is enforced on every query. Health at GET /healthz. db healthy, redis healthy, migrate completed
worker ghcr.io/l0rdg3x/opngms-backend arq app.worker.WorkerSettings — background jobs and cron: device polling, event/config ingest, perimeter attacker rollup + retention purge, config apply, firmware actions, report generation/delivery, cert renewal, session reap, orphan sweeper, silent-tenant alerts. Connects as the owner (RLS-exempt) and scopes every query by explicit tenant_id. redis healthy, migrate completed
frontend ghcr.io/l0rdg3x/opngms-frontend nginx serving the React SPA bundle and reverse-proxying /api to the api container (single origin). TLS terminated here (built-in overlays) or upstream (base model). api
opensearch (log lake) opensearchproject/opensearch:2.17.1 Stores per-device, per-tenant device logs in daily indices (opngms-logs-*). Plain HTTP, security plugin disabled, internal-network only. —
syslog-bootstrap (log lake) ghcr.io/l0rdg3x/opngms-backend One-shot: python -m app.cli syslog-bootstrap — generates the CA + receiver cert/key into the shared certs volume. opensearch started, migrate completed
syslog-ng (log lake) balabit/syslog-ng:4.5.0 mTLS syslog receiver on port 6514 (RFC 5424). Verifies each device's client cert against the CA, derives tenant_id from the cert O= RDN and device_id from CN=, and ships parsed events to OpenSearch. syslog-bootstrap completed

Data flows

1. Login + MFA

Implemented in app/api/auth.py and app/api/mfa.py (services app/services/auth.py, app/services/mfa.py).

  1. Password step — POST /api/login authenticates email + password (AuthService.authenticate). On success it rotates any incoming session token (anti-fixation) and decides the session kind based on the user's MFA enrolment and the tenant-wide MFA policy (off / all / privileged).
  2. The server issues a session cookie + a CSRF cookie (both Secure, SameSite=Lax; the session is HttpOnly). If MFA is enrolled the cookie is a short-lived mfa_pending challenge session and the response is status: "mfa_required"; if policy requires MFA but the user hasn't enrolled, it's a mfa_setup session; otherwise a full-TTL session.
  3. TOTP step — POST /api/login/mfa verifies the 6-digit TOTP code (or a one-time recovery code) against the user's encrypted secret, with a last_used_step replay guard, and upgrades the mfa_pending session to a full session.
  4. Enrolment — POST /api/me/mfa/setup returns a TOTP secret (stored encrypted with MASTER_KEY); POST /api/me/mfa/confirm verifies the first code and returns one-time recovery codes (hashed at rest). Every step is recorded by AuditService.

A host-level break-glass CLI (python -m app.cli mfa-reset) clears MFA for a locked-out superadmin. See Security for session TTLs and cookie hardening.

2. Scheduled device polling and telemetry ingest

Driven by arq cron in app/worker.py; per-device work uses OpnsenseClient (app/connectors/opnsense/).

  1. Metrics (every minute) — enqueue_device_polls fans out a poll_device job per device. Each job builds an OpnsenseClient from the device's decrypted API key/secret, calls collect_and_store (telemetry → metrics hypertable + device status), then evaluate_alerts.
  2. Events (every N minutes) — enqueue_event_ingests → ingest_device_events pulls IDS (Suricata) alerts, DNS events, service / reliability events, and config-change audit events via the connector. A per-(device, source) cursor plus an ON CONFLICT DO NOTHING insert deduplicates, so re-polling never double-counts (app/services/ingest.py). The service source classifies the OPNsense system log into reliability events (reboots, service crashes/restarts, disk-FS warnings — a curated, fail-safe rule set; only recognized lines are stored, not the whole log); a new high-severity event raises a deduped alert at ingest. The config_audit source parses the OPNsense audit log (diagnostics/log/core/audit, searched server-side for the changed configuration line family so it isn't buried under configd.py noise) into config-change events, attributing each by channel from the request path and the actor IP — gui (a human in the WebGUI), system (console / script), and the api channel split by OPNGMS's auto-learned management IP into opngms (OPNGMS's own change, from the learned IP) vs api_external (an API client from any other IP). OPNGMS learns the management IP by correlating the box's api changes with its own apply ledger (an applied change within a few minutes of the logged API change) and only when the correlated changes agree on a single, unambiguous IP; the learning is conservative and self-correcting (a clean OPNGMS apply re-learns the right IP) and a no-op until learned. Anything OPNGMS did not make — api_external, gui, system — is the drift signal: stored at higher severity and raising a deduped alert. (This is the box's config log, distinct from OPNGMS's own Security write-ledger.) This API-pull event path is independent of the syslog log lake. The same per-device cron also runs the Security / Perimeter attacker rollup (ingest_perimeter) inside a SAVEPOINT — see flow 6.
  3. Config backup (daily) — enqueue_config_backups → backup_device_config stores an encrypted, gzip-compressed snapshot of config.xml, deduplicated on a canonical hash so an unchanged config doesn't create a new snapshot. These snapshots are the staleness baseline and rollback points for the apply pipeline.

3. Config-push / apply pipeline

The version-aware editor and curated templates (Configuration-Editor, Configuration) all funnel into one typed, audited, per-device-serialized apply pipeline.

  1. Propose — a change is created as a config_change row (create_change in app/services/config_push.py), capturing the current snapshot's canonical hash as baseline_hash. It originates either from a direct editor edit (POST /api/.../config/changes) or from applying a template/catalog item (POST /api/templates/.../apply), or as an ordered profile of several member changes.
  2. Schedule — the change moves to status="scheduled" and an apply_config_change (or apply_profile_changes) job is enqueued, optionally deferred to a future scheduled_at.
  3. Apply — the worker takes a Postgres transaction-scoped advisory lock keyed on device_id (per-device serialization), re-reads the live config.xml, and re-computes its canonical hash. Staleness guard: if the live config drifted from baseline_hash, the change is marked conflict and not applied — no clobber.
  4. Connector → OPNsense API — when LIVE_PUSH_ENABLED is set, it first persists a pre-apply snapshot (the rollback point), then dispatches by kind (apply_for_kind → apply_alias / apply_firewall_rule / apply_monit_test / catalog setting/grid appliers), which calls the OPNsense REST API (addItem/setItem/apply/reconfigure). Without LIVE_PUSH_ENABLED the same path runs as a dry-run (no mutation). The row ends applied, failed, or conflict, and the result is audited.
  5. Verify / refresh — after a successful apply the worker re-enqueues a backup_device_config so the next change's baseline reflects reality.
  6. Revert — an operator can revert an applied/failed change whose kind has a registered inverse builder (app/services/config_revert.py): it reconstructs the prior state from the pre-apply snapshot and pushes the inverse as a new change.
  7. Sweeper — a cron (sweep_orphaned_actions) re-enqueues scheduled config/firmware actions that were dropped by a lock-miss; after MAX_REENQUEUE_ATTEMPTS it marks the row failed and raises an action_orphaned alert, so nothing is silently lost.

4. Report generation and delivery

Per-tenant white-labelled PDF reports (app/services/reporting/, Reporting).

  1. Schedule fires (hourly cron) — enqueue_due_reports finds every enabled report_schedule whose next_run_at is due (UTC) and enqueues deliver_scheduled_report. Send now enqueues the same job with manual=True.
  2. Build — ReportService.build_report aggregates the tenant's metrics/events for the window, renders an HTML template (per-tenant title, logo, language, optional device scope), and converts it to PDF with WeasyPrint. The PDF is stored as a generated_report row; a build failure still advances the schedule's cadence so it won't re-fire hourly.
  3. Deliver — send_report_email_job loads the tenant's SMTP settings (credentials decrypted from MASTER_KEY), applies any per-tenant white-label From address, and emails the PDF, retrying up to MAX_SEND_ATTEMPTS with backoff. Every outcome (delivered / no-recipients / failed) is audited.

The worker runs as the RLS-exempt owner, so each repository (GeneratedReportRepository, ReportSettingsRepository) is constructed with an explicit tenant_id that scopes every query.

5. Log lake ingest and investigation

The opt-in overlay (app/services/log_search.py, Log-Lake).

  1. Ingest — a managed OPNsense device forwards syslog (RFC 5424) to syslog-ng:6514 over mTLS, presenting a per-device client cert signed by the CA. syslog-ng enforces peer-verify(required-trusted), derives tenant_id from the cert O= RDN and device_id from the CN, refuses any message it can't attribute to a tenant, and writes to a daily OpenSearch index (opngms-logs-YYYY.MM.DD).
  2. Investigate — the Logs page calls POST /api/.../logs/search. The API (not the browser) is the only OpenSearch client: it opens a Point-In-Time, builds a bool query whose tenant_id and time-range filters are always injected from the RBAC-verified path, and pages with search_after. A user can never query another tenant's logs because the tenant filter is server-side and non-removable.

6. Security / Perimeter attacker rollup

A per-tenant view of who is hammering the perimeter — failed admin logins and dropped firewall packets, grouped by attacker IP (app/services/perimeter.py). It rides the existing source-pluggable event ingest rather than introducing a new path, but deliberately stores a bounded rollup instead of per-packet rows.

  1. Two new connector capabilities — the version-aware OpnsenseClient gains auth_failures (POST diagnostics/log/core/audit, paged like the IDS query) and firewall_blocks (diagnostics/firewall/log), with parsers in the existing parsers module. Like every outbound call they go through the SSRF-guarded client; like every event source they advance a per-(device, source) IngestCursor watermark, so re-polling never re-counts.
  2. Bounded rollup, not events — rather than appending per-packet rows to the events hypertable (which would grow without bound for a box under attack), ingest_perimeter groups the watermarked batch by src_ip and UPSERTs a dedicated perimeter_attacker table keyed on (device_id, kind, src_ip) (kind ∈ {login_failed, firewall_block}). Each row carries count, first_seen, last_seen, and a detail JSONB (top ports / attempted usernames). The table therefore stays bounded to the set of distinct attacker IPs, and is decoupled from both the events table and the existing IDS attacker-countries view.
  3. Failure-isolated cron — ingest_perimeter runs inside the per-device events cron, wrapped in a SAVEPOINT, so a perimeter parse/UPSERT error rolls back only its own savepoint and can never roll back the IDS/DNS events ingest that shares the transaction. A separate daily purge_perimeter cron applies retention, keeping the rollup trimmed over time.
  4. Aggregation + API — ReportAggregator.perimeter_top ranks attacker IPs across the tenant's devices (SUM(count), MAX(last_seen), latest detail) and resolves each IP's country via the GeoIP provider. GET /api/tenants/{tid}/perimeter/attackers serves both the Overview cards and the dedicated Perimeter page. In the PDF report the two perimeter sections are standard report sections — failed_logins and firewall_blocks live in sections.py (SECTION_KEYS + BUILTIN_DEFAULTS, default on), and build_context gates each one tenant-wide by its resolved section toggle (enabled["failed_logins"] / enabled["firewall_blocks"]), exactly like attacker_countries. Toggle resolution follows the existing report_settings.sections / report_schedule.sections precedence.

Multi-tenancy model

Tenant isolation is enforced in the database with PostgreSQL Row-Level Security, not just in application code — a defence-in-depth design with two distinct database roles.

Role Used by RLS How tenant scope is applied
opngms_app (non-superuser, NOBYPASSRLS) api (request path) Enforced Every request sets app.current_tenant for the transaction (set_tenant_context); the RLS policy filters every tenant table
DB owner (POSTGRES_USER) migrate, worker Exempt (bypasses RLS) Repositories are constructed with an explicit tenant_id that scopes each query

How it works (app/core/db.py, rls.py, db_roles.py, deps.py):

  • Migration 0003 creates the opngms_app role as NOSUPERUSER NOBYPASSRLS. PostgreSQL superusers always bypass RLS even under FORCE, so the API must connect as this non-superuser role for RLS to bite.

  • Every tenant-scoped table (devices, metrics, alerts, events, perimeter_attacker, config_snapshots, config_changes, report_settings, generated_reports, firmware_actions, template_overrides, report_schedule, device_log_forwarding, revoked_syslog_certs) is ENABLE + FORCE ROW LEVEL SECURITY with a single tenant_isolation policy:

    tenant_id = NULLIF(current_setting('app.current_tenant', true), '')::uuid

  • The NULLIF makes the policy fail-closed: if app.current_tenant is unset or empty it resolves to NULL, which matches no rows. A request that never went through the tenant dependency sees nothing.

  • On the request path, the require_tenant / membership dependency resolves the caller's membership + role (RBAC), then calls set_tenant_context(session, tenant_id) for that transaction only before any tenant query runs.

  • The worker and migrate connect as the owner (RLS-exempt) because cron jobs operate across all tenants; they compensate by passing an explicit tenant_id into every repository and audit record.

  • The new perimeter_attacker table joins this set under the same shared tenant_isolation policy — no bespoke RLS. Whether the perimeter data reaches the PDF report is not an RLS concern but a plain, tenant-wide report-section toggle (failed_logins / firewall_blocks), resolved by build_context via the existing report_settings.sections / report_schedule.sections precedence — exactly like every other report section.

See Security for the full threat model, credential encryption (MASTER_KEY / Fernet), CSRF, and session handling.

Scope & limitations

OPNGMS manages each firewall through OPNsense's own API — it is a client of that API, not a replacement for it. Its reach is bounded accordingly:

  • Bounded by the OPNsense API. OPNGMS can only do what OPNsense exposes over its API. Anything the firewall offers no API for cannot be automated from here — for example there is no firmware rollback or full config.xml restore API, so neither is offered, and legacy non-MVC settings the API can't write are surfaced read-only in the live config.xml map. As OPNsense widens its API surface, OPNGMS can cover more, but it never reaches past the API.
  • Built from public, open-source OPNsense. The version-aware catalog — and the plugin coverage built on it — is generated from the public opnsense/core and opnsense/plugins source. Community (public) plugins are covered; proprietary / Business-only plugins that are not published on public GitHub are not in the generated catalog. A Business box is still managed for everything its API exposes, but those closed plugins have no generated configuration models. See Configuration-Editor → Coverage limits.

Tech stack

Layer Technology
Backend Python 3.14, FastAPI (uvicorn ASGI), SQLAlchemy async + Alembic migrations, arq task queue/cron, WeasyPrint (PDF), httpx (OPNsense + OpenSearch clients), cryptography/Fernet (secrets at rest), pyotp (TOTP)
Frontend React 19, Mantine v9, Vite, TypeScript — single-page app served by nginx
Datastores TimescaleDB (PostgreSQL 16 + Row-Level Security) for relational + time-series data; Redis 7 as the arq broker; OpenSearch 2.17 for the optional log lake
Infra Docker Compose; pre-built multi-arch images from GHCR (ghcr.io/l0rdg3x/opngms-{backend,frontend}, published only from semver release tags, pinned by OPNGMS_VERSION); TLS via your own proxy or a built-in nginx / Caddy / Traefik overlay; syslog-ng 4.5 as the mTLS log receiver

For setting up a local dev environment (Python venv, uvicorn --reload, Vite dev server) see Development.

OPNGMS Wiki

Home Screenshots

Deploy & operate

Understand & extend

Clone this wiki locally