Skip to content

Development

Jump to bottom
l0rdg3x edited this page Jul 29, 2026 · 2 revisions

Development

This guide sets up a local OPNGMS development environment — backend API and worker in a Python venv against Compose-run infra, and the React/Vite SPA — plus the test, lint, CI, and contribution workflow. For the production deploy see Installation; for every environment variable see Configuration; for the system design see Architecture.

Contents

Prerequisites

Requirement Notes
Docker Engine + Compose v2 Used for infrastructure services only — TimescaleDB and Redis. The API and worker run on the host from a venv, not in containers.
Python 3.14 The backend targets 3.14 (ruff target-version = "py314"; CI runs the suite on Python 3.14). The package metadata permits >=3.12, but develop on 3.14 to match CI.
Node.js 24+ The frontend builds and tests on Node 24 in CI (@types/node@^24).

Note: Docker is needed only to host db (TimescaleDB) and redis. You can also point at an existing Postgres/TimescaleDB + Redis if you prefer — just adjust the connection URLs below.

Two dependencies are deliberately held below their newest release, in both cases because a tool that consumes them has not caught up:

Held at Why
redis 5.x (Python client) arq, the task queue, pins redis[hiredis]<6. Raising the floor past 6 makes the dependency set unsolvable and the install fails outright. Unrelated to the Redis server, which the stack runs at 7.
typescript below 6.1 (pinned ~6.0.2) No published typescript-eslint declares support for TypeScript ≥ 6.1 (upstream issue #10940). On TypeScript 7 the build and the whole test suite pass — npm run lint is what aborts.

Repository layout

Path Holds
backend/ The FastAPI app (app/: api, connectors, core, models, repositories, schemas, services, worker.py, cli.py), Alembic config (alembic.ini + migrations/), the test suite (tests/), the offline tools/ packages, pyproject.toml, and a dev-only docker-compose.yml for db/redis.
frontend/ The React 19 / Mantine v9 / Vite SPA (src/), package.json, vite.config.ts, and tests under src/**/__tests__/ and src/**/*.test.ts(x).
docker-compose.*.yml Production/overlay stacks at the repo root: prod, tls, caddy, traefik, logs, logs.multinode, and the all-in-one full. See Installation.
.github/workflows/ CI and automation: ci.yml (tests/lint/build/audit), trivy.yml, gitleaks.yml, dependency-review.yml, scheduled-audit.yml, publish-images.yml, and publish-catalogs.yml.
docs/superpowers/ Design specs/ and implementation plans/ for features (reference material, not shipped code).
backend/tools/opnsense_catalog/ The offline OPNsense API-model catalog generator — turns OPNsense's tagged source into versioned JSON catalogs for the Configuration-Editor.

Backend development

1. Bring up infrastructure (db + redis)

The backend ships a dev-only Compose file that runs only TimescaleDB and Redis (the app itself runs on the host):

cd backend
docker compose up -d db redis

This exposes TimescaleDB on localhost:5432 (user/password/db opngms / opngms / opngms) and Redis on localhost:6379.

2. Create the venv and install the package

cd backend
python -m venv .venv
. .venv/bin/activate
pip install -e .[dev]        # the [dev] extra adds pytest, pytest-asyncio, respx, ruff

3. Set the database URLs and required secrets

The app role (opngms_app) is RLS-constrained; the owner role (opngms) runs migrations and the worker, and is RLS-exempt. Generate SESSION_SECRET and MASTER_KEY fresh — the app fails closed on the change-me placeholders.

export DATABASE_URL=postgresql+asyncpg://opngms_app:opngms_app@localhost:5432/opngms
export ADMIN_DATABASE_URL=postgresql+asyncpg://opngms:opngms@localhost:5432/opngms
export REDIS_URL=redis://localhost:6379

# SESSION_SECRET — server-side session signing key
export SESSION_SECRET="$(python -c 'import secrets; print(secrets.token_urlsafe(48))')"

# MASTER_KEY — Fernet key encrypting device + SMTP credentials at rest
export MASTER_KEY="$(python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')"

Note: On a fresh dev DB the opngms_app role does not exist yet — it is created by the first migration (run as the owner). Until then, only ADMIN_DATABASE_URL connects.

4. Run the Alembic migrations

migrations/env.py reads ALEMBIC_DATABASE_URL (falling back to the app DATABASE_URL). Migrations must run as the owner, so point ALEMBIC_DATABASE_URL at ADMIN_DATABASE_URL:

ALEMBIC_DATABASE_URL="$ADMIN_DATABASE_URL" alembic upgrade head

This creates the schema, the non-superuser opngms_app role, RLS policies, and grants. (backend/Makefile wraps the same command as make migrate.)

5. Run the API and worker

uvicorn app.main:app --reload        # API on http://localhost:8000

# In a second shell with the same env exported:
arq app.worker.WorkerSettings        # ARQ worker: polling, ingest, backups, report delivery

Create the first superadmin once via POST /api/setup (it refuses if any user already exists). A locked-out superadmin recovers MFA on the host with python -m app.cli mfa-reset --email <email>.

Frontend development

cd frontend
npm ci --legacy-peer-deps        # a peer-dep range conflict requires --legacy-peer-deps
npm run dev                      # Vite dev server on http://localhost:5173

The Vite dev server proxies /api → http://localhost:8000 (configured in vite.config.ts), so run the backend API alongside it for a single-origin experience with no CORS.

To (re)generate the typed API client from the backend's OpenAPI schema (requires the backend venv):

npm run gen:api        # exports OpenAPI from the backend, then runs openapi-typescript → src/api/schema.d.ts

Running tests locally

Backend

The pytest suite needs a live TimescaleDB (the tests run real async SQLAlchemy against Postgres + RLS). Bring up the db service, create the test database, and point the test URLs at it:

cd backend
docker compose up -d db
docker compose exec -T db psql -U opngms -d opngms -c "CREATE DATABASE opngms_test;"   # make createtestdb

export TEST_DATABASE_URL=postgresql+asyncpg://opngms:opngms@localhost:5432/opngms_test
export ADMIN_DATABASE_URL=postgresql+asyncpg://opngms:opngms@localhost:5432/opngms_test

python -m pytest -q              # or .venv/bin/python -m pytest -q

pytest is configured in pyproject.toml (asyncio_mode = "auto", testpaths = ["tests"], pythonpath = ["."] so the offline tools.* packages import).

Backend lint (ruff — the same gate CI runs, no DB needed):

ruff check app/

Frontend

cd frontend
npm test                 # vitest run (jsdom)
npm run build            # tsc -b && vite build
npm run lint             # eslint .

Note (build gate): Always run npm run build before pushing a frontend change, not just npm test. npm run build runs tsc -b, which type-checks the test files too — so a type error that only surfaces in a test still fails CI even when vitest passes. Skipping the build locally is the most common cause of a red frontend job.

CI & contribution flow

main is a protected branch: no direct pushes. Every change lands via a pull request to main, all required checks must be green, and merges are squash-merged. All files, comments, UI strings, and commit messages are English only.

A green check is not proof that main is green. A pull_request run tests your head merged into the base tip at the time the run happens, and the ruleset does not force a rebase before merging — so on a PR that sat while main moved, the checks validated a tree that no longer exists. On Dependabot PRs there is a second gap: the three required CodeQL Analyze (…) contexts never run, so those merges pass through the admin ruleset bypass rather than a satisfied gate. After merging a batch of PRs — a Dependabot round, for instance — run the suite locally on merged main before tagging anything.

The .github/workflows/ci.yml workflow runs on pull requests to main and provides these required jobs:

Job What it does
Backend tests (Python 3.14 + TimescaleDB) pip install -e backend[dev], then python -m pytest -q against a timescale/timescaledb:2.17.2-pg16 service (TEST_DATABASE_URL/ADMIN_DATABASE_URL → opngms_test). Installs the WeasyPrint system libs first.
Backend lint (ruff) ruff check app/ — fast, no DB.
Frontend (test, build, lint) On Node 24: npm ci --legacy-peer-deps, then npm test, npm run build, npm run lint.
Dependency audit (pip-audit + npm audit) Runs scripts/security_audit.sh; fails on any real app-dependency vulnerability.

Additional security scans gate or baseline the repo:

Workflow Trigger Purpose
Dependency Review (dependency-review.yml) pull request Blocks PRs that introduce dependencies with known high-severity CVEs.
Container Image Scan (trivy.yml) PR + push to main + weekly Builds the prod images and scans them with Trivy (results in the Security tab).
Secret Scan (gitleaks.yml) PR + push to main + weekly gitleaks over the repo and history.
Scheduled Dependency Audit (scheduled-audit.yml) weekly Surfaces new CVEs in already-pinned deps without a code change.

Open the PR only after the backend tests, ruff, the frontend test/build/lint, the dependency audit, dependency review, Trivy, and gitleaks are green.

The OPNsense catalog generator

The generic Configuration-Editor is driven by versioned JSON catalogs of OPNsense's API-modifiable models. Those catalogs are produced offline by the generator at backend/tools/opnsense_catalog/ (CLI: python -m tools.opnsense_catalog.cli, subcommands generate, generate-all, diff, business-base, list-versions). It needs no live device — --fetch downloads the tagged opnsense/core source tarball.

Run it locally from backend/ (with the venv active):

cd backend
# Generate one version's catalog
.venv/bin/python -m tools.opnsense_catalog.cli generate \
    --edition community --version 26.1.8 --fetch --out ../catalog/community/26.1.8.json

# Diff two versions (what changed between releases)
.venv/bin/python -m tools.opnsense_catalog.cli diff \
    ../catalog/community/26.1.7.json ../catalog/community/26.1.8.json

Catalogs are not committed — the running app fetches them at runtime and SHA-256-verifies each against a manifest. The .github/workflows/publish-catalogs.yml Action keeps the rolling catalogs GitHub release fresh: it runs every 6 hours (cron 23 */6 * * *) and on manual workflow_dispatch. Each run discovers the Community release tags, incrementally generates a catalog for any new version (already-published versions are carried from the live manifest via --prior-manifest), refreshes the Business→Community base map, and uploads the assets with gh release upload catalogs ... --clobber. New OPNsense releases are therefore picked up automatically with no app release. The force_all dispatch input regenerates every version — run it after improving the generator so older catalogs are refreshed too.

For how the catalogs power live editing (live option lists, diff badges) see Configuration-Editor and Configuration.

OPNGMS Wiki

Home Screenshots

Deploy & operate

Understand & extend

Clone this wiki locally