v0.2.0

Release theme: production deployability. The dev compose became
an actual production compose (overlay + self-bootstrap), the HA
control plane went from doc to demo profile, and the observability

• multi-tenancy + API-versioning work that landed since v0.1.0 came
  along for the ride. Every shipped path picked up a regression test
  the end-to-end verification work surfaced.

make prod-up is the single command from a fresh checkout to a
fully usable production stack: edit five values in .env.prod,
wait ~90 seconds, get an mTLS API + Celery worker + Next.js
dashboard + hardened MySQL + Vault + Valkey on the box.

Added

• Vault Transit engine + master key auto-bootstrap. Closes the
  last gap in make prod-up: scripts/transit_bootstrap.py is
  invoked by scripts/prod_bootstrap_substrate.sh so a fresh
  Vault gets the Transit mount + the wg-manager master key
  (derived=True) provisioned automatically. Before this, the
  dashboard's Crypto page 500'd with
  no handler for route transit/keys/wg-manager on first load
  because VaultTransitBackend is deliberately dumb and the
  cookbook documented Transit setup as a manual step. Also exposed
  as make transit-bootstrap for operators driving Vault by hand.

• Linux portability fixes for the self-bootstrap (PR #47). PR
  #46's self-bootstrap worked on Docker Desktop on Mac but failed
  on every step against a real Linux host. Four distinct fixes:

  • Bootstrap containers declare user: "0:0" so they can
    write to the bind-mounted ./tls on Linux hosts where the
    operator's UID (typically 1000) doesn't match the wg-manager
    image's wgmanager UID (1001). Docker Desktop on Mac papers
    over the mismatch transparently — pure Linux doesn't. End-of-
    script chown -R 1001:1001 puts the cert files back into the
    runtime UID so api/worker/web can read them.
  • *Bootstrap scripts chmod .key files to 0644 so mysql:8
    inside its container (UID 999) can read the server.key even
    though the file is owned by UID 1001. Without this, mysqld
    fails to load TLS and the app gets
    SSL is required but the server doesn't support it on every
    connection attempt.
  • WG_MANAGER_API_BIND_PORT + WG_MANAGER_WEB_BIND_PORT
    env overrides for operators running multiple stacks on one
    host (or fronting the API behind a reverse proxy that owns
    443).
  • .env.prod.example now recommends openssl rand -hex 32
    for VALKEY_PASSWORD + MYSQL_APP_PASSWORD — these values
    get interpolated into redis://:${pw}@valkey:... and
    mysql+pymysql://wg:${pw}@mysql:... URLs, and the previous
    openssl rand -base64 recommendation produced strings with
    /+= characters that broke URL parsing at startup.

• Self-bootstrapping docker-compose.prod.yml. Edit .env.prod,
  run make prod-up, get a fully usable production-shaped stack on
  the first invocation — no manual cert minting, no shell exports,
  no separate make migrate step. The previous 9-step runbook
  collapses to "fill in 5 values, run one command".

  • Two run-to-completion bootstrap containers break the
    cert/MySQL chicken-and-egg cleanly via Compose's
    service_completed_successfully primitive:
    • bootstrap-substrate — waits for Vault healthy, runs the
      Vault PKI / SSH CA / audit bootstraps (each idempotent), mints
      the MySQL server + client cert pair (--type mysql +
      --type mysql-client for the correct EKU split), exits.
      mysql + valkey depend on it completing successfully.
    • bootstrap-app — waits for MySQL + valkey healthy, runs
      alembic upgrade head, registers the bootstrap operator,
      mints the API server cert + operator CLI client cert, exits.
      api + worker + web depend on it completing.
  • scripts/prod_bootstrap_substrate.sh +
    scripts/prod_bootstrap_app.sh are the two scripts the
    bootstrap containers exec. Both set -euo pipefail and guard
    every mutating step with a file-existence test, so re-running
    make prod-up against persistent state is a no-op (cert files
    present → skip mint; operator registered → skip; alembic at
    head → no-op).
  • alembic moved from dev to runtime deps so the wg-manager
    image bakes /app/.venv/bin/alembic — the bootstrap container
    needs it, and operators pulling
    ghcr.io/jfudally/wg-manager:vX.Y.Z shouldn't have to install
    dev deps to apply migrations.
  • make prod-up gains --wait so the command blocks until
    every service is healthy or exited 0. When the make target
    returns, the stack is fully usable end-to-end.
  • .env.prod.example now documents the operator identity
    (BOOTSTRAP_OPERATOR_CN required, BOOTSTRAP_OPERATOR_ROLE
    optional with default admin) and the API server cert subject
    (API_SERVER_CN + API_SERVER_SANS, both optional with
    localhost-friendly defaults). The old comment block that left
    BOOTSTRAP_OPERATOR_CN commented-out is gone — the prod stack
    requires it.
  • Runbook rewrite at docs/deploy/single-host-prod.md: the
    "Bootstrap (one command)" section is the canonical flow; the
    old 9-step manual flow is preserved as an "Advanced: manual
    bootstrap (for debugging)" section operators can fall back to
    when self-bootstrap fails partway and they want shell-level
    introspection.
  • End-to-end verification on a clean Docker host:
    make prod-down -v → strip tls/ → make prod-up. Sequence:
    bootstrap-substrate exited → mysql + valkey healthy →
    bootstrap-app exited → api + worker + web healthy. Curl on
    /v1/healthz + /v1/readyz returns 200, /v1/servers
    authenticates via the operator cert and returns [], dashboard
    returns 200. Re-running make prod-up is a clean no-op (all
    bootstrap steps log "already exists, skipping").
  • Tests: 16 compose-shape cases
    (tests/test_compose_prod_bootstrap.py) + 18 script-shape
    cases (tests/test_prod_bootstrap_scripts.py) = 34 new. The compose
    suite pins the dependency graph (the bootstrap ordering is
    encoded entirely in depends_on conditions, so a future edit
    that breaks the graph trips at test time). The scripts suite
    pins shebangs, set -euo pipefail, the substrate script's
    invocation of all three Vault bootstrap helpers, the app
    script's alembic upgrade head + operators add + the two
    cert mints, and the file-existence-guard pattern.

• Phase 3d cycle 4a — docker-compose ha profile + nginx
  passthrough LB. Materialises the two-replica + LB topology from
  docs/deploy/ha-control-plane.md on a single host so an operator
  can verify failover end-to-end before deploying a real two-host
  setup.

  • Compose ha profile. Three new services in
    docker-compose.yml, all gated behind profiles: ["ha"]:
    api1 (host port 8001), api2 (host port 8002), and lb
    (host port 8443, nginx:1.27-alpine). Both replicas build the
    existing Phase 2f Dockerfile, bind-mount the dev cert bundle
    in tls/ read-only, and reuse the default-profile data tier
    (mysql + valkey + vault + vector — unprofiled so it comes up
    under both flows).
  • nginx LB config at docker/nginx/wg-manager.conf is
    stream {}-mode TCP passthrough — the HA topology forbids TLS
    termination at the LB so the mTLS handshake lands on the
    replica intact. Two upstreams (api1:8000, api2:8000) with
    passive max_fails=3 fail_timeout=10s checks, single listener
    on 8443.
  • Makefile grew ha-up / ha-down / ha-logs mirroring the
    db-up family.
  • Docs. docs/deploy/ha-control-plane.md gained a "Running
    the ha profile locally" section with the three-endpoint table,
    a failover-smoke recipe, and the explicit list of what cycle
    4a does not ship (active /readyz probing at the LB, TLS
    termination at the LB, MySQL primary→replica plumbing).
  • Tests: 19 compose-shape cases
    (tests/test_compose_ha_profile.py) + 7 nginx-config cases
    (tests/test_nginx_lb_config.py) — pure YAML/text parse-and-
    assert, matches the Phase 2f test_dockerfile.py pattern. CI's
    image-build workflow remains the live compose validator; these
    tests pin the source-of-truth shape so a refactor that drops a
    service or breaks the passthrough contract trips before merge.

• End-to-end verification of the prod overlay surfaced four bugs;
  all four fixed. Walking the runbook on a clean Docker host
  exercised every shipped code path the overlay touches and turned
  up code-level shape mismatches between what the docs promised and
  what the code delivered. Each is fixed with a regression test.

  • Compose ports: list merge — the overlay's
    127.0.0.1:3306:3306 mappings were being concatenated with the
    dev file's 0.0.0.0:3307:3306 instead of replacing it, so the
    second host-port bind failed silently. Fixed with Compose v2.20's
    !override tag on mysql / valkey / vault ports. Test
    fixtures grew a custom ComposeLoader so PyYAML tolerates the
    !override / !reset Compose-specific tags.
  • PEM chain newline missing. VaultPKI._issue_leaf joined the
    Vault ca_chain list with "".join(...) and LocalDevPKI
    used intermediate_pem + root_pem. When the upstream PEM
    lacked a trailing newline (Vault's modern API output), the
    result was -----END CERTIFICATE----------BEGIN CERTIFICATE-----
    on one line — rejected by strict consumers (openssl x509,
    Python ssl.SSLContext.load_verify_locations, pymysql's
    ssl={ca: ...}). New wg_manager.pki._join_pems(*pems) helper
    normalises to strict-parser-safe output and is used at both
    sites. 10 regression cases in tests/test_pki_chain_join.py.
  • Alembic env.py ignored MySQL TLS args.
    engine_from_config(config_section, prefix="sqlalchemy.") built
    its engine without the connect_args = {"ssl": {...}} dict
    wg_manager.db._resolve_mysql_ssl produces — so make migrate
    against a require_secure_transport=ON MySQL failed with
    (3159) Connections using insecure transport are prohibited.
    Fixed by importing _resolve_mysql_ssl and passing
    connect_args=... into engine_from_config. 3 regression
    cases in tests/test_alembic_env_tls.py.
  • API healthcheck assumed /healthz bypasses mTLS at the TLS
    layer; uvicorn's ssl.CERT_REQUIRED says otherwise. The
    Phase 3d cycle 1 doc claim is at the app layer
    (MTLSAuthMiddleware skips auth for /healthz), but uvicorn
    drops the TLS handshake first when no client cert is present.
    The overlay's healthcheck now presents the operator client
    cert (the ./tls bind-mount already makes it available); the
    inline comment + the runbook's "Known limitations" table both
    flag the underlying doc-vs-implementation gap as planned
    follow-on work.

• Production-shaped docker-compose overlay
  (docker-compose.prod.yml). Adds an operator path from
  "dev stack on my laptop" to "single-host non-HA stack on a real
  box". Layers on top of the existing docker-compose.yml (which
  stays the dev file) via docker compose -f docker-compose.yml -f docker-compose.prod.yml up — or the new make prod-up
  wrapper.

  • Three new services. api (mTLS-enforcing FastAPI on public
    443), worker (Celery), web (Next.js dashboard on public
    3000). Both API + worker build the Phase 2f Dockerfile; web
    builds web/Dockerfile. All three pin restart: always so the
    box rebooting brings the stack with it.
  • Production posture on the API + worker. TLS_REQUIRED=true,
    DATABASE_TLS_REQUIRED=true, all three substrate backends
    (CRYPTO_BACKEND / SSH_CA_BACKEND / PKI_BACKEND) pinned to
    vault. The HA startup guard (Phase 3d cycle 1) rejects
    local + TLS_REQUIRED=true without pinned PEMs anyway —
    pinning to vault is the only honest production posture.
  • Hardened data tier. Overrides on mysql / valkey /
    vault source every secret from ${VAR} interpolation
    (.env.prod) using Compose's ${VAR:?msg} fail-loud syntax
    so a missing value blocks compose up instead of silently
    defaulting to a vulnerable string. The dev compose's well-
    known dev-only-root Vault token, rootpw MySQL root
    password, and unauthenticated Valkey are all replaced; data
    tier host port mappings drop to 127.0.0.1 only.
  • .env.prod.example documents every interpolation the
    overlay reads, with inline comments on how to generate strong
    values for each. Gitignore extended to keep the populated
    .env.prod out of the repo.
  • make prod-up / prod-down / prod-logs / prod-config
    wrappers around docker compose --env-file .env.prod -f docker-compose.yml -f docker-compose.prod.yml .... The
    prod-up target guard refuses to start without .env.prod
    on disk.
  • Operator runbook at docs/deploy/single-host-prod.md
    covers the one-time bootstrap sequence (mint MySQL TLS bundle
    → bring up data tier → bootstrap Vault substrate → mint API
    • operator certs → apply migrations → restart api/worker/web),
      the day-2 reference (volume → state mapping, restart behaviour,
      backup commands), the documented limitations vs. fully
      production-ready (Vault still in dev mode, no reverse proxy,
      no in-stack Prometheus, single MySQL, single worker), and the
      upgrade-to-HA path.
  • Tests: 33 overlay-shape cases
    (tests/test_compose_prod_overlay.py) + 3 env-template cases
    (tests/test_env_prod_example.py) = 36 new. Pure parse-and-assert,
    matches the Phase 2f test_dockerfile.py and cycle 4a
    test_compose_ha_profile.py pattern. Pins: service presence
    (api/worker/web), restart-always, mTLS + DB-TLS + Vault
    backends, no hardcoded dev secrets, every ${VAR} documented.

• Phase 3d cycle 3 — per-row advisory locks on mutating Celery
  tasks. Closes the multi-worker concurrency gap cycle 2
  flagged (BENIGN_OVERWRITE on contention). The 4 mutating tasks
  now serialize on a MySQL GET_LOCK advisory lock keyed on the
  row they mutate.

  • Lock helper. New wg_manager.locks module exposes
    lock_name_for(scope, row_id) (wgm:server:7 shape) and
    task_row_lock(session, scope, row_id, timeout_seconds=5)
    context manager. On MySQL the lock uses
    GET_LOCK(name, timeout) + RELEASE_LOCK(name); on SQLite
    (test suite) a no-op acquire that yields True. Failed
    acquire yields False — the caller decides whether to skip
    or retry. Connection-scoped so a worker crash leaves no
    stranded lock.
  • Applied to the 4 mutating tasks. provision_server_task,
    rotate_host_cert_task, reconfigure_server_task lock on
    wgm:server:<server_id>; provision_client_task locks on
    wgm:client:<client_id>. On contention the task returns
    {"status": "skipped", "reason": "concurrent_run", ...}
    without any SSH / DB-mutation side effects. Skipped result
    rides through the GET /tasks/{id} API so operators see the
    skip.
  • Verdicts updated. The 4 mutating tasks' Phase 3d cycle 2
    docstring stanzas flipped from BENIGN_OVERWRITE to
    GUARDED_BY_ROW_LOCK. The cycle 2 marker test still passes.
  • Docs. docs/deploy/ha-control-plane.md Celery section
    rewrote the per-task verdict table + added an "Advisory lock
    contract" subsection explaining the name shape, the
    GET_LOCK timeout, the SQLite no-op path, and the
    monkey-patch pattern tests use to exercise the contended
    branch.
  • ROADMAP scope note. Original ROADMAP cycle 3 wording was
    "MySQL primary + read-replica routing"; cycle 3 instead
    landed the advisory locks deferred from cycle 2 (more
    immediate safety win), with read-replica routing folded into
    cycle 4 alongside the compose ha-profile.
  • Tests: 8 lock-helper cases (tests/test_locks.py) + 8
    task-level integration cases (tests/test_task_locks.py)
    pinning the lock-acquired path (each task records the
    expected (scope, row_id)) and the contended path (each
    task returns skipped and fires zero SSH commands). Backend
    pytest 953/953 in local mode (was 937 on cycle 2's merge).

• Phase 3d cycle 2 — Celery worker scaling guarantees. Makes the
  Celery worker side safe to run as 2+ replicas behind the same
  broker. Codifies the at-least-once delivery contract every task is
  written against.

  • Idempotency audit of the 6 shipped tasks
    (provision_server, rotate_host_cert, reconfigure_server,
    provision_client, discover_peers, discover_all_peers). Verdict:
    4× BENIGN_OVERWRITE + 2× NATURALLY_IDEMPOTENT. No
    NEEDS_GUARD findings. Per-row advisory locks (the natural
    cycle 2 follow-on) deferred to cycle 3, which brings MySQL's
    GET_LOCK() into scope.
  • task_reject_on_worker_lost=True added to
    celery_app.conf. Pairs with the existing task_acks_late=True
    to form the at-least-once contract: a SIGKILL'd / OOM'd worker
    mid-task triggers broker requeue instead of silent task loss.
  • Per-task contract pinned in docstrings. Each task grew a
    Phase 3d cycle 2 stanza naming its idempotency classification
    • reasoning. A regression test greps for the marker so a
      refactor that rewrites a task body without re-examining the
      audit trips a clear failure.
  • No beat scheduler. Zero periodic tasks exist in the
    codebase. The "single-beat vs distributed-beat" decision is
    deferred until a periodic task is actually needed (cert renewal
    sweep is the obvious first candidate).
  • Docs. docs/deploy/ha-control-plane.md grew a "Celery
    worker scaling" section with the at-least-once contract table,
    the per-task idempotency table, an "Adding a new task"
    checklist (matches the existing Statelessness checklist
    pattern), and the cycle 3 advisory-lock deferral note.
  • Tests: 9 new cases (tests/test_celery_ha_config.py) —
    config flags pinned, every task name registered with
    celery_app, every task's __doc__ carries the cycle 2
    audit-verdict marker. Backend pytest 937/937 in local mode
    (was 928 on cycle 1's merge).

• **Phase 3d cycle 1 — statelessness audit + /healthz + /readyz

  • HA startup guards.** Foundational slice of Phase 3d (HA control
    plane). Verifies the API is safe to run as two+ replicas behind a
    load balancer and adds the probes the LB uses to route traffic.
  • Statelessness audit. Walked every module on the request
    path and classified module-level state. Verdict: the API is
    mostly stateless. Two genuine cross-replica hazards surfaced
    — the LocalDevPKI and LocalDevSSHCA per-process root-cert
    caches in dev backends would mint divergent roots across
    replicas if unpinned. Production (Vault for both) eliminates
    the hazard.
  • Probe surface. New wg_manager.routers.health ships
    /healthz (liveness — unconditional 200; does not touch the
    DB) and /readyz (readiness — 200 when MySQL is reachable,
    503 with per-dep status otherwise). Dual-mounted at /v1 per
    Phase 3c. Both bypass mTLS via the new
    MTLSAuthMiddleware.is_health_path exemption (load balancers
    don't carry operator certs) and are exempt from the
    deprecation envelope (operational, not API surface).
  • HA startup guards. New
    wg_manager.main._enforce_ha_startup_guards hard-fails at
    create_app() time when TLS_REQUIRED=true AND
    PKI_BACKEND=local (or SSH_CA_BACKEND=local) without the
    corresponding *_LOCAL_DEV_* PEMs pinned. Error names the env
    vars to set so an operator fixes the misconfiguration without
    reading source. Dev posture (TLS_REQUIRED=false) is
    permitted to run the local backends unpinned.
  • Deployment doc. New docs/deploy/ha-control-plane.md
    captures the topology (passthrough LB, no session stickiness,
    mTLS termination at the replica), the probe contract (why two
    probes, not one), a Statelessness checklist for future
    maintainers, and an nginx LB example.
  • Tests: 9 health-probe cases (tests/test_health.py) + 5
    startup-guard cases (tests/test_ha_startup_guards.py). Two
    existing test_main_tls_wiring.py cases updated to pin
    PKI_BACKEND=vault + SSH_CA_BACKEND=vault since they
    exercise production posture and would otherwise trip the new
    guard. Backend pytest 928/928 in local mode (was 914 on
    Phase 3c's merge).

• Phase 3c — public API versioning (/v1 namespace + deprecation
  policy). Every router that shipped under an unprefixed path is
  now dual-mounted at the same path under /v1. Existing
  integrations keep working unchanged; new callers opt into the
  explicit version. The CLI and dashboard BFF are cut over to
  /v1; third-party callers have until the operator-configured
  sunset date to migrate.

  • Dual mount. /ssh-keys, /servers, /clients, /certs,
    /tenants, /audit, /crypto, /tasks all answer at both
    /<resource> and /v1/<resource>. Identical handler, identical
    response body + status.
  • Deprecation envelope (RFC 9745). Every legacy-path response
    carries Deprecation: true, Sunset: <date>, and
    Link: <doc>; rel="deprecation". New
    wg_manager.api_versioning.DeprecationMiddleware stamps the
    headers and emits one api.deprecation audit line per legacy
    hit so operators can SIEM-query for callers still on the legacy
    surface. The sunset date and link target are operator-tunable
    via API_LEGACY_SUNSET_DATE + API_DEPRECATION_DOC_URL settings.
  • Versioned OpenAPI surface. /v1/openapi.json filters the
    spec to /v1/* paths only and pins info.version = "1.0". The
    existing /openapi.json continues to surface both spaces.
  • CLI cutover. cli._make_http_client suffixes the base URL
    with /v1 so every existing call site (which uses relative
    /ssh-keys etc) lands on the versioned namespace.
  • Dashboard BFF cutover. lib/proxy.forwardToUpstream rewrites
    every inbound /api/proxy/<path> to <upstream>/v1/<path>.
    Strips an existing v1/ prefix so callers that pre-pended it
    don't get a double prefix.
  • Docs. New docs/api-versioning.md walks the deprecation
    envelope, the semver contract, the cutover guidance, and the
    removal timeline.
  • Tests: 13 new versioning cases
    (tests/test_api_versioning.py) — dual mount, deprecation
    headers on legacy / absent on v1, audit emission, OpenAPI
    filtering — plus 1 new vitest case pinning the BFF's no-double-
    prefix guard. Backend pytest 914/914 in local mode (was 901
    on Phase 3b's close); vitest 57/57; tsc --noEmit clean.

• Phase 3b cycle 5 — explicit tenant on resource POSTs + tenant SAN on
  certs. Closes Phase 3b. Cycle 4 inherited the server's tenant from
  the SSH key; cycle 5 makes the resolution explicit. Cycle 3 added
  per-operator tenant scope from the OperatorTenant join; cycle 5
  lets non-operator service identities (CI runners, automation
  accounts) carry a tenant binding via a tenant:<slug> SAN baked
  into the leaf.

  • Resource POST tenant resolution. POST /ssh-keys,
    POST /servers accept an optional tenant_id in the body. The
    new wg_manager.tenant_scope.resolve_create_tenant helper
    centralises the four decision branches: super-admin without
    tenant_id → default tenant (id=1); single-tenant operator
    without tenant_id → auto-derive; multi-tenant operator
    without tenant_id → 422 demanding an explicit choice (body
    names every candidate so the dashboard can render the select
    widget straight from the error); no-tenant operator → 403. The
    resolved tenant must permit the operator's per-tenant
    admin/operator role; auditor 403s. Servers' cycle 4
    pool-containment check now runs against the resolved tenant,
    not the SSH key's.
  • Tenant SAN convention on cli / dashboard certs.
    wg-manager certs issue --type cli --tenant acme (and the
    matching POST /certs with tenant_slug: "acme") appends a
    tenant:acme DNS-SAN to the leaf and populates
    Certificate.tenant_id on the audit row. Refused on the three
    server-EKU cert types (api, mysql, mysql-client).
    Unknown slug → 422.
  • Dashboard parity. SSHKeyCreate.tenant_id + ServerCreate.tenant_id
    • CertificateIssueRequest.tenant_slug added to
      web/lib/types.ts. Certificates page Issue form grows a
      "Tenant slug (optional)" input that appears only for cli /
      dashboard types and rides through to the POST body. Backend's
      422 / 403 errors render in the existing Alert pattern.
  • Conftest seed. The in-memory engine fixture now seeds the
    default tenant at id=1 to mirror Alembic 0014; existing test
    helpers that re-inserted the default tenant became upserts.
  • Tests: 12 new resource-resolution cases
    (tests/test_resource_tenant_resolution.py) + 7 new tenant-SAN
    cases (tests/test_cert_tenant_san.py) + 2 new vitest specs
    for the cert form. Backend pytest 901 passed in local mode
    (was 882 on cycle 4's merge); vitest 56/56; tsc --noEmit clean.
  • Phase 3b closes here. Cycles 1-5 shipped; every bullet on
    the Phase 3b sub-roadmap is [x]. ROADMAP header updated.

• Phase 3b cycle 4 — per-tenant peer pools (IPAM). Each tenant
  carries its own subnet_pool CIDR; every server's subnet must
  lie inside the pool, and two tenants' pools must be disjoint —
  so a client IP in one tenant cannot collide with a client IP in
  another. Closes the "IP collisions between tenants" half of the
  Phase 3b design lock.

  • Schema. Tenant grows a subnet_pool NOT NULL VARCHAR(64)
    column via Alembic 0016. The migration back-fills the reserved
    id=1 default tenant with Settings.default_subnet so a v0.1.0
    deployment keeps every existing server inside its tenant's pool
    without operator action; any other tenant rows added between
    cycles 2 and 4 back-fill to the RFC1918 fallback 10.0.0.0/8
    (the largest private block — operators tighten via PATCH).
  • IPAM helpers in wg_manager.ipam: subnet_in_pool(subnet, pool) (strict containment check) + pools_overlap(a, b)
    (overlap check). The existing allocate_client_ip walks the
    server's subnet unchanged; cross-tenant non-collision falls out
    of pools being disjoint by construction.
  • CLI. wg-manager tenants create --subnet-pool 10.42.0.0/16
    stores the pool; without the flag the row carries the model
    default. Overlap with an existing tenant is rejected with a
    non-zero exit + a message naming the colliding tenant. tenants list/get JSON output grows the subnet_pool field.
  • HTTP. TenantCreate body grows optional subnet_pool;
    TenantRead surfaces it; new PATCH /tenants/{slug} accepts a
    TenantUpdate body to widen/narrow the pool. Overlap is
    rejected with HTTP 409 (and excludes the row being updated on
    PATCH so an in-place narrow doesn't self-collide). Malformed
    CIDR → 422.
  • Per-server pool enforcement. POST /servers rejects a
    subnet that lies outside the resolved tenant's pool with HTTP
    422; the existing default-subnet path inherits the SSH key's
    tenant so a v0.1.0 deployment keeps working. The row's
    tenant_id is populated from the resolved tenant.
  • Dashboard parity. Tenants page inventory grows a "Subnet
    pool" column; create form grows a Subnet pool (optional)
    input; the per-tenant detail panel header surfaces the pool in
    a monospace span. Tenant / TenantCreate / new TenantUpdate
    types updated; new api.updateTenant method.
  • Tests: 9 new alembic-0016 cases (tests/test_alembic_0016.py)
    • 18 new plumbing cases (tests/test_tenant_subnet_pool.py)
      covering CLI create/list/get with pool, API POST + overlap
      rejection, IPAM helpers, per-server pool enforcement, and
      cross-tenant non-collision. 2 new vitest specs covering the
      inventory subnet column + the create-form pool submission.
      Backend pytest 882/882 in local mode (was 855 on cycle 3's
      merge); vitest 54/54; tsc --noEmit clean.

• Phase 3b cycle 3 — tenant-aware filtering + per-tenant role gate.
  The first cycle that actually enforces the multi-tenant model.
  Cycles 1 + 2 shipped pure schema groundwork; cycle 3 reads the
  OperatorTenant join at request time and uses it to narrow every
  list query, gate every mutation, and tag every audit event with
  the affected resource's tenant.

  • Middleware tenant resolution. MTLSAuthMiddleware.dispatch
    now also reads the operator's OperatorTenant join rows once
    per admitted request and stashes
    request.state.tenant_ids / tenant_roles / is_super_admin.
    Super-admin = global Operator.role == admin per the ROADMAP
    design lock — bypasses every per-tenant gate. The OPTIONS
    preflight and TLS_REQUIRED=false passthrough branches both
    leave the slots as None so handlers can distinguish "auth
    disabled" from "auth admitted with empty set".
  • Tenant scope helper (wg_manager.tenant_scope). New
    TenantScope frozen value object + get_tenant_scope FastAPI
    dependency + scope_filter(scope, Model) (returns a
    Model.tenant_id IN (...) expression, or None when no filter
    applies) + require_tenant_role(scope, tenant_id, *allowed)
    (HTTP 403 unless the operator has one of allowed per-tenant
    roles on tenant_id; super-admin bypass).
  • List filtering applied to /servers, /clients,
    /ssh-keys. Non-super-admin operators see only the rows whose
    tenant_id is in their OperatorTenant join set; an operator
    with no joins gets [] (not 403). Super-admin sees every row.
  • 404-on-out-of-scope for the single-row GET /
    PATCH / DELETE shapes — the existence of a row in another
    tenant is never leaked to a probing operator.
  • Per-tenant role gate on PATCH / DELETE of /servers,
    /clients, /ssh-keys. admin and operator per-tenant roles
    admit; auditor 403s. Super-admin bypasses.
  • AuditEvent.tenant_id populated. audit.persist() grew an
    optional tenant_id kwarg; the servers / clients / ssh_keys /
    certs mutating endpoints thread the resource's tenant_id
    through so an auditor reviewing the trail can filter per
    tenant. The audit log line emitted alongside the row also
    carries tenant_id.
  • Dashboard surface. Server / Client / SSHKey /
    Certificate schemas + TypeScript types grow an optional
    tenant_id field so the dashboard can render tenant tags. The
    deeper "tenant picker on resource create" + per-list tenant
    column polish lands in cycle 5 alongside the IPAM partitioning.
  • Tests: 6 new middleware tenant-resolution cases
    (TestTenantSetResolution in tests/test_auth.py), 25 new
    cases in tests/test_tenant_scope.py covering the value
    object, scope_filter, require_tenant_role, server list +
    get-by-id + patch + delete scoping, and audit-event tenant_id
    population. Backend pytest 855 passed in local mode (was 824
    on cycle 2's merge); vitest 52/52; tsc --noEmit clean.

• Phase 3b cycle 2 — OperatorTenant join + tenant CRUD surface.
  Cycle 1 shipped the Tenant row + nullable FKs. Cycle 2 layers the
  many-to-many association: one operator can be attached to many
  tenants, one tenant can host many operators, and the per-tenant
  role lives on the join — so a user can be admin in their own
  tenant and auditor in another without two separate operator
  rows. Zero behaviour change for callers — the auth middleware
  still consults the operator's global role; cycle 3 is what
  flips per-tenant enforcement on.

  • New OperatorTenant SQLModel: id / operator_id FK /
    tenant_id FK / role / created_at. Unique constraint on
    (operator_id, tenant_id) so a duplicate attach is rejected at
    the DB layer as the last line of defence.
  • Alembic 0015 creates the join table + back-fills one row per
    existing operator pointing at the default tenant (id=1) and
    mirroring the operator's existing global role as the
    per-tenant role. Idempotent upgrade → downgrade → upgrade
    round-trip.
  • CLI surface. New wg-manager tenants create/list/get +
    wg-manager operators attach-tenant/detach-tenant/list-tenants
    direct-DB subcommands. Mirrors the wg-manager operators add/list shape (works before the API listener is up — same
    canonical bootstrap path).
  • HTTP surface. New /tenants router exposes the CLI shape
    over mTLS: GET /tenants, GET /tenants/{slug} (admin or
    auditor); POST /tenants (admin); POST /tenants/{slug}/operators,
    DELETE /tenants/{slug}/operators/{cn} (admin);
    GET /tenants/{slug}/operators (admin or auditor). Role
    gating mirrors /certs byte-for-byte.
  • Dashboard parity. New web/app/tenants page with the
    tenant inventory table, a Create form, a per-tenant detail
    panel rendering the attached-operator table with per-tenant
    role badges, an Attach form, and per-row Detach buttons. New
    "Tenants" nav entry; new Tenant, TenantCreate,
    OperatorTenantRead, OperatorTenantAttachRequest types and
    matching api.*Tenant* methods.
  • 16 new alembic test cases (tests/test_alembic_0015.py) — join
    table shape, FK references, unique constraint at the DB layer,
    per-row backfill from each of the three operator roles + the
    two-operator-count sanity, downgrade + idempotent round-trip,
    model-surface defaults + repr safety.
  • 16 new CLI test cases (tests/test_cli_tenants.py) — tenants
    create / list / get happy + failure paths, slug derivation from
    name, duplicate-slug refused; operators attach/detach/list-tenants
    happy + unknown-cn / unknown-tenant / duplicate-pair errors.
  • 21 new API test cases (tests/test_tenants_api.py) — list +
    detail + create (admin / auditor / plain operator role gates;
    duplicate slug → 409; slug-from-name derivation); attach
    (happy / unknown-cn → 422 / unknown-tenant → 404 / duplicate
    → 409 / default-role / auditor 403); detach (happy / unknown
    pair → 404 / auditor 403); per-tenant operator list.
  • 6 new vitest specs (web/__tests__/tenants.test.tsx) — list
    render, empty state, create form POST, per-tenant detail
    operator table, attach form POST, detach DELETE.
  • Fix-along: tests/test_alembic_0014.py's downgrade-round-trip
    tests pinned to the explicit pre-revision name instead of
    -1 — -1 silently turned into "downgrade only 0015" once
    0015 landed on top, and the prior tests would have looked
    green while testing nothing.

• Phase 3b cycle 1 — multi-tenant schema groundwork. Opens
  Phase 3b. Zero behaviour change — pure schema migration so
  operators upgrade a v0.1.0 deployment without re-issuing certs
  or re-bootstrapping. Cycles 2-5 layer enforcement on top.

  • New Tenant SQLModel: id / name (unique) / slug
    (unique) / created_at.
  • Alembic 0014 creates the tenant table, inserts a default
    tenant row at id=1, adds a nullable tenant_id FK
    column + index to each of the six tenanted resource tables
    (operator, server, client, sshkey, certificate,
    auditevent), and back-fills every existing row to the
    default tenant. Nullable for cycle 1 so the migration is
    non-breaking; cycle 3 will tighten to NOT NULL once auth-side
    filtering enforces the invariant.
  • 25 new test cases in tests/test_alembic_0014.py: tenant
    table shape (3), default tenant row inserted at id=1,
    tenant_id column added to each of 6 tables × 3 assertions =
    18 parametrised, FK references tenant(id), back-fill from
    prior-revision data assigns existing rows to default tenant,
    downgrade reverses cleanly + idempotent round-trip.
  • ROADMAP grew a Phase 3b sub-phase with the locked design
    decisions (namespace-style tenancy, OperatorTenant join,
    incremental enforcement) and the 5-cycle plan; cycle 1
    flipped to [~] in progress.

• Phase 3a cycle 3 — cert-lifecycle dashboard + alerting recipes.
  Closes Phase 3a (observability). A v0.1.0 operator can now
  answer "which certs are due for renewal this week?" and "should
  on-call be paged right now?" from Grafana + Prometheus alone.

  • New gauge metric wg_manager_cert_not_after_seconds{serial, cn, cert_type} emitted by a custom
    CertificateLifecycleCollector that walks the certificate
    table on every scrape. Revoked rows excluded (emitting their
    expiry would either fire noisy "expiring soon" alerts on
    decommissioned certs, or mask the absence of a real
    replacement).
  • New docs/observability/grafana-cert-lifecycle.json dashboard
    with 5 panels: nearest-expiry top-20 table, expiring-in-7-days
    • expiring-in-30-days stats by cert type, lifecycle event-rate
      timeseries (issue / renew / revoke), active cert count by type.
  • New docs/observability/prometheus-alerts.yaml ships three
    alerting rules:
    • Wg5xxSurge — 5xx fraction > 5% over 5m
    • WgVaultLatencyHigh — Vault round-trip p95 > 2s for 5m
    • WgCertExpiringSoon — non-revoked cert TTL < 7 days
      Each rule includes a runbook annotation pointing at the
      corresponding wg-manager runbook so Alertmanager templates can
      render clickable links in the page payload.
  • docs/observability.md grew Cert-lifecycle + Alerting-recipes
    sections covering the new metric, useful PromQL recipes, the
    dashboard panels, and the alert tuning knobs.
  • 20 new test cases:
    tests/test_cert_lifecycle_collector.py × 4 (gauge appears on
    /metrics, includes active certs, excludes revoked, carries
    cn+cert_type labels);
    tests/test_grafana_cert_lifecycle.py × 5 (file exists, valid
    JSON, top-level shape, panels cover the cert gauge + lifecycle
    counters);
    tests/test_prometheus_alerts.py × 11 (file exists, valid
    YAML, all three alerts present, exprs reference the canonical
    metrics, every alert has expr + annotations.summary).

• Phase 3a cycle 2 — OTLP trace exporter on the provisioning path.
  Three span families covering the full provisioning trace: Celery
  task root spans (auto-instrumented), Vault round-trip sub-spans
  (cycle 1's vault_call extended to start a span at the same
  wrap site), and SSH command sub-spans (new ssh_span helper
  wrapped around SSHRunner.run / .sudo).

  • New wg_manager.tracing module sets up the OTel SDK with four
    exporter modes (none default = zero overhead, console for
    dev, otlp-http for production, memory for tests).
  • Cycle 1's vault_call context manager extended to also start
    a vault.<engine>.<operation> span — one wrap site, two
    streams. A metric-only deployment and a metric+trace
    deployment never drift apart.
  • New ssh_span(operation, **attrs) helper wraps
    SSHRunner.run and SSHRunner.sudo so the trace shows every
    command-exec as a sub-span under the parent Celery task.
  • CeleryInstrumentor() instruments every task automatically.
  • setup_tracing invoked at import time from both
    wg_manager.main (API) and wg_manager.celery_app (worker)
    so spans land under whichever process executed them.
  • Cycle 1 gap closure: tests/test_call_sites_traced.py
    greps the source for every expected vault_call(...) +
    ssh_span(...) + setup_tracing invocation. The cycle 1
    commit shipped a gap where the ROADMAP claimed Vault wraps
    were in place but no test pinned the source-level contract;
    cycle 2 closes that gap so the same drift can't recur silently.
  • docs/observability.md grew a Tracing section: span topology
    diagram, sample OTLP collector config, Honeycomb / Jaeger /
    Tempo pointers, worker-vs-API setup parity.
  • 21 new test cases: tests/test_tracing.py × 11 (behavioural —
    exporter selection, vault_call span emission + attributes +
    ERROR status, ssh_span helper, Celery instrumentor); plus
    tests/test_call_sites_traced.py × 10 (source-level grep
    pinning every wrap site).
  • New runtime deps: opentelemetry-api, opentelemetry-sdk,
    opentelemetry-exporter-otlp-proto-http,
    opentelemetry-instrumentation-celery.
  • New Settings fields: otel_exporter,
    otel_exporter_otlp_endpoint, otel_service_name.

• Phase 3a cycle 1 — Prometheus metrics + Grafana dashboard.
  Opens Phase 3 (Scale / Polish). v0.1.0 operators can now answer
  "is wg-manager healthy right now?" from Prometheus + Grafana,
  not log greps.

  • New wg_manager.metrics module declares nine metric families:
    HTTP (requests_total + request_duration_seconds), Celery
    (tasks_total + task_duration_seconds), Vault round-trips
    (requests_total + request_duration_seconds), and cert
    lifecycle (certs_issued_total + _revoked_total +
    _renewed_total).
  • MetricsMiddleware (ASGI) records every HTTP request,
    skipping OPTIONS preflight and /metrics itself. The path
    label uses the FastAPI route template, not the raw URL,
    so cardinality stays bounded by the route table.
  • Celery task_prerun + task_postrun signal handlers record
    every task automatically — the side-effect import lives in
    celery_app.py so the metrics fire under the worker process,
    not just the API.
  • vault_call(engine, operation) context manager records
    latency + outcome (ok / error) on every Vault round-trip.
    Call sites in crypto / ssh_ca / pki get a one-line wrap.
  • GET /metrics endpoint exposes the Prometheus text format on
    the existing mTLS listener. Scrapers configure a client cert
    the same way operators do — keeping the security posture
    uniform.
  • docs/observability/grafana-dashboard.json ships a starter
    dashboard with 7 panels covering every metric family (HTTP
    request rate + p95 latency, Celery throughput + p95 duration,
    Vault round-trip latency + rate, cert lifecycle events).
    Importable via Grafana UI's Upload JSON flow.
  • docs/observability.md walks the scrape config, metric
    families, dashboard panels, and the three instrumentation
    patterns for future call sites.
  • New prometheus-client>=0.20.0 runtime dependency.
  • 26 new test cases:
    tests/test_metrics.py × 18 (metric family declarations + labels,
    /metrics endpoint shape, middleware records-a-request +
    route-template-not-raw-path + skips OPTIONS + skips /metrics,
    vault_call records ok/error/duration, Celery signal
    registration);
    tests/test_grafana_dashboard.py × 8 (file exists, valid JSON,
    has title + panels, every metric family covered by a PromQL
    query in at least one panel).

---

Published images

• ghcr.io/jfudally/wg_manager:v0.2.0 — API + worker
• ghcr.io/jfudally/wg_manager-web:v0.2.0 — dashboard

Pull either with a granularity-appropriate tag (vX.Y.Z, vX.Y, vX, or latest).

Supply-chain attestation

Both images are signed via cosign keyless OIDC and carry an
in-toto CycloneDX SBOM attestation. Verify with:

cosign verify \
    --certificate-identity-regexp \
        'https://github.com/jfudally/wg_manager/.github/workflows/release.yml@.*' \
    --certificate-oidc-issuer \
        'https://token.actions.githubusercontent.com' \
    ghcr.io/jfudally/wg_manager:v0.2.0

The SBOMs are also attached as release assets:

• sbom-api.cdx.json — CycloneDX SBOM of the API + worker image's Python deps
• sbom-web.cdx.json — CycloneDX SBOM of the dashboard image's Node deps