Sysible Controller manages a fleet of Linux hosts and can run privileged commands on them. Treat the controller as a high-value system: anyone who controls it (or its API key) effectively has root on every enrolled host. This document describes the security controls in place and how to deploy the controller safely.

• The controller API (backend.app, port 9000) is the brain. It is protected by an admin API key.
• Agents on managed hosts do not use the admin API key. They authenticate with a per-host secret issued at enrollment, and enrollment itself requires a single-use token generated by an authenticated admin.
• The web console (sysible-webgui, port 8800) is a browser front end for administrators, and the only administrative interface. It is a backend-for-frontend (BFF): the service reads the admin API key locally (root-only file) and holds it server-side, so the browser never receives it. A separate administrator username/password gates the console itself; the browser authenticates only with a signed, http-only session cookie; the administrator's login token is encrypted into that cookie with a server-side key. It enforces administrator accounts, roles, and run-as identity server-side. Like the API it is network-reachable and must be firewalled to a trusted subnet/VPN.
• The Webserver Portal (optional, started on demand) is a host-facing service for agent-bundle/file download. It has its own login and runs as a separate process on its own port.

Transport

• The API and the portal are served over HTTPS (TLS). On a LAN with no public domain, a self-signed certificate scoped to the controller's hostname/IPs is generated by install_sysible.sh. Clients pin to that certificate (trust-on-first-use). To confirm the pin is the genuine cert rather than a machine-in-the-middle, read the certificate's SHA-256 fingerprint out-of-band and compare it: the console shows it in Settings → TLS Certificate (and the /controller-config/tls/info API returns sha256_fingerprint), matching openssl x509 -noout -fingerprint -sha256 -in server.crt. The fingerprint changes whenever the certificate is regenerated (address change) or replaced (PKI import) — redistribute the trust bundle / re-download the agent bundle afterwards so hosts re-pin.

• SSH-managed hosts are authenticated with a per-fleet key pair the controller generates and owns (never a stored password). The controller also verifies each host's SSH host key on a trust-on-first-use basis against one controller-side remote_keys/known_hosts (mode 600): the key is pinned on first contact and checked on every subsequent connection across all SSH paths — command exec, the interactive terminal, SFTP file transfer, and enrollment. A changed host key is refused (HTTP 409) rather than silently re-trusted, so a man-in-the-middle or a swapped box is caught; removing a host forgets its pinned key so a legitimate rebuild can re-enroll at the same IP.

• Controller-layer RBAC. Role enforcement is on the controller, not only the web console: every privileged route resolves the caller's admin token against the controller's own DB (require_superuser / require_activity_viewer dependencies) and refuses the read-only auditor role at the task-dispatch path — so holding the API key alone doesn't grant a role the token lacks. The actor attributed in the audit log and the run-as identity handed to the agent come from that validated token, never a client-supplied field.

• Optional mutual TLS (client certificates). Off by default. For a hardened deployment the controller API on :9000 can require every caller to present a client certificate signed by a CA you control, so a leaked API key alone can't reach the API from an unprovisioned host. Enable it at install with --mtls --mtls-ca=/path/to/client-ca.crt (add --mtls-mode=optional for a staged rollout that logs but still accepts certless clients); this writes /opt/sysible/mtls.env, which the systemd unit sources to pass --ssl-cert-reqs/--ssl-ca-certs to uvicorn. The console/CLI present their cert via SYSIBLE_CLIENT_CERT (+ SYSIBLE_CLIENT_KEY). Recommended rollout: (1) issue client certs and set those vars on the BFF/CLI; (2) start the controller with --mtls-mode=optional; (3) once all clients present certs, re-run with --mtls-mode=required. Each change is a controller restart (uvicorn reads TLS once at start). Turn it off by emptying/removing mtls.env and restarting. Fail-closed: if you request required mTLS but the CA is missing/unreadable, the installer (and the container entrypoint) refuse to proceed rather than silently starting server-auth-only — so you never believe client-cert auth is on when it isn't. --mtls-mode=optional with a missing CA warns and continues (staged rollout).

Authentication & secrets

• Admin API key: 256-bit random, stored at /opt/sysible/api_key.txt mode 600 (root-only), compared in constant time. Never sent to a browser.
• Admin and portal passwords: hashed with PBKDF2-HMAC-SHA256, 600,000 iterations (OWASP-current), per-credential random salt, constant-time verification, with transparent cost-upgrade on the next password change.
• Agent enrollment: single-use, admin-generated tokens, bound to the first host that claims them and valid for 30 days by default (SYSIBLE_ENROLL_TOKEN_VALID_DAYS) so a leaked-but-unused bundle can't enroll a rogue host indefinitely; per-host secrets thereafter. A replayed token (within its reuse window) cannot take over a host that is still live or was administrator-revoked — re-enrollment is refused in both cases, so a leaked bundle can't hijack an active host's identity or resurrect a revoked one. A removed/disenrolled host can be force-deleted from the console even if its agent is a zombie that won't tear down; deleting the record invalidates the secret on the next heartbeat.
• Optional enroll source allowlist (enroll_ip_allowed) and a per-IP enrollment flood guard key on the real socket peer (never a spoofable X-Forwarded-For). Deployment note: run the controller so it terminates TLS itself (the default). If you front it with a reverse proxy that re-originates the connection, every enroll appears to come from the proxy (loopback), which is always allowed — so the allowlist and per-IP flood guard would no longer distinguish real clients. Terminate TLS on the controller, or restrict enroll reachability at the network layer, to keep those controls effective.
• Credentials at rest are owner-only. The controller's SQLite database (administrator password hashes, agent bearer secrets, live login tokens), the API key, the TLS private key, the sudo-store key, and the SSH keys all live under /opt/sysible with the DB (and its WAL/SHM sidecars) forced to 0600 and the install tree to 0700 — so no other local account can copy the database and impersonate agents or admins.
• Secrets (SSH/CIFS/subscription credentials) are kept off the command line where the underlying tool supports it, and shell arguments are quoted (shlex.quote).
• Service hardening. The sysible-backend and sysible-webgui systemd units ship with a baseline of hardening directives (ProtectKernelTunables, ProtectKernelModules, ProtectControlGroups, ProtectClock, RestrictRealtime, RestrictSUIDSGID, LockPersonality) that don't restrict the file/exec/network access the controller legitimately needs.

HTTP hardening

• Security response headers on both apps: HSTS, X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer, Cache-Control: no-store, and a Content-Security-Policy.
• Interactive API docs (Swagger/ReDoc) are disabled on both apps so the API surface isn't published as a browsable console / request builder. The web console (BFF) also disables its OpenAPI schema entirely; the controller keeps /openapi.json (no Swagger UI) because its own readiness self-check fetches it to confirm the running process is current code — and the controller is meant to be firewalled to a trusted network with every real endpoint behind the API key.
• Portal and web-console session cookies are HttpOnly, Secure (under TLS), SameSite=Strict — which also closes CSRF on the state-changing POSTs without a separate token.
• Brute-force throttling: per-IP lockout on the public portal login (5 failures / 15 min → 15 min lockout) — shared by the portal's /cli/bundle HTTP-Basic path, so the copy-paste curl route can't be used as an unthrottled password oracle — and, as defense-in-depth, on the admin login (10 / 15 min → 10 min lockout) and the web-console login (per-IP attempt cap → HTTP 429 cooldown; a successful login rotates the session as session-fixation hardening).
• File-upload endpoints (web console and portal) reject an oversized Content-Length and bound how much of a body they buffer, so a single upload can't exhaust memory on the shared process.
• The agent channel is size-capped. A managed host runs the agent, so a compromised one is in the threat model: the per-request payloads it can post (metrics, snapshot, integrity measurements, task results, PTY output) are each bounded, and the controller-side PTY buffer keeps only the most recent bytes — so a hostile agent can't bloat the DB/state files or drive controller memory to OOM. Caps are env-tunable (SYSIBLE_MAX_*_BYTES).
• No CORS is enabled; the apps are same-origin only.

Web console (BFF) specifics

• The controller API key never reaches the browser — the sysible-webgui service holds it and the SPA only ever sends {action, params, targets}.
• The administrator login token is encrypted into the session cookie with a persisted server-side key, so it survives restarts, is never stored or transmitted in the clear, and is never echoed to the client.
• Actions and terminals dispatch as the signed-in administrator (the controller derives the run-as user from the token, not from anything the browser sends), so the host's sudo policy and audit trail stay authoritative.
• A superuser can reset another administrator's password (and promote or demote an existing account's role); the target is required to change it at next login. Role-gated surfaces — administrator management, the Live Activity & Controller Log views, the Webserver Portal controls, the controller TLS-certificate install, and the controller/agent software updates — require a superuser token, enforced server-side at both the web BFF and the controller (the portal/TLS/update controller routes are require_superuser, not just API-key, so a sysadmin can't drive them by hand).
• Immediate session revocation. Removing, demoting, or password-resetting an administrator invalidates their live login token at once — the token is cross-checked against the account's current existence and role on every request, and those actions also purge the target's tokens. A fired or demoted admin therefore loses access immediately, not whenever their token would have expired. Attribution for admin-account changes is taken from the acting superuser's validated token, never a client-supplied field.
• File transfer honours the run-as model. Uploads/downloads to an agent host go through the agent as the operator's own account (bounded to a small in-task size); to a pure-SSH host they use SFTP with the shared controller key (the SSH login user, often root) and are therefore superuser-only, checked before the request body is buffered. Every SSH transfer is attributed in the activity feed.
• Read-only Auditor role. An auditor account has oversight without any ability to act. Enforced server-side, independent of the UI: the controller refuses to queue any host task for an auditor token (queue_agent_task → 403), the web BFF rejects every write/dispatch route (tool runs, fleet actions, check-in, SSH enroll, user/service/package ops, file transfer, the terminal websocket) via a require_operator gate. Auditors may read the activity log, fleet health, posture/compliance, and metrics.
• Read-only health/posture/metrics dispatch. The fleet-health, posture, and on-demand metric probes are dispatched without an operator identity (no run-as token) and run as the agent itself — they are non-mutating, unattributed, and need no local account, which is what lets the read-only auditor view them. They reuse the same cmd_* builders and are read-only by construction.
• Controller & agent self-update. Update controller (superuser-only) launches the in-place redeploy as a transient systemd unit (its own cgroup) so restarting the backend/web-console can't kill the updater mid-flight. Update agents (superuser-only) ships the controller's current agent.py to managed hosts over the existing authenticated task channel (no new inbound path, no SSH): the host verifies the file compiles (py_compile) before swapping it in, and restarts its own agent out-of-band so the task reports success before the bounce. The agent source carries no secrets; host secrets live in the host's own state file, never in the pushed code.
• "Become" (password) sudo passwords are kept encrypted at rest on the controller (Fernet), namespaced per administrator, and fed to sudo -S over stdin only — never on the command line, in the environment, the database, or logs.
• Pluggable key custody. The controller has a secret vault (backend/secret_vault.py) whose master key resolves, highest priority first, from SYSIBLE_SECRET_KEY (a Fernet key injected from your platform's secret store), SYSIBLE_SECRET_KEY_CMD (a command whose stdout is the key — the hook for fetching it from a KMS/Vault at startup), or a 0600 auto-generated local file (run/controller_secret.key, the zero-config default). The sudo-password store derives its key from this master key, so moving the master key off-box (env/KMS) protects those passwords too and nothing sensitive need touch the filesystem. Encryption is strict by default (SYSIBLE_SECRET_REQUIRED=1): if the configured key source fails to resolve, a write of a new secret is refused rather than stored in cleartext; set SYSIBLE_SECRET_REQUIRED=0 only for a deliberate dev/degraded setup. Existing values written before a master key was configured keep working (they are re-wrapped under the derived key on their next save).
• The terminal's "Send sudo password" button is opt-in per administrator: off by default, granted only by a superuser. The web console enforces it server-side (the inject request is refused for accounts that lack the grant), so it isn't merely a hidden button.

Destructive-action guards

• System-critical paths (/, /etc, /etc/fstab, /boot, /usr, init/systemd binaries, …) are protected from delete / unmount / fstab-removal: sysadmins are blocked outright, and a superuser is warned and must confirm. The check lives in the cmd_* builders (client/system_paths.py), so it holds for every front end and even a crafted API request — the override flag is honoured only for a confirmed superuser.
• Generated passwords always satisfy the admin password policy (the seeded/reset-admin/web generators guarantee the required character classes and length), so a generated value can't fail the policy check that follows.

Auditing

• Two logs are kept: an admin-audit log (logins, lockouts, and every administrator-account change — add/remove/role/password/sudo-grant, plus the controller self-update) and a fleet activity feed (who did what on which hosts). Both attribute the actor from the validated login token, never a client-supplied name. Opening and closing an interactive terminal is recorded in the activity feed too (attributed to the operator), and a live terminal session is bound to the operator who opened it.
• Secrets are scrubbed from stored command text. Before a command is written to the activity feed, well-known secret-bearing arguments (--password, --token, api-key, Authorization: Bearer, KEY=value forms) are redacted to ***, so a credential passed on a command line isn't persisted in cleartext.
• The activity feed is retained to a configurable depth (SYSIBLE_ACTIVITY_LOG_MAX_ROWS, ~500k rows by default; 0 disables local trimming).
• Tamper-evident hash chain. Both logs are hash-chained: each row's digest covers the previous row's digest plus its own content, so editing, reordering, or deleting a middle row breaks the chain. GET /activity-log/verify (superuser/auditor) recomputes both chains and reports the first break with its row id. When a secret-vault master key is resolvable the digest is a keyed HMAC-SHA256; without a key it degrades to plain SHA-256 (still tamper-evident, but forgeable), and verify reports keyed:false so you can tell which posture you're in. Trimming the oldest rows for retention is not a break — the chain verifies forward from the oldest retained row. Rows that predate the upgrade are hash-chained once, on first start.
  • Know the trust boundary. The keyed HMAC is only unforgeable when the master key lives off-box (SYSIBLE_SECRET_KEY / SYSIBLE_SECRET_KEY_CMD → KMS/Vault). In the zero-config default the key is a 0600 file next to the database, so a root / full-DB-write attacker can read it and recompute the whole chain — "tamper-evident against root" requires the off-box key. Deploy the recommended posture (off-box key) for genuine unforgeability. The same off-box key is what makes the at-rest agent-secret encryption resistant to a host-filesystem compromise.
  • Tail truncation. Deleting rows off the end of the chain is caught only by a high-water mark (audit_chain_head) that lives in the same database, so a full DB-write attacker can truncate and rewrite the mark together. The keyed HMAC still stops forging or reordering retained rows, but the guarantee "the newest N records can't be silently dropped" is delivered only by shipping events to an append-only external store. Forward the logs to an external SIEM/log aggregator and treat that as the authoritative record.

1. Keep the controller off the public internet. The API binds 0.0.0.0:9000 because remote agents must reach it; it cannot be localhost-only. Restrict access at the network layer — firewall port 9000, the web console's port 8800, and the portal's port (if used) to the trusted subnet or VPN that your managed hosts and admins are on. This is the single most important control. The web console is a full administrative surface, so treat reaching it the same as reaching the API: anyone who can log in can run privileged commands fleet-wide.
   • Example (firewalld): allow only a trusted subnet firewall-cmd --add-rich-rule='rule family=ipv4 source address=10.0.0.0/24 port port=9000 protocol=tcp accept' (repeat for 8800) and ensure those ports aren't open to public.
   • To bind only one interface instead of all of them, set SYSIBLE_CONTROLLER_BIND=<nic-ip> before running the installer (it fills the controller unit's --host), so the API listens on just the NIC your agents use rather than every address. Firewalling is still the primary control; this narrows the exposed surface further.
2. Use a strong, unique admin password, and rotate it if it may have been exposed. Set an admin password policy in Sysible Controller Settings. On a fresh install the web console seeds a default admin with a one-time random password printed (in red) at the end of the install — change it on first login under Settings → My Account, or reset it with sudo sysible_controller reset-admin.
3. Protect the API key file. It stays 600 root-only; don't copy it onto untrusted machines. Rotating it (delete the file and restart) invalidates every existing client until they re-read it.
4. Run agents non-root where you can. The agent supports an opt-in dedicated sysible user with scoped sudo instead of running as root.
5. Only start the Webserver Portal when you need it, and stop it when you're done — it's the most exposed surface, and it doesn't run by default.
6. Keep the host patched and limit who can log in to the controller machine itself; local root there is equivalent to controlling the whole fleet.
7. Serve the web console over TLS. Its session cookie is marked Secure only when TLS is on (SYSIBLE_WEBGUI_HTTPS_ONLY=1, set automatically by the installer's TLS setup) — don't run it over plain HTTP. If you front it with a reverse proxy, set SYSIBLE_WEBGUI_TRUSTED_PROXY=1 so the login throttle reads the real client IP from X-Forwarded-For; leave it unset for a direct bind so a client can't spoof that header.
8. SSH host-key verification is trust-on-first-use. The controller pins each SSH host's key on first contact and then verifies it on every later connection, refusing a changed key (see Controls in place). Because the first contact establishes that trust, perform the initial enrollment of an SSH host over a trusted network. If a host is legitimately rebuilt, remove and re-add it (which forgets the old pinned key) so it can re-pin.

The web console is the administrative interface, and it is built so that providing a browser UI does not widen the trust boundary beyond the API that already had to be network-reachable:

• It exposes no new privilege: signing in requires a controller administrator account, and every action runs as that administrator with that user's sudo rights.
• The admin API key stays server-side (BFF); a compromised browser session is bounded by the signed-in administrator's role and the host sudo policy, not by possession of the key.
• It is hardened for network exposure (session cookie, CSRF-closing SameSite=Strict, login throttle, strict CSP / anti-clickjacking headers, API docs disabled).

Treat it like the API: firewall its port to a trusted subnet/VPN, and if you don't use it, don't run it.

These are intentional trade-offs or scale boundaries, not defects. Know them before a production rollout:

• Replacing the TLS certificate needs a fleet trust-bundle refresh. Enrolled agents pin the controller's certificate. Installing a new/renewed cert and restarting the controller will break every already-enrolled agent's connection until the new Trust Certificate (Settings → TLS) is redistributed to each host out-of-band (re-run the agent bundle, or copy it to the pinned cert path). The console warns and requires confirmation before replacing the cert. Plan a maintenance window and stage the bundle push.
• The activity/audit log is a local convenience record, not a system of record. It lives in the controller's SQLite DB: it has no hash-chaining or WORM guarantee, and anyone with DB/filesystem (root) access can alter it. Command text is scrubbed of well-known secret-bearing arguments before storage, but for tamper-evident, durable retention forward events to an external SIEM.
• Single-node SQLite has a write ceiling. Every agent heartbeat is a small serialized write. One controller node comfortably handles a modest fleet; for very large fleets, raise SYSIBLE_POLL_INTERVAL/SYSIBLE_METRICS_INTERVAL to cut write pressure, and treat the single-writer DB as the scaling limit when sizing. Backup/restore is file-level (stop the service, copy sysible.db* and the keys/certs, restart).
• Enrollment tokens are bearer credentials. A token grants enrollment on its own (no second factor). Within the reuse window a token can re-enroll the host it was bound to; the controller refuses to re-enroll a host that is still live or was explicitly revoked, but treat bundles/tokens as secrets and revoke a host to invalidate its access.
• Scheduled jobs run in the target host's local timezone, not the operator's browser — cron and systemd timers use the host clock. Set each host to the timezone you expect maintenance windows in.
• Agent-channel payloads are size-capped (metrics/snapshot/measurements, task results, PTY output/buffer) to bound controller memory against a misbehaving agent; the caps are generous but env-tunable (SYSIBLE_MAX_*_BYTES) if a legitimate payload is ever truncated.

Report suspected security issues privately to the Sysible maintainers rather than in a public issue. Include the version/commit, affected component, and reproduction steps.