Skip to content

PostgreSQL & Native install as optionals

Latest

Choose a tag to compare

@btouchard btouchard released this 28 Aug 23:04
· 2 commits to main since this release
864d69e

Lets the server keep its data on a PostgreSQL you already run, so an instance watching a fleet is no longer tied to the machine it runs on. SQLite stays the default, untouched, and the only storage an agent will ever use. This is also the first release to ship a binary, with a script that installs it as a systemd service on a host with no Docker at all. Plus a resource history that every edition can see, recovery notifications that read as recoveries, a per-trigger opt-out for them, on-demand checks for endpoints and certificates, and fixes for issues reported from real multi-host installs.

PostgreSQL as an optional storage

The server holds one class of data the fleet cannot rebuild: agent identities and their enrolment. Kept in a file on the machine that runs the process, that data makes the instance irreplaceable: lose the machine and every host has to be re-enrolled by hand, while monitoring is blind. Pointing the server at a PostgreSQL database removes that. A replacement started elsewhere with the same connection string picks the fleet back up, and the agents reconnect with the identity they already had.

One setting, MAINTENANT_DATABASE_URL (or --database-url), in server or embedded mode, any edition, PostgreSQL 14 or newer. Without it, nothing changes: no new message, no new step, SQLite exactly as before. An agent handed the setting refuses to start rather than ignoring it; an agent always stores its state locally.

The product provides one property, keeping nothing essential on its own machine, and leaves the rest where it belongs: backups, replication, and detecting a dead instance to restart it elsewhere are the operator's. There is no leader election and no mutual exclusion. Instances register in a table and beat; a second one on the same database is reported (peers in /api/v1/health, a warning in the log), never arbitrated.

What it does when things go wrong

  • A misconfigured database stops the boot, with a message naming the cause among five an operator can act on: an unreadable connection string, an unreachable host, refused credentials, an unsupported version, a schema written by a newer release. There is deliberately no fallback to the local file, which would look like it worked and silently strand the fleet.
  • An outage while running is survived, not escalated. The instance stays up, the pool renews its connections, and it recovers with no restart. Meanwhile the interface shows a STORAGE OFFLINE banner and keeps what it already knew instead of emptying out, API reads that need the database answer 503 STORAGE_UNAVAILABLE, and a new storage.availability_changed SSE event marks each transition.
  • /api/v1/health keeps answering 200 during that outage, with storage.connected: false. It is the target of the Kubernetes liveness and startup probes, and failing it on a ten-second blip would restart the instance exactly when the database needs to be left alone. Do not wire the probe to the database.

Credentials

The connection string never reaches the logs (at any level, wrapped errors included), the API, the interface or the telemetry; where a target must be named it appears redacted, postgres://maintenant@db.internal:5432/maintenant. A test injects a sentinel password and fails on any output containing it. Towards a non-local host, sslmode=require is added when the string carries none, so an external database is reached over TLS by default; an explicit value always wins.

Moving an existing install

maintenant --copy-store-to <dsn> carries an install from the local file into an empty PostgreSQL database without touching a single monitored machine. One rule decides what travels: a table goes if its content cannot be reproduced without human intervention. Agent identities and enrolment tokens, declared monitors, channels and their secrets, triggers, silences, escalation policies, webhook subscriptions, per-container thresholds, the whole status page including its subscribers, published incidents, maintenance windows, and operator decisions (update exclusions, version pins, acknowledged findings). Everything the fleet rebuilds by itself stays behind: inventories, check and resource history, active alerts, CVE and image-update intelligence.

The command says what it will and will not carry, with counts, before writing anything, and names the two effects you would otherwise discover afterwards: an alert acknowledged before the copy comes back unacknowledged if still active, and curves start from zero. Schema installation and copy share one transaction, so a failure half-way rolls the schema back with the data and a retry works. The source is opened read-only. A non-empty target is refused before anything is created. Row counts are compared table by table before the command reports success. --yes skips the prompt for scripts.

The PostgreSQL guide shows the whole thing in Docker Compose: a one-shot copy-store service next to the instance, the connection string held once in a YAML anchor so the instance and the copy cannot drift onto two databases, and three commands: stop, run, up.

Two files stay in the data directory rather than in the database, the signed licence cache and the update window record, so keep the /data volume with the instance when it moves.

Under the hood

One storage package, two dialects. Every engine difference goes through a Dialect, and there are six of them (placeholder syntax, batched deletes, opening PRAGMAs, error classification, write serialization, SQL-side UUID generation); the fifty or so query files do not know which engine they run on. PostgreSQL starts from a baseline numbered at the SQLite head (28); from 29 on, a migration is written for both engines under the same number or not at all, enforced by a test that migrates a fresh database on each engine and compares the two heads, column by column. The driver is pgx, pure Go: go-sqlite3 stays the only CGO dependency, and CI now fails if another one appears.

CI runs the store, app, API, agent-server and alert suites on both engines, the store package on PostgreSQL 14, 16 and 17, and an end-to-end stack that builds the image and drives the real product on both engines with the same 16 checks, migration included. MAINTENANT_REQUIRE_POSTGRES=1 turns a skipped PostgreSQL test into a failure, so a green run cannot mean "tested nothing".

Native install, without Docker

Until now the only way to run maintenant was a container. Releases carried a provenance
file and an SBOM, and no binary. Some hosts have no container runtime, and a monitoring
tool that requires the thing it monitors is a poor fit for a bare VM or an appliance.

curl -fsSL https://install.maintenant.dev | sudo bash

That installs the binary in /usr/local/bin, creates a maintenant system user, writes
a hardened systemd unit and starts it. Configuration goes on the same line, and lands in
/etc/maintenant/maintenant.env:

curl -fsSL https://install.maintenant.dev | sudo bash -s -- \
  --addr 0.0.0.0:8080 \
  --baseUrl https://monitoring.example.com \
  --organisationName "Acme Corp"

The file is plain KEY=value, loaded by the unit through EnvironmentFile=. Editing it
and restarting the service is a supported way to reconfigure: re-running the installer is
not required, and a second run merges key by key instead of overwriting what is there.
--no-service installs the binary alone, --uninstall removes it while keeping data and
configuration, and --purge alongside it removes those too.

The binaries are statically linked against musl, so one file per architecture (amd64,
arm64) runs on Debian, Ubuntu, RHEL, Rocky, Alpine or Arch, with no libc version to match
and nothing to install next to it.

Every setting is reachable from the command line. Each MAINTENANT_* variable now has
a --flag equivalent, from one registry that also generates --help, so the two can no
longer describe different things: two tests fail the build if the registry and
.env.example disagree in either direction. Precedence is flag, then environment
variable, then default. --version reports the version, the commit and the build date.
Six settings the binary had always read, among them the MCP OAuth redirect URIs and the
per-agent rate limit, had neither a flag nor a line of documentation; they have both now.

A remote host is enrolled the same way, with --mode agent --server ? --enrollment-token ?,
which gives an agent on a machine that runs no containers but has endpoints, heartbeats or
certificates worth watching.

What the installer verifies

The binary is checked against SHA256SUMS before anything is installed, and that is
mandatory: a mismatch aborts with exit code 21. If cosign is on the path, the checksum
file's keyless Sigstore signature is verified against the transparency log, asserting it
was produced by this repository's release workflow on the tag being installed. Absent
cosign, the script says so and continues; --skip-cosign states the intent explicitly.
Releases also carry a SLSA build provenance attestation for the binaries.

The script itself ships as a release asset, checksummed with the binaries and stamped with
the tag it came from, and that tag is written into the header of every
/etc/maintenant/maintenant.env it produces, so a file on disk always names the script
that wrote it. MAINTENANT_VERSION=v1.4.0 installs a specific release rather than the
latest one; asking for a version that predates the binaries fails with a message saying so
rather than a confusing 404.

The script is POSIX sh, verified by shellcheck and covered by 42 tests that run in CI on
every change to it, and a smoke job installs from the published release on both
architectures before anyone else does.

Recovery notifications, and the right to skip them

When a heartbeat recovered, channels received the original alert record: failure message, critical severity, red colour, with only event: alert.resolved telling it apart. Some heartbeat alerts also never resolved at all. (#67)

  • Recovery notifications are still routed by the original alert, so a trigger filtered on critical keeps receiving them, but carry the recovery message and info severity; id, resolved_at and resolved_by_id are unchanged.
  • Heartbeats emit the recovery after a down, start, finish sequence, which used to leave the alert state stuck on alerting. A ping reporting a non-zero exit code now fires exit_code_failure, with a matching message, instead of being typed as a missed deadline. Deleting or pausing a heartbeat clears its active alerts, as does deleting an endpoint or a certificate monitor.
  • The startup sweep that resolves alerts whose entity no longer exists now covers heartbeats, endpoints and certificate monitors, and runs before the container runtime connects, so alerts orphaned by an earlier version are cleared on the first restart.
  • MCP list_alerts: active_only: false was indistinguishable from the zero value and always returned active alerts. It now returns resolved ones as asked.

Triggers gain notify_on_resolve (boolean, default true). Set it to false and the trigger relays failures only; fires are unaffected. Exposed on the REST API and the MCP tools, as a "Notify on recovery" toggle in the trigger editor, and as a "fires only" badge in the list.

Check an endpoint or a certificate now

A certificate monitor is scanned once a day, so a certificate renewed by hand kept alerting until the nightly run came round. (#44) POST /endpoints/{id}/check and POST /certificates/{id}/check run the check immediately, replaying the scheduler's own path: same persistence, same status, same alert evaluation, and the next scheduled scan pushed a full interval out. Each detail panel gets a Check now button. Only monitors the server probes itself qualify: an agent's targets are unreachable from here and it re-probes them every 30 to 60 seconds, so the button is hidden and the API answers 409. One check at a time per monitor.

This surfaced a dead broadcast: previousStatus was read after the scan had already overwritten it, so certificate.status_changed never left the server, leaving the certificate list and the webhook subscribers on an event that never fired.

Resource history, tiered instead of gated

The resource charts were all-or-nothing: the history endpoint sat behind a capability whose minimum edition was Personal, so a Community instance saw no chart at all, not even the last hour, while Personal and Pro opened exactly the same windows the pricing page claimed were different. The top consumers endpoint accepted its 30-day period with no edition check at all, so the split was decoration on the interface rather than a rule the product enforced.

Every edition now sees a history; what an edition buys is how far back it goes. The cap is a duration, declared once: Community 7 days, Personal 30 days, Pro 90 days. Community gets the windows it never had, up to a week, and Pro gains a 90-day window. Seven days is where the free edition stops being a demo: a shorter cap cannot answer "was it like this last night, or last Friday", which is the first question of any diagnosis. What an edition buys is the trend over a month and a quarter, not the ability to look at yesterday.

Window Edition Served from Granularity
1 hour Community raw samples raw
6 hours Community raw samples 1 minute
24 hours Community raw samples 5 minutes
7 days Community hourly rollup 1 hour
30 days Personal hourly rollup 1 hour
90 days Pro daily rollup 1 day

Each window reads a table kept strictly longer than the window itself, so a retention pass can never shorten a chart mid-read: that is why 90 days comes from the daily rollup, kept a year, and not from the hourly one, kept exactly ninety days. The daily rollup carries CPU, memory and network but not the block I/O counters, so Disk I/O reads zero on the 90-day window only; every shorter window has it. Adding it would need a migration.

The catalogue and the cap travel beside the capabilities in GET /api/v1/edition, so the interface holds no edition table of its own. A window above the cap is refused with 403 EDITION_REQUIRED, naming the edition that opens it, the window and your current cap; it is never silently shortened to the cap. An unknown window is a 400: a bad request, not an edition question. Per-container charts, /resources/top and the get_top_consumers MCP tool resolve through the same registry, so no window is served by one surface and refused by another. In the interface, the window selector marks closed windows with the edition that opens them and fires no request for them, and a refusal on a window that closed mid-session falls back to the largest one still open.

Two things fixed on the way: get_top_consumers passed current straight to the store, a value it has never known, so the MCP tool failed on every call made without a period; and /resources/top now accepts the same 6h and 90d as the charts.

A feedback link

The Editions page and the header now link to maintenant.dev/feedback, a two-minute form read by the developer and quoted only if you allow it. It is a plain link: the instance sends nothing, your browser opens the page with the edition and version prefilled. The telemetry contract is unchanged.

Bug fixes

  • A floating tag was read as a pinned version. (#62) traefik:v3 parsed as the semver 3.0.0, so every v3.x.y in the registry looked newer and the scanner proposed installing the release the container was already running. Partial tags (fewer than three numeric components) now follow the same path as latest and lts: the registry keeps moving them, so the digest is the signal. A higher tag of the same shape, v3 to v4, is still reported as a major upgrade.
  • An agent could go missing unnoticed. (#54) The disconnect alert only fired on a connect/disconnect transition, so an agent that went down while the server was stopped, or during the shutdown storm where alerts are suppressed on purpose, stayed silently disconnected. The stale watcher now derives the offline state instead of catching the transition, announcing each outage once, with a two-minute startup grace so agents merely reconnecting do not page. The agent source was also missing from the trigger editor and the Alerts filter, so the alert could not reach a channel.
  • Every agent container reported unhealthy. (#58) The image probed http://localhost:8080/api/v1/health, but an agent serves no HTTP at all, and a server bound to another address failed the same probe. A healthcheck subcommand reads a liveness file the agent refreshes every 15 seconds, and falls back to the health endpoint on the configured MAINTENANT_ADDR otherwise. Reachability of the server is deliberately not part of the signal: restarting an agent that cannot reach it would fix nothing, and the outage is already reported server-side.
  • One-off containers minted duplicate monitors. docker compose run copies the service definition onto a throwaway container, labels included, under a generated name; since a label monitor is keyed on the container name, the one-off created a second monitor on the same target, and that monitor outlived the container. Reported from a storage migration that left an endpoint stuck Down on the URL a live endpoint reported Up. Anything stamped com.docker.compose.oneoff=True is now skipped by discovery and the event stream. Startup reconciliation retires label endpoints whose container is gone, since a destroy event arriving while the instance is down is an event nobody hears. A retired endpoint can now be deleted; the list hides them behind an Include retired filter and leaves them out of the counters. A live label endpoint still cannot be deleted (409 ENDPOINT_LIVE), as the next discovery pass would recreate it.
  • One instance could rewind another's migration. The migrator read the version and recovered the dirty flag before taking its lock, so an instance starting while another one migrated took that instance's in-flight flag for a crash and forced the version backwards under it, leaving every later start on "no migration found". The whole read, recover, apply sequence now runs under a single lock. The recovery target was wrong on its own too: an install interrupted on its very first migration recovered to a version no file describes and stayed broken.
  • A clean shutdown left the instance registered, so a plain restart reported itself as a second instance for five minutes, warning included.
  • PostgreSQL rollups never landed. Averaging mem_limit or the cumulative network counters overflowed with integer out of range, since PostgreSQL INTEGER is 32-bit, so the hourly and daily aggregates were never written and every window served from them came back empty. SQLite hid it: its INTEGER is 64-bit. The averages are cast to BIGINT.
  • An agent could write to its data directory after shutdown. The liveness reporter ran in a goroutine nobody could wait for, so a tick firing at the same moment as the cancellation could recreate the liveness file after the agent believed the reporter had stopped. It is now waitable, and the agent waits on it before returning.
  • Go toolchain bumped to 1.26.6 and golang.org/x/net to v0.56.0 (CVE-2026-46600), clearing every advisory on the CI gates; the frontend lockfile refreshed for one advisory in build tooling, none of it shipped in the bundle.

Upgrade notes

  • Automatic DB migrations 28 (notify_on_resolve on triggers) and 29 (instances table). Back up the .db first as usual.
  • A binary older than the schema it opens now refuses to start, on SQLite too. This is the one behaviour change for an existing local install, and it only triggers on a downgrade: rolling back to 1.3.9 after this upgrade means restoring the backup, where it used to mean writing silently into a schema the older binary did not know.
  • Nothing to do to stay on SQLite. PostgreSQL is opt-in by connection string; without one, the instance behaves exactly as before.
  • Moving to PostgreSQL: stop the instance, run the copy, restart with MAINTENANT_DATABASE_URL set. The agents reconnect on their own. Keep the /data volume with the instance, and keep the Kubernetes probes on /api/v1/health as they are: it answers 200 through a database outage on purpose. The Helm chart and the raw manifest carry a commented-out database secret to fill in.
  • Alerts left behind by 1.3.8 or 1.3.9 (heartbeats, endpoints, certificates whose entity is gone) are resolved on the first restart.
  • Agent containers report healthy with the stock image healthcheck; a compose override that disabled it can be removed.
  • Resource history needs nothing done: no migration and no retention change. A Community instance gains its charts on upgrade, up to 7 days; a Pro instance gains 90 days as the daily rollup fills up.
  • Triggers keep relaying recoveries unless you set notify_on_resolve: false; a create or update that omits the field leaves it at true.
  • Nothing changes for a container install. The native installer is a new way in, not a replacement: same binary, same image, same configuration.
  • The standalone tab of the enrolment modal still reads "Coming soon", on purpose. It will hand out the installer command once this release is out and the host serving it is in place; the Docker, Compose and Kubernetes tabs are unaffected.

Pull the new image to upgrade, or install natively with the command above.