Repository navigation
maintenant 1.12.0 keeps every notification until it is delivered, raises the alerts that outlast a silence or a maintenance window, sends escalation acknowledgments as acknowledgments, keeps update and security data per agent, and counts endpoints and certificates the same way whether they come from the interface or from container labels. Some behaviours change for webhook receivers, MCP and API clients, and installs behind an outbound proxy: read Before you upgrade first.
Before you upgrade
- Escalation acknowledgments on webhooks. The acknowledgment notice of an escalation policy arrives as
"event": "alert.acknowledged", withalert.statusstillactive, instead ofalert.resolved. A receiver that treats every event other thanalert.resolvedas a firing alert must ignorealert.acknowledged. By email, the subject reads[maintenant] ACKNOWLEDGED: <message>. Webhook subscriptions are unchanged. - Outbound proxy. Webhook subscriptions, notification channels and outbound heartbeats connect directly to their destination and do not use
HTTP_PROXYorHTTPS_PROXY, so the address they reach is the one that was checked. An install that reaches the internet only through such a proxy must allow these direct connections. - Community: endpoints and certificates declared by labels. Community includes 10 endpoints and 5 certificate monitors, and counts them the same way whether they come from the interface, the API or a container label. Everything monitored before the upgrade stays monitored. See Community edition below.
- MCP read tools.
list_heartbeatsdoes not return ping tokens, andlist_endpointsmasks the values of probe headers. The REST API is unchanged. - Channel URL. Changing the host of a channel URL without sending
headersclears the stored headers, so they only ever go to the destination they were set for. Send the headers in the same request to keep them. The channel form says so. - Request validation. An escalation policy is refused with
400(validation_failedover MCP) when a level names an unknown channel id, an empty or repeated one, or when a filter uses a severity other thancritical,warningorinfo. An alert rule refuses an unknown severity in its filter and a repeatedchannel_idsentry. A status component refuses a repeated monitor. - Same container id on several agents. Update, CVE and risk routes take an optional
agent_id. When several remote agents report the same container id and none is named, the request answers400 AGENT_REQUIREDand changes nothing. Nothing changes with a single agent. - Request sizes and rate limits.
/mcpbodies are limited toMAINTENANT_MAX_BODY_SIZE(1 MiB by default),/oauth/authorizeand/oauth/tokento 64 KiB. An IPv6 client is rate-limited under its /64. - Notifications pending at upgrade. Notifications the previous version had not finished sending are closed during the upgrade, so nothing old goes out afterwards.
Notifications
- Each notification is stored, then sent at once. One that could not go out stays stored and is picked up within 5 seconds, after a restart too.
- A failed attempt is retried 5 seconds later, then 10, 20, 40 and so on, the wait doubling up to one hour. A notification still undelivered 24 hours after it was created is recorded as
failed. - A refusal that will not change, an HTTP 4xx other than 408 and 429, ends the delivery at the first attempt. 5xx answers and network errors are retried.
- Webhook, Slack, Teams and Discord requests carry
X-maintenant-Delivery, the id of the delivery. It stays the same from one attempt to the next, so a receiver can drop a copy. - The notifications of an alert reach a channel in order: a recovery waits for the alert it closes. Of several pending reminders of one alert, only the latest is sent.
- An attempt is sent once, even by several instances sharing one PostgreSQL database.
- A channel waiting for a retry shows
health: failing. Disabling a channel cancels what it still had to send. - Webhook subscriptions have their own queues and workers, apart from alert notifications.
Alerts
- Silences and maintenance windows. An alert raised during a silence or a maintenance window is held. If its cause is still there when the window ends, the alert becomes active, exactly as if it had just fired: notifications go out and escalation starts, once. A recovery during the window closes it quietly. Held alerts are re-evaluated every 30 seconds, and those of the last 24 hours are held again after a restart. During the window, a repeated event adds no entry to the alert history.
- Certificate reminders. A certificate alert notifies again at each threshold (30, 14, 7, 3 and 1 days by default), with the days left in the message. An acknowledged alert is refreshed without a new notification.
- Container health. The alert is raised on any change to
unhealthy, including for a recreated container that fails its healthcheck from the start, and cleared on any change tohealthy. - Recreated containers. The endpoint of a recreated container starts from a clean alert state, so a service that is still down alerts again.
- Docker signals. A
killevent does not mark a container as exited: a process that survives the signal, such as nginx reloading on SIGHUP, staysrunning.diereports the actual exit. - Resource alerts. Disabling the resource alert configuration of a container resolves its active alerts.
- Database outages. The alert engine retries its database writes and its rule and channel lookups three times (at once, after 100 ms, after 500 ms) before giving up.
Escalation
- The acknowledgment notice is an acknowledgment on every channel:
alert.acknowledgedon webhooks,Acknowledged: <entity>on Slack, Teams, Discord and Telegram (Telegram also says who acknowledged and when),[maintenant] ACKNOWLEDGED: <message>by email. - A level counts as sent once each of its channels has a delivery. If the database is unavailable while a level goes out, the level stays due and is tried again at the next evaluation, without notifying a second time the channels it already reached.
- No level goes out once the acknowledgment or the resolution is recorded, even when an evaluation is under way at that moment.
- A run that could not be recorded when the alert fired starts at the next evaluation, with the times it would have had. A policy runs at most once per alert.
- A channel deleted since the policy was written is recorded as
failedwith the errorchannel deleted: <id>, and the run goes on with the next level. - Escalation follows the edition at every call: a license activated on a fresh install takes effect without a restart.
GET /api/v1/escalation-runs/{id}gives thechannel_nameof each delivery. Over MCP,create_escalation_policyreturns itsoverlapping_warningsand accepts theinfoseverity.
Heartbeats
- After a restart, the deadlines that passed while maintenant was stopped are pushed back by one interval plus the grace period, so a heartbeat is only judged on pings the server could receive.
- A ping and a deadline check that land at the same moment are applied one after the other: the ping wins, and a ping on a
downheartbeat always emits the recovery. - The list follows live pings: last ping, next deadline and duration update without a reload.
Containers
- The local Docker or Kubernetes runtime is reconciled every 30 seconds, health, replica counts and error detail included. A health change that happened while maintenant was stopped is picked up on the first pass.
- Kubernetes Deployments, StatefulSets and DaemonSets take their state and counts from this pass. A new controller appears within one pass.
- When the Docker API cannot be reached, the error says how to mount the socket or set
DOCKER_HOST.
Updates
- Image variants are matched:
3.14.2-slimis offered a newer-slim, and the same goes for-fpm,-trixie,-alpine3.22or any suffix that starts with a letter. A tag without a variant is only offered tags without one. A tag whose suffix changes at every build (1.26.2-r0-ls290) gets no proposal: usemaintenant.update.tag-includefor those. - An image is up to date when any of its repo digests is the one the tag points at. This covers images pulled again after their registry republished the tag, such as
postgres:18ormariadb:13. - Registry calls have a response timeout and each image has a time budget, so a silent registry does not hold the scan.
- Vulnerability lookups know
trixie,alpine3.22andalpine3.23.
Multi-host
- The server acknowledges an agent event once it is stored. When its database cannot store one, it closes the stream without acknowledging it, and the agent reconnects and resends from that event. This covers container events, probe results, resource samples, certificate scans and heartbeat pings. A replayed event rewrites the same row.
- On shutdown, the server closes the agent streams before its database.
- An event is accepted up to 5 minutes ahead of the server clock, the same tolerance as the handshake. Refused events are counted per agent in
rejected_events(GET /api/v1/agents, MCPlist_agents). - Update, CVE, version pin and risk data are kept per agent. Two clusters that run a workload under the same name each have their own pins, scores and acknowledgments, and
?agent_id=names the agent (localfor the server). - An agent whose on-disk spool cannot be opened sends its buffered events from memory, within its memory budget.
- An agent restarts its collector after an error, with a backoff.
- The dashboard reloads containers and endpoints when an agent connects or disconnects.
The last two agent items are in the agent binary: upgrade the agents with the server.
Community edition
- Community includes 10 endpoints and 5 certificate monitors. They are counted the same way whether they were added from the interface, through the API or by a container label. Auto-detected certificates do not count, and neither does a declared certificate on the same host and port as a monitored HTTPS endpoint.
- Everything monitored before the upgrade stays monitored, beyond these numbers too, and across container recreation.
- A label endpoint or certificate beyond the count is listed as Not monitored, with
limited: true. It is not checked and raises no alert, and an on-demand check on it answers403 QUOTA_EXCEEDED. It starts being checked as soon as a slot frees up, oldest first. A destroyed container frees its slot at once. - An edition change takes effect within a minute, without a restart.
- The Editions page, the edition API and the MCP
get_editiontool show usage per resource, withunmonitoredfor endpoints and certificates. Two SSE events follow the changes:endpoint.limit_changedandcertificate.limit_changed. - Email and Telegram channels and resource history beyond 7 days say which editions include them.
License
- A license refusal (
401or403) is applied only when it carries the signature of the license server, like a valid answer. Any other401or403, from a proxy or a firewall on the way for instance, is handled like an unreachable server: the cached license is kept. - Maintenance windows and update enrichment follow the edition at every call, so a license that starts or ends takes effect without a restart.
API and MCP
- A storage outage answers
503, a name already taken409(renaming a channel to an existing name, for instance), and anything else a generic500with the detail in the server log. MCP tools follow the same rules. - Alert rules, incidents and their updates, maintenance windows and status components are written in one transaction: a request either applies entirely or not at all.
- Alert rules created, updated or deleted through MCP broadcast the same live events as through REST.
get_updatesreturns changelog fields when the edition includes them, like the REST API.- The MCP security tools take the internal container id their description gives.
- A resource alert configuration saved without
enabledkeeps the stored value. - Pinning a version reports a write that failed instead of answering
200.
Interface
- Live updates run on every page for the whole session, and every page reloads its data when the stream reconnects.
- The dashboard has a Data unavailable state: when a list cannot be loaded, the banner names it instead of giving a verdict, and "updated" is only stamped when everything loaded.
- The service worker caches static assets only. The dashboard always shows what the server just answered.
- Editing a maintenance window keeps its times as entered, in the time zone of the browser.
- Creating a public incident, scheduling a maintenance, creating a silence, deleting or testing a channel and pinning a version show the error returned by the server, and ignore a second click while the first request is pending.
- The public status page recovers by itself after a failed load: it reloads when its stream reopens and refreshes while it is visible.
- The interface keeps the last known edition when reloading it fails.
Storage
- The alert retention purge runs in batches on a new index, and endpoint sparklines read only the last points of each endpoint. Both stay fast on large histories, and sparklines are served on every supported PostgreSQL version, 14 and 15 included.
- Incidents and maintenance windows are listed with one query per relation, which keeps the public status page responsive under concurrent requests.
- After a SQLite to PostgreSQL copy, a failure still in progress is raised again on the new database: the copy resets the alert state of endpoints, heartbeats, certificate reminders, resource thresholds and container health. Acknowledgments have to be redone.
- On shutdown, maintenant ends streaming requests at once, then waits for its background tasks and its pending notifications before closing the database.
Deployment
- Portainer template: a field turns the anonymous telemetry off, and a note says what is sent and which hosts maintenant contacts. The stack passes
MAINTENANT_DISABLE_TELEMETRYandMAINTENANT_ALLOWED_HOSTS, and follows the major image tagghcr.io/kolapsis/maintenant:1. - maintenant is built with Go 1.26.11. Building from source needs at least that version.
Upgrading
Migrations 44 to 47 run at startup on SQLite and PostgreSQL:
- 44 records the endpoints and certificate monitors that exist at upgrade, so they stay monitored.
- 45 adds an index on
alerts.resolved_by_id. - 46 adds
agent_idto the update, CVE, version pin and risk tables. Existing rows are kept for every agent that reports the container. - 47 adds the retry time, the event type and the payload to
notification_deliveries.
Migration 46 rebuilds its tables: back up the database before upgrading.
The Helm chart moves to 1.12.0, the same as the application version.
The full documentation is at docs.maintenant.dev.