Alerts and Digest

Alerts and Digest

Trove can send instant notifications for important transitions and scheduled email digests summarising the fleet.

Alerts are configured on the server. Agents do not send notifications.

Instant channels

Trove can fan out alerts to multiple channels.

Channel	Variable
Generic webhook	TROVE_ALERT_WEBHOOK_URL
Discord webhook	TROVE_ALERT_DISCORD_URL
ntfy	TROVE_ALERT_NTFY_URL

Optional ntfy auth:

TROVE_ALERT_NTFY_TOKEN

If one channel succeeds and another fails, Trove records the successful delivery and retries only the failed channel. This avoids losing a notification to one destination or duplicating it to destinations that already accepted it.

Generic webhook payload

The generic webhook receives JSON:

{
  "kind": "health",
  "level": "critical",
  "title": "gitea unhealthy",
  "body": "gitea @ nuc01: health healthy -> unhealthy",
  "host": "nuc01",
  "service": "gitea",
  "from": "healthy",
  "to": "unhealthy",
  "at": "2026-07-03T10:15:00Z"
}

Levels:

• info
• warning
• critical
• resolved

Sign generic webhooks

Set TROVE_ALERT_WEBHOOK_SECRET to add HMAC-SHA256 authentication headers:

TROVE_ALERT_WEBHOOK_SECRET=WEBHOOK_SIGNING_SECRET_VALUE

Signed requests include:

• X-Trove-Timestamp: Unix timestamp
• X-Trove-Signature: sha256=<hex hmac>

The signature payload is timestamp + "." + raw_json_body. Verify it with the shared secret before trusting the body, and reject old timestamps if replay protection matters to your receiver.

What triggers alerts

Kind	Fires on	Level
agent	agent stale/offline and recovery	warning, critical, resolved
health	service unhealthy and recovery	critical, resolved
state	service stopped/failed/removed/degraded and recovery	warning, resolved
freshness	running image behind registry tag and recovery	warning, resolved

All four are enabled by default.

Limit event types with:

TROVE_ALERT_EVENTS=agent,health

Noise control

Trove avoids notification spam by design.

Transitions only

An ongoing bad state does not repeatedly alert by itself.

New services do not alert

A new service appearing is feed-only. Deployments should not page you.

Cooldown

TROVE_ALERT_COOLDOWN defaults to 5m.

Cooldown is per incident key. Repeated flaps inside the window are suppressed.

Escalations can bypass cooldown once. For example, an agent going from stale to offline can send a warning and then a critical alert close together.

No boot flood

The alert engine seeds from current state. A fresh Trove install should not immediately notify for every existing problem it discovers.

One agent outage, not fifty service alerts

When an agent goes offline, Trove sends the agent alert. It does not emit one health event per service just because the agent stopped reporting.

Alert state model

Trove tracks alert state per incident key in alert_state.

The critical design detail is that notified is a separate boolean.

That means Trove distinguishes:

• the current observed bad value
• whether a human was actually notified for that bad value

This prevents a class of bugs where reconnects, stale mass-updates, or freshness unknown blips erase delivery state and cause resolved notices to disappear.

Successful fan-out deliveries are tracked per incident and channel in alert_channel_deliveries while any configured channel still needs a retry. The temporary records are removed after every channel succeeds.

Email digest

The email digest summarises:

• service counts
• unhealthy services
• outdated images
• agents not reporting
• activity since the last digest

Configure SMTP on the server:

TROVE_SMTP_HOST=smtp.example.com
TROVE_SMTP_PORT=587
TROVE_SMTP_USERNAME=nick@example.com
TROVE_SMTP_PASSWORD=app-password
TROVE_SMTP_FROM=trove@example.com
TROVE_SMTP_TO=nick@example.com
TROVE_DIGEST=daily@08:00

Required for digest:

• TROVE_SMTP_HOST
• TROVE_SMTP_FROM
• TROVE_SMTP_TO

Username and password are optional for open relays.

Digest schedule

Supported examples:

daily@08:00
weekly@mon:08:00
off

Times use the server's local timezone.

If the server was down during a scheduled slot, Trove sends once on the next check. It should not double-send.

Test alerts

Run:

trove-server alert test

With Compose:

docker compose exec server trove-server alert test

The command sends a test notification through configured instant channels and a sample digest if SMTP is configured.

If it reports no channels configured, the environment variables did not reach the server process.