Events and State Flow

Events and State Flow

Trove's event stream is the bridge between inventory, dashboard activity, alerts, and digests.

The current catalog lives in services, hosts, and agents.

The history of meaningful transitions lives in events.

Event kinds

Kind	Scope	Example
state	service	running -> exited, running -> removed
health	service	healthy -> unhealthy, stale -> healthy
agent	agent	ok -> stale, stale -> offline, offline -> ok

Service appearances

When a new service appears, Trove records a state event with an empty from_state.

from_state = ""
to_state = "running"

This is useful for the activity feed.

The alert engine does not treat new appearances as incidents. Otherwise every deploy would become an alert storm.

State changes

When a known service changes state, Trove records a state event.

Examples:

• running -> exited
• running -> stopped
• running -> failed
• running -> removed
• removed -> running

State events can become warning alerts depending on the target state and configured alert event types.

Health changes

When a known service changes health, Trove records a health event.

Examples:

• healthy -> unhealthy
• unknown -> unhealthy
• unhealthy -> healthy
• stale -> healthy

Health events can become critical or resolved alerts.

Agent heartbeat changes

The staleness loop evaluates agents independently from report ingest.

When an agent changes status, Trove records an agent event.

Examples:

• ok -> stale
• stale -> offline
• offline -> ok

When an agent goes stale or offline, Trove marks its services as health="stale" so the dashboard stops showing old healthy data as current truth.

That mass health update does not create one event per service. The agent event is the notification-worthy signal.

Why events denormalize names

Events copy display fields at write time:

• service name
• hostname
• agent name

This means an event remains understandable even if the service row is later removed or pruned.

It also makes alert and digest rendering simpler and more robust.

Alert cursor

The alert engine reads events using a persistent cursor stored in metadata.

That means it can resume after a restart without replaying the full history.

Freshness flow

Image freshness is not generated from normal report transitions. It is derived by comparing:

• a service's running image_digest
• the latest registry digest stored in image_checks

Freshness alerts are still tracked in alert_state, but the check source is the freshness sweep rather than a report event.