Daemon Lifecycle and Cache Management

Daemon Lifecycle and Cache Management

Relevant source files

The following files were used as context for generating this wiki page:

• hack/verify-repo-truth.sh
• internal/daemon/adversarial_test.go
• internal/daemon/client.go
• internal/daemon/client_test.go
• internal/daemon/daemon.go
• internal/daemon/daemon_test.go
• internal/daemon/refresh.go
• internal/mcp/server.go
• internal/mcp/server_tools_test.go
• internal/models/warnings.go
• internal/models/warnings_test.go

The AXIS daemon is a long-running process that serves as the cluster's "Source of Truth" by maintaining a high-fidelity ClusterSnapshot. It manages the lifecycle of node discovery, fact collection, and resource reservation, exposing this state via an HTTP API and a Model Context Protocol (MCP) server.

Daemon Initialization

The Daemon struct is initialized via NewDefault or New internal/daemon/daemon.go:171-174. Upon startup, it performs the following steps:

1. Beacon Registry Setup: Initializes a discovery.BeaconRegistry to track UDP heartbeats from other nodes internal/daemon/daemon.go:172.
2. Mesh Networking: If enabled in nodes.yaml, it initializes the mesh.Mesh gossip layer, identifying the local node and seeding peers from the configuration internal/daemon/daemon.go:177-210.
3. Reservation Ledger: Loads the reservation.Ledger from disk (~/.axis/ledger.json) to recover active RAM/VRAM commitments internal/daemon/adversarial_test.go:22-26.
4. First Refresh: Triggers an initial startup refresh to populate the cache before the HTTP server begins accepting requests internal/daemon/refresh.go:14.

Daemon Initialization and Component Ownership

The following diagram illustrates how the Daemon class orchestrates its internal components.

"Daemon Component Architecture"

graph TD
    subgraph "Code Entity Space"
        D["Daemon (internal/daemon/daemon.go)"]
        L["Ledger (internal/reservation/ledger.go)"]
        M["Mesh (internal/mesh/mesh.go)"]
        BR["BeaconRegistry (internal/discovery/beacon.go)"]
        SC["SnapshotCache (Interface)"]
    end

    subgraph "Data Storage"
        LEDGER_FILE["~/.axis/ledger.json"]
        SNAP_FILE["~/.axis/snapshot.json"]
    end

    D -->|owns| L
    D -->|owns| M
    D -->|owns| BR
    D -.->|implements| SC
    
    L -.->|persists to| LEDGER_FILE
    D -.->|persists to| SNAP_FILE

Loading

Sources: internal/daemon/daemon.go:79-116, internal/daemon/daemon.go:171-210, internal/daemon/adversarial_test.go:22-26

The Seven Refresh Triggers

The daemon does not poll blindly; it uses a reactive refresh model. Any refresh request is normalized and deduplicated via NormalizeRefreshTrigger internal/daemon/refresh.go:27-67.

Trigger	Constant	Description
Startup	startup	Initial collection when the process begins internal/daemon/refresh.go:14.
Timer	interval	Periodic background refresh (default: 1 minute) internal/daemon/daemon.go:39.
Config Change	config-change	Triggered when nodes.yaml is modified on disk internal/daemon/refresh.go:16.
State Change	state-change	Triggered by execution lifecycle events (e.g., execution-finished) internal/daemon/refresh.go:17.
Skills Change	skills-change	Triggered when local tool discovery detects new capabilities internal/daemon/refresh.go:18.
Beacon	beacon-change	Triggered when a new UDP beacon is received from a previously unknown node internal/daemon/refresh.go:19.
Manual	manual	Triggered via axis daemon refresh or the API /refresh endpoint internal/daemon/daemon.go:147-149.

Refresh Coalescing

To prevent "thundering herd" probes when multiple events occur simultaneously, the daemon implements coalescing in refreshWithTrigger. It uses an atomic.Bool (refreshing) and a pendingRefresh channel to ensure that at most one refresh is active while one is queued internal/daemon/adversarial_test.go:101-124.

Cache Management and Fingerprinting

The daemon maintains the ClusterSnapshot in memory and optionally persists it to a JSON file internal/daemon/daemon.go:92-93.

Filesystem Fingerprinting

The daemon monitors nodes.yaml using a configFingerprint which stores a SHA-256 hash of the file contents internal/daemon/daemon.go:137-140. If the hash changes during a poll, a config-change refresh is triggered.

Snapshot Hook Dispatch

Subscribers can register for snapshot changes using AddOnSnapshotChanged internal/daemon/daemon.go:110-116.

• Debouncing: Hooks are only fired if the SHA-256 hash of the new snapshot differs from the previous snapshot for that specific subscriber internal/daemon/daemon.go:118-125.
• Concurrency: Hooks run synchronously on the refresh goroutine, but the daemon releases its primary mutex (d.mu) before dispatching to prevent deadlocks internal/daemon/daemon.go:127-135.

Resource Reservation Ledger Overlay

The reservation.Ledger acts as a double-entry bookkeeping system for compute resources. It tracks Entry objects representing RAM/VRAM commitments internal/daemon/adversarial_test.go:68-73.

1. Overlay Logic: When a user requests a snapshot via the API, the daemon retrieves the raw ClusterSnapshot and overlays the active reservations from the ledger using snapshotview internal/mcp/server.go:22.
2. Reclaim Logic: The ledger enforces a HeartbeatStaleWindow. If an execution process (the owner of a reservation) dies without releasing its RAM, the ledger automatically reclaims the resources during the next Load() or refresh cycle internal/daemon/adversarial_test.go:58-99.
3. Corruption Recovery: If the ledger file is corrupted, the daemon moves the file to a quarantine location and initializes a fresh ledger to ensure system availability internal/daemon/adversarial_test.go:17-56.

Snapshot Data Flow

The following diagram traces how raw node facts become a client-visible snapshot with reservation overlays.

"Snapshot Assembly and Overlay Flow"

sequenceDiagram
    participant D as Daemon (internal/daemon/daemon.go)
    participant C as Collector (internal/daemon/daemon.go)
    participant S as snapshot.Build (internal/snapshot/build.go)
    participant L as Ledger (internal/reservation/ledger.go)
    participant V as snapshotview (internal/snapshotview/view.go)

    D->>C: Invoke Collector (SSH/Local)
    C-->>D: Raw NodeFacts[]
    D->>S: Build(NodeFacts[])
    S-->>D: ClusterSnapshot (Physical Truth)
    Note over D: SHA256 Fingerprinting
    D->>L: Get Active Reservations
    L-->>D: Reservation List
    D->>V: ApplyReservations(Snapshot, List)
    V-->>D: ClusterSnapshot (Effective Truth)
    D->>D: Dispatch SnapshotHooks

Loading

Sources: internal/daemon/daemon.go:147-157, internal/daemon/daemon.go:110-135, internal/mcp/server.go:22, internal/daemon/adversarial_test.go:159-187

Metadata and Health Monitoring

The daemon exposes a Metadata struct internal/daemon/daemon.go:55-77 that provides observability into the cache state:

• Stale Tracking: If the cache age exceeds defaultStaleThreshold (5 minutes), the Stale flag is set to true internal/daemon/daemon.go:41, 68.
• Freshness: Includes a DiscoveryFreshness object indicating how recently nodes were seen via UDP beacons or SSH probes internal/daemon/daemon.go:76.
• Error Propagation: The LastError field captures the string representation of the most recent failed refresh attempt internal/daemon/daemon.go:63.

When a client calls FetchSnapshot, AXIS automatically appends warnings to the snapshot if the daemon cache is stale or if nodes have failed their most recent probes internal/daemon/client.go:72-84.

Sources:

• internal/daemon/daemon.go: Core daemon logic, triggers, and metadata.
• internal/daemon/refresh.go: Trigger normalization and constants.
• internal/daemon/client.go: Client-side cache staleness handling.
• internal/daemon/adversarial_test.go: Ledger recovery and coalescing behavior.
• internal/mcp/server.go: Integration of snapshot hooks with MCP.

---