Snapshot Assembly and Daemon Cache

Snapshot Assembly and Daemon Cache

Relevant source files

The following files were used as context for generating this wiki page:

• internal/agent/cortex_memory.go
• internal/agent/cortex_memory_test.go
• internal/daemon/daemon.go
• internal/daemon/daemon_test.go
• internal/mcp/server.go
• internal/mcp/server_tools_test.go
• internal/placement/explain.go
• internal/snapshot/snapshot.go
• internal/snapshot/snapshot_test.go

Layer 2 (the Snapshot Plane) is responsible for transforming raw, per-node telemetry into a unified, cluster-wide view known as the ClusterSnapshot. This layer acts as the "source of truth" for the placement engine and AI surfaces, ensuring that scheduling decisions are based on a coherent and deduplicated representation of the cluster state.

The daemon package manages the lifecycle of this snapshot, maintaining a high-performance cache and orchestrating periodic refreshes triggered by network events, configuration changes, or task executions.

Snapshot Plane Overview

The Snapshot Plane bridges the gap between raw discovery and intelligent scheduling. It takes NodeFacts collected by Layer 1 and performs three critical operations:

1. Deduplication: Resolving conflicts when a node is discovered via multiple paths (e.g., nodes.yaml vs. UDP beacon).
2. Aggregation: Computing cluster-wide totals for RAM, CPU, and reachability.
3. Health Assessment: Assigning a global status (Healthy vs Degraded) and generating actionable warnings.

Code Entity Mapping: Snapshot Assembly

The following diagram illustrates how the snapshot.Build function processes raw inputs into the structured ClusterSnapshot used by the rest of the system.

"Snapshot Assembly Logic"

graph TD
    subgraph "Input: NodeFacts List"
        A["models.NodeFacts (Raw)"]
    end

    subgraph "internal/snapshot"
        B["snapshot.Build()"]
        C["classifyNetwork()"]
        D["PopulateMemoryMetrics()"]
    end

    subgraph "Output: models.ClusterSnapshot"
        E["models.ClusterSummary"]
        F["models.Warning List"]
        G["models.SnapshotStatus"]
    end

    A --> B
    B --> C
    B --> D
    C --> G
    B --> E
    B --> F
    B --> G

Loading

Sources: internal/snapshot/snapshot.go:13-16, internal/snapshot/snapshot.go:53-57, internal/snapshot/snapshot.go:116-121

---

4.1 Snapshot Building

The snapshot.Build() function is the entry point for assembling the cluster view. It prioritizes nodes defined in the local configuration over those discovered via mesh gossip to prevent identity spoofing. During assembly, it classifies the NetworkClass for every node (e.g., DirectLAN, Tailscale, VPN) based on SSH handshake latency and IP address ranges.

Key behaviors include:

• Deduplication: Nodes are matched by StableID and Name, with configuration-backed facts taking precedence internal/snapshot/snapshot.go:17-51.
• Aggregation: Total and Free RAM are summed across all reachable nodes to populate the ClusterSummary internal/snapshot/snapshot.go:59-74.
• Health Warnings: If a node is Unreachable or experiencing RAM pressure (less than 10% free), the snapshot is marked Degraded and a Warning is appended internal/snapshot/snapshot.go:76-114.

For details, see Snapshot Building.

---

4.2 Daemon Lifecycle and Cache Management

The Daemon struct serves as the long-running process that hosts the ClusterSnapshot. It provides a SnapshotCache interface that allows other components to retrieve the latest state without re-probing the network.

The daemon is driven by Refresh Triggers. A refresh can be initiated by:

• Timer: Periodic background updates (default: 1 minute) internal/daemon/daemon.go:39.
• State Change: When a task finishes or a reservation is made internal/daemon/daemon.go:151-157.
• Manual: Triggered via axis daemon refresh or the API internal/daemon/daemon.go:147-149.

Code Entity Mapping: Daemon Refresh Loop

This diagram shows how the Daemon manages the transition from a refresh trigger to a cached snapshot.

"Daemon Refresh Workflow"

graph LR
    T["Refresh Trigger (Timer/Event)"] --> D["daemon.Daemon.refreshWithTrigger()"]
    D --> C["daemon.Collector (Discovery)"]
    C --> B["snapshot.Build()"]
    B --> S["d.snapshot (In-Memory Cache)"]
    S --> P["~/.axis/snapshot.json (Persistence)"]
    S --> H["Snapshot Hooks (Dispatch)"]

Loading

Sources: internal/daemon/daemon.go:79-116, internal/daemon/daemon.go:147-157, internal/daemon/daemon_test.go:29-51

For details, see Daemon Lifecycle and Cache Management.

---

4.3 Persistent State Management

To ensure fast restarts and survival across reboots, the daemon persists the cluster state to the local filesystem.

• Snapshot Persistence: The full ClusterSnapshot is written to a JSON file (usually ~/.axis/snapshot.json) upon every successful refresh internal/daemon/daemon_test.go:53-58.
• Reservation Ledger: Active RAM/VRAM commitments are tracked in a reservation.Ledger. This ledger is overlaid onto the snapshot to provide an "allocatable" view of resources internal/daemon/daemon.go:89.
• Failure Memory: The daemon integrates with state.ClusterState to track FailureRecords. This allows the system to remember nodes that recently failed a specific workload class and avoid them in future placements internal/placement/explain.go:73-82.

For details, see Persistent State Management.

---

Summary Table: Snapshot Metadata

The daemon exposes metadata about its current cache state via the /snapshot/meta endpoint.

Field	Description	Code Reference
Ready	Boolean indicating if a valid snapshot has been collected.	internal/daemon/daemon.go:57
LastRefreshTrigger	The reason for the most recent refresh (e.g., manual, timer).	internal/daemon/daemon.go:59
CollectedAt	Timestamp of the last successful fact collection.	internal/daemon/daemon.go:61
ReservedMB	Total RAM currently locked by the reservation.Ledger.	internal/daemon/daemon.go:65
Stale	True if the cache age exceeds the stale_threshold.	internal/daemon/daemon.go:68

Sources: internal/daemon/daemon.go:55-77

---