Snapshot Building

Snapshot Building

Relevant source files

The following files were used as context for generating this wiki page:

• internal/agent/cortex_memory.go
• internal/agent/cortex_memory_test.go
• internal/facts/local.go
• internal/facts/parsers_test.go
• internal/facts/remote.go
• internal/models/types.go
• internal/placement/explain.go
• internal/snapshot/snapshot.go
• internal/snapshot/snapshot_test.go
• internal/snapshotview/overlay_test.go
• internal/snapshotview/snapshotview_test.go

The Snapshot Plane (Layer 2) is responsible for transforming raw, heterogeneous telemetry collected from nodes into a unified, deduplicated, and analyzed ClusterSnapshot. This process is primarily orchestrated by the snapshot.Build() function, which serves as the bridge between raw discovery and the deterministic placement engine.

The Building Process

The snapshot.Build() function follows a multi-pass pipeline to ensure that the resulting snapshot represents the "Truth" of the cluster at a specific point in time.

1. Deduplication and Prioritization

AXIS discovers nodes through multiple channels: static configuration (nodes.yaml), UDP beacons, and mesh gossip. snapshot.Build() resolves conflicts using a priority system:

• Pass 1 (Config Priority): Nodes explicitly defined in the configuration or having a defined Role are processed first. These are considered authoritative internal/snapshot/snapshot.go:21-32.
• Pass 2 (Discovery): Nodes found via mesh or beacons are added only if their StableID or Name does not conflict with a configuration-verified node internal/snapshot/snapshot.go:34-51.

2. Resource Aggregation

The builder computes cluster-wide aggregates into a ClusterSummary internal/models/types.go:116-121. This includes:

• Reachable Nodes: Count of nodes with StatusComplete or StatusPartial.
• Aggregate RAM: Summation of RAMTotalMB and RAMFreeMB across all reachable members internal/snapshot/snapshot.go:68-74.

3. Health Classification and Warnings

The snapshot's overall status is derived from its members. If any node has a status other than complete, the entire snapshot is marked as degraded internal/snapshot/snapshot.go:77-79. Specific warnings are generated for:

• Unreachable/Error: SSH failures or collector panics internal/snapshot/snapshot.go:82-101.
• RAM Pressure: Triggered if a node's free RAM falls below 10% internal/snapshot/snapshot.go:104-113.

Data Flow: Facts to Snapshot

The following diagram illustrates how raw telemetry is transformed into the structured ClusterSnapshot.

Telemetry Transformation Flow

graph TD
    subgraph "Fact Collection Space"
        A["LocalCollector.Collect()"] -->|NodeFacts| C["NodeFacts Array"]
        B["RemoteCollector.Collect()"] -->|NodeFacts| C
    end

    subgraph "snapshot.Build() Logic"
        C --> D["Deduplicate (Config > Mesh)"]
        D --> E["classifyNetwork()"]
        E --> F["PopulateMemoryMetrics()"]
        F --> G["Aggregate ClusterSummary"]
        G --> H["Generate Warnings"]
    end

    subgraph "Code Entities"
        H --> I["models.ClusterSnapshot"]
        I --> J["models.ClusterSummary"]
        I --> K["[]models.NodeFacts"]
    end

Loading

Sources: internal/snapshot/snapshot.go:16-123, internal/facts/local.go:52-143, internal/facts/remote.go:36-139

---

Network Topology Classification

A critical role of the builder is classifyNetwork(). This function determines the NetworkClass (DirectLAN, Tailscale, VPN, Relayed) for each node based on the dial target and observed latency internal/snapshot/snapshot.go:126-207.

Network Class	Detection Logic	Handshake Threshold
DirectLAN	Local node OR Private IP (RFC1918) OR Handshake < 100ms (non-IP target)	< 100ms
Tailscale	IP in fd7a:115c:a1e0::/48 or 100.64.0.0/10	< 150ms
VPN	IP in known WireGuard/VPN ranges	< 150ms
Relayed	Any overlay/remote path with high latency	>= 150ms

Sources: internal/snapshot/snapshot.go:126-207, internal/models/types.go:203-210

---

RAM Aggregates and Memory Topology

AXIS differentiates between raw physical RAM and "Allocatable" RAM. During snapshot building, PopulateMemoryMetrics() is called for every node to calculate these values.

Memory Metrics Calculation

1. System Reserved: Defaults to 1024MB or the value in nodes.yaml internal/models/types.go:151-160.
2. Eviction Reserved: A safety floor (512MB or 5% of total RAM) to prevent OOM kills internal/models/types.go:162-168.
3. Allocatable RAM: Calculated as FreeRAM - SystemReserved - EvictionReserved internal/models/types.go:170-174.

Topology Mapping

The builder identifies MemoryTopology (Standard vs. Unified) to assist the placement engine in understanding GPU/CPU memory sharing internal/models/types.go:46-52.

Memory Topology Mapping

graph LR
    subgraph "Hardware Discovery"
        A["uname -m (Arch)"]
        B["sysctl hw.model (Model)"]
    end

    subgraph "facts.detectMemoryTopology()"
        A & B --> C{Is Apple Silicon?}
        C -- "Yes" --> D["MemoryTopologyUnified"]
        C -- "No" --> E["MemoryTopologyStandard"]
    end

    subgraph "models.Resources"
        D --> F["MemoryClass (1-4)"]
        E --> G["MemoryClass (0)"]
    end

Loading

Sources: internal/facts/remote.go:164-165, internal/facts/parsers_test.go:212-228, internal/models/types.go:78-107

---

Health Warnings and Diagnostics

The snapshot generator produces models.Warning objects that are consumed by the CLI (axis status) and the AI Advisory layer to explain cluster degradation.

Warning Types

• unreachable: Node failed the initial SSH handshake internal/snapshot/snapshot.go:83-88.
• partial: Node was reached, but specific probes (like nvidia-smi or ollama) failed internal/snapshot/snapshot.go:89-94.
• ram_pressure: Node is operational but lacks sufficient headroom for new workloads internal/snapshot/snapshot.go:104-113.

Deterministic Placement View

Before the placement engine uses a snapshot, it is often wrapped in a snapshotview. This overlays the Reservation Ledger onto the snapshot, subtracting currently leased RAM from the RAMAllocatableMB field to provide a real-time view of available capacity without requiring a new fact collection cycle.

Sources: internal/snapshot/snapshot.go:81-114, internal/snapshotview/snapshotview_test.go:240-270

---