Empirical Feedback and Failure Memory

Empirical Feedback and Failure Memory

Relevant source files

The following files were used as context for generating this wiki page:

• internal/execution/observations_test.go
• internal/execution/rusage_other.go
• internal/execution/rusage_unix.go
• internal/facts/resident_models_test.go
• internal/facts/tools.go
• internal/failures/expiry.go
• internal/failures/failures_test.go
• internal/failures/record.go
• internal/failures/scope.go
• internal/models/types_test.go
• internal/placement/empirical.go
• internal/placement/empirical_test.go
• internal/placement/infer.go
• internal/placement/placement_test.go
• internal/placement/ranker.go
• internal/placement/selector.go
• internal/placement/warmth_test.go

AXIS implements a closed-loop feedback system that refines task placement based on actual execution outcomes. Rather than relying solely on static hardware specs or transient "free RAM" snapshots, the system maintains a persistent memory of past performance (Observations) and past mistakes (Failures). This allows the Placement Engine to avoid "poison" nodes, back off from flaky hardware, and select nodes that have historically proven efficient for specific workloads.

Empirical Observation System

The observation system tracks the actual resource utilization and performance of tasks. When a task completes, its resource usage is recorded as an ExecutionObservation. This data is then used as a primary ranking signal in the RankCandidates pipeline internal/placement/ranker.go:55-68.

Data Flow: Recording Observations

1. Execution: RunGuarded executes a command and measures peak RSS (Resident Set Size) and wall-clock time internal/execution/observations_test.go:14-70.
2. Scoping: The system generates an ObservationScope internal/placement/empirical.go:64-73. This scope includes:
   • Node: The specific machine used.
   • Workload: The class of task (e.g., ClassLocalLLMInference).
   • Tool/Backend: The specific engine used (e.g., ollama, mlx).
   • Model Name: For inference tasks, the specific model name is extracted from the command line internal/placement/empirical.go:78-86.
3. Persistence: The observation is stored in the ClusterState internal/execution/observations_test.go:126-132.

Ranking with Observations

During the ranking phase, AXIS looks for a "fresh" observation matching the current task's scope. Observations are considered stale after a configurable duration internal/placement/empirical_test.go:43-65.

Priority	Comparison Metric	Logic
1	Last Success	Nodes with a successful history outrank those with recent failures internal/placement/empirical.go:143-148.
2	Peak RAM	Nodes that used less RAM for the same task are preferred internal/placement/empirical.go:149-151.
3	Wall Time	Faster nodes win ties in resource usage internal/placement/empirical.go:155-160.

Sources: internal/placement/empirical.go:64-168, internal/placement/ranker.go:55-68, internal/execution/observations_test.go:14-145

---

Failure Memory and Immune System

The Failure Memory system protects the cluster from repeatedly attempting tasks on nodes that are physically or logically incapable of handling them. It uses a FailureRecord to track errors and apply penalties or blocks.

Failure Classes and Severity

Failures are categorized by severity, which determines if they are "blocking" (hard filter) or "soft penalties" (ranking adjustment) internal/failures/record.go:20-29.

Failure Class	Severity	Behavior
FailureExecCrash, FailureThermal, FailureBattery, FailureBackendMisfit	Blocking (100)	Node is excluded from FilterCandidates for that workload internal/placement/ranker.go:47-53.
FailureTimeout, FailureNetwork	Transient (50)	Applied as a ranking penalty; node is moved to the bottom of the list internal/placement/selector.go:178-186.
Minor/Specific	Low (10)	Minimal impact on placement.

Exponential Back-off

When a failure is recorded via Store.Record, the system calculates an expiry time based on the failure count internal/failures/record.go:33-75. Each subsequent failure in the same scope increases the duration the node is "tombstoned" or penalized, preventing rapid-fire retries on broken hardware.

Recovery (The "Immune" Response)

• Success Clearance: If a task succeeds on a node that previously had a recorded failure for that workload, the failure record is immediately deleted internal/failures/record.go:78-85, internal/execution/observations_test.go:72-145.
• Pruning: Expired records are automatically removed during cluster state maintenance internal/failures/record.go:115-125.

Sources: internal/failures/record.go:1-125, internal/placement/ranker.go:47-53, internal/placement/selector.go:178-186

---

Code Entity Map: Feedback Loop

This diagram maps the logical feedback loop to the specific functions and structs in the codebase.

Feedback Loop Architecture

graph TD
    subgraph "Execution Layer (Layer 4)"
        A["RunGuarded()"] -- "Measures rusage" --> B["ExecutionObservation"]
        A -- "Catch Error" --> C["FailureRecord"]
    end

    subgraph "State Layer (Snapshot)"
        B -- "RecordObservation()" --> D["state.ClusterState"]
        C -- "Store.Record()" --> D
    end

    subgraph "Placement Layer (Layer 3)"
        E["FilterCandidates()"] -- "Checks isBlockingFailure()" --> F["Eligible Nodes"]
        D -- "Lookup Scope" --> G["empiricalObservation()"]
        F -- "RankCandidates()" --> H["Selected Node"]
        G -- "compareObservationPreference()" --> H
    end

    H -- "Task Execution" --> A

Loading

Sources: internal/placement/ranker.go:37-68, internal/placement/empirical.go:114-168, internal/execution/observations_test.go:14-145, internal/failures/record.go:31-75

---

Resident Model Locality and Warmth

AXIS tracks which AI models are currently "resident" in memory on specific nodes. This is used to favor nodes that already have a model loaded, minimizing "cold start" latency.

Resident Model Discovery

The system uses specialized scripts to probe inference backends:

• Ollama: OllamaDiscoveryScript parses ollama ps to find loaded models and their expires_at times internal/facts/tools.go:23-102.
• llama-server: LlamaServerDiscoveryScript inspects process command lines to find the -m flag internal/facts/tools.go:111-142.

Warmth Scoring

For backends like Ollama that provide TTL (Time-To-Live) for models, AXIS computes a WarmthScore [0.0 - 1.0] based on how much time is left before the model is unloaded internal/facts/warmth_test.go:159-179.

In the RankCandidates pipeline, this is converted into a discrete modelWarmthRank internal/placement/empirical.go:222-234:

1. Hot (>0.9): Rank 2
2. Warm (>0.5): Rank 1
3. Cold (<=0.5): Rank 0

This rank serves as a tiebreaker (priority level 10) to ensure that if two nodes are otherwise identical, the one with the "warmest" model wins internal/placement/ranker.go:181-183.

Sources: internal/facts/tools.go:23-142, internal/placement/empirical.go:170-234, internal/placement/ranker.go:142-194, internal/facts/warmth_test.go:1-179

---

Code Entity Map: Model Discovery to Placement

This diagram shows how raw bash discovery scripts result in placement decisions.

Discovery to Decision Flow

graph LR
    subgraph "Fact Collection (Layer 1)"
        A["OllamaDiscoveryScript"] -- "stdout JSON" --> B["ollamaDiscoveryPayload"]
        C["LlamaServerDiscoveryScript"] -- "stdout JSON" --> D["llamaServerDiscoveryPayload"]
    end

    subgraph "Data Models"
        B --> E["models.ResidentModel"]
        D --> E
        E -- "WarmthScore" --> F["models.NodeFacts"]
    end

    subgraph "Placement Engine (Layer 3)"
        F --> G["residentModelRank()"]
        F --> H["modelWarmthRank()"]
        G --> I["RankCandidates()"]
        H --> I
        I --> J["PlacementDecision.Reasoning"]
    end

Loading

Sources: internal/facts/tools.go:16-151, internal/placement/empirical.go:213-234, internal/placement/ranker.go:151-183, internal/placement/selector.go:83-85

---