-
Notifications
You must be signed in to change notification settings - Fork 0
Resource Reservation Ledger
Relevant source files
The following files were used as context for generating this wiki page:
The Resource Reservation Ledger is the central accounting system for tracking commitments of RAM and VRAM across the AXIS cluster. It implements a double-entry style system to ensure that resource allocations are deterministic, persistent, and guarded against overcommitment. The ledger acts as an overlay on top of raw hardware facts, allowing the Placement Engine to reason about "allocatable" headroom rather than just "free" hardware memory.
The ledger tracks individual commitments via the reservation.Entry struct. Each entry represents a specific resource claim tied to an execution lifecycle.
| Field | Type | Description |
|---|---|---|
ID |
string |
Unique identifier for the reservation. |
Node |
string |
The target node where resources are reserved. |
OwnerExecID |
string |
The ID of the task or execution owning this entry. |
RAMMB |
int64 |
Amount of system RAM reserved in MB. |
VRAMMB |
int64 |
Amount of GPU VRAM reserved in MB. |
CreatedAt |
time.Time |
Timestamp when the reservation was created. |
LastHeartbeat |
time.Time |
Last recorded liveness signal from the owner. |
ExpiresAt |
time.Time |
Optional hard deadline for the reservation. |
Sources:
-
Entrystruct definition: internal/reservation/ledger.go:22-35 - Liveness classification: internal/reservation/ledger.go:60-76
The reservation.Ledger manages the lifecycle of these entries through explicit creation, heartbeat updates, and reclamation.
To prevent "zombie" reservations from orphaned processes or crashed nodes, the ledger enforces a heartbeat policy. Entries are classified into three states:
-
Active: Heartbeat received within the
HeartbeatStaleWindow(default 2 minutes). - Stale: No heartbeat within the window, but not yet expired.
-
Expired: Passed the hard
ExpiresAtdeadline or theLegacyStaleWindow.
The Reclaim() function is called periodically (and during daemon startup) to prune stale or expired entries, returning the resources to the allocatable pool.
The following diagram illustrates how the Ledger interacts with execution and persistence.
Reservation Lifecycle Flow
graph TD
subgraph "Execution Plane"
A["RunGuarded"] -- "Reserve()" --> B["Ledger.Reserve"]
A -- "Heartbeat()" --> C["Ledger.Heartbeat"]
A -- "Release()" --> D["Ledger.Release"]
end
subgraph "reservation.Ledger"
B -- "lockFileLocked" --> B1["Validate Limits"]
B1 -- "saveLocked" --> B2["Atomic Write"]
C -- "Update LastHeartbeat" --> C1["saveLocked"]
D -- "Delete Entry" --> D1["saveLocked"]
E["Reclaim()"] -- "IsStale / IsExpired" --> D
end
subgraph "Persistence"
B2 -- "persist.WriteFileAtomic" --> F["~/.axis/ledger.json"]
end
Sources:
- Heartbeat logic: internal/reservation/ledger.go:260-279
- Reclamation logic: internal/reservation/ledger.go:281-309
- Atomic persistence: internal/reservation/persist.go:137-171
- Liveness classification: internal/reservation/ledger.go:38-76
The ledger enforces strict resource boundaries defined in reservation.Limits. By default, AXIS adopts a conservative "no overcommit" policy.
-
MaxOvercommitRatio: Defaults to
1.0. A value of1.5would allow reserving 150% of physical RAM. internal/reservation/ledger.go:102-102 - SystemReserveMB: RAM held back for the OS and AXIS daemon (default 1024MB). internal/reservation/ledger.go:104-104
- MaxEntriesPerNode: Caps concurrent tasks on a single node (default 32). internal/reservation/ledger.go:108-108
The Ledger calculates AllocatableRAM for a node by taking the total capacity (provided by snapshots via SetNodeCapacity) and subtracting both the system reserve and the sum of all active Entry.RAMMB values.
Sources:
-
DefaultLimits: internal/reservation/ledger.go:114-122 -
AllocatableRAMcalculation: internal/reservation/ledger.go:311-324 - Node capacity updates: internal/reservation/ledger.go:165-169
While the Ledger maintains the "truth" of commitments, the rest of the system (CLI, Placement Engine) interacts with these commitments via the snapshotview package. The ApplyReservationView function overlays ledger data onto a ClusterSnapshot.
This creates a unified view where NodeFacts fields like RAMReservedMB and RAMAllocatableMB are populated based on the current ledger state.
Entity Association: Snapshot vs Ledger
graph LR
subgraph "models.ClusterSnapshot"
N["models.NodeFacts"]
end
subgraph "reservation.Ledger"
L["Ledger.Entries()"]
S["Ledger.NodeSummaryFor()"]
end
subgraph "snapshotview.ApplyReservationView"
V["Overlay Logic"]
end
L --> V
S --> V
V -- "Updates" --> N
N -- "RAMReservedMB" --> Final["Final Snapshot View"]
N -- "RAMAllocatableMB" --> Final
Sources:
- Overlay implementation: internal/snapshotview/overlay.go:65-114
- RAM logic helpers: internal/models/memory.go:40-49
Users can inspect the ledger state using the axis reservations command group.
| Command | Function | Description |
|---|---|---|
axis reservations |
runReservationsTable |
Displays a formatted table of all active commitments. |
axis reservations list |
reservationsListCmd |
Lists entries in text, JSON, or NDJSON format. |
axis reservations inspect |
reservationsInspectCmd |
Shows full metadata for a specific reservation ID. |
axis reservations release |
reservationsReleaseCmd |
Manually deletes a reservation from the ledger. |
The CLI primarily communicates with the daemon via the /v2/reservations endpoint. If the daemon is unreachable, the CLI attempts a "fallback" mode by loading the local ledger file directly from ~/.axis/ledger.json.
Sources:
- CLI implementation: cmd/axis/reservations.go:32-45
- Daemon fallback logic: cmd/axis/reservations.go:66-74
- Persistence path: internal/reservation/persist.go:16-19
- Table rendering: cmd/axis/reservations.go:120-167