-
Notifications
You must be signed in to change notification settings - Fork 0
Execution Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Execution Engine (Layer 4) is the gatekeeper of the AXIS cluster. It is responsible for transforming a high-level task description into a secured, resource-bounded process running on the most appropriate node. It ensures that no command is executed without passing safety checks, verifying resource availability, and maintaining a verifiable audit trail of the execution lifecycle.
The engine operates on a "Guarded" principle: every execution is wrapped in a safety and resource envelope that monitors the process from pre-flight to post-mortem observation.
The execution flow is orchestrated by the RunGuarded function internal/execution/guarded.go:217-219, which follows these primary phases:
- Intent Resolution: Parsing the request into a specific command, script, or skill.
- Safety Evaluation: Passing the command through the structured safety engine to determine if it is safe, requires a prompt, or must be blocked.
- Placement & Fit: Selecting the optimal node based on the task's inferred requirements (RAM, GPU, Tools).
-
Resource Reservation: Commitments are recorded in the
reservation.Ledgerto prevent oversubscription. -
Execution Dispatch: Running the command locally via
bashor remotely viaSSH. - Observation & Feedback: Recording peak RAM usage and exit codes to improve future placement decisions.
The following diagram illustrates how high-level execution requests flow through the core code entities of the Execution Engine.
Execution Engine Architecture
graph TD
subgraph "Request Entry"
REQ["GuardedExecutionRequest"]
end
subgraph "Safety & Requirements"
SAFE["safety.Check()"]
EVAL["safety.Evaluator"]
INF["placement.InferRequirements()"]
end
subgraph "Resource Management"
LEDGER["reservation.Ledger"]
SNAP["models.ClusterSnapshot"]
end
subgraph "Execution Transport"
LOCAL["RunLocalShell"]
REMOTE["transport.SSHExecutor"]
end
REQ --> SAFE
SAFE --> EVAL
REQ --> INF
INF --> LEDGER
LEDGER --> SNAP
LEDGER -- "If Granted" --> LOCAL
LEDGER -- "If Granted" --> REMOTE
LOCAL -- "ProcessState" --> OBS["Post-Execution Observation"]
REMOTE -- "Exit Status" --> OBS
Sources: internal/execution/guarded.go:72-88, internal/execution/guarded.go:217-240, internal/safety/blocker.go:20-30, internal/reservation/ledger.go:1-20
The pipeline is the central coordinator. It handles the "context injection" where an AXIS_CONTEXT_FILE is generated and provided to the task, containing relevant cluster facts internal/execution/guarded.go:430-445. It also manages streaming output via NDJSON for remote callers and monitors the process heartbeat to ensure reservations are released if the execution engine crashes.
For details, see Guarded Execution Pipeline.
AXIS uses a multi-tiered safety system. The safety.Evaluator applies a RuleSet (like the DefaultRuleSet) to categorize commands into risks such as CategoryDestructive or CategoryPrivilegeEscalate internal/safety/structured.go:22-31. It provides a definitive Verdict (Allow, Deny, or Prompt) internal/safety/structured.go:34-40. Additionally, it performs live cluster-aware checks, such as blocking heavy model inference if the cluster lacks sufficient allocatable RAM internal/safety/blocker.go:51-59.
For details, see Safety Evaluation System.
To prevent "thundering herd" problems where multiple tasks exhaust a node's memory simultaneously, AXIS uses a double-entry reservation.Ledger. Before execution begins, the engine requests a RAM/VRAM reservation internal/execution/guarded.go:338-345. These reservations are "overlaid" onto the raw node facts during the placement phase so the scheduler sees the future state of the cluster, not just the current idle state.
For details, see Resource Reservation Ledger.
This table maps the conceptual "Execution Plane" components to their specific implementations in the codebase.
| Concept | Code Entity | File Path |
|---|---|---|
| Execution Entrypoint | RunGuarded |
internal/execution/guarded.go:217-219 |
| Safety Gatekeeper | safety.Check |
internal/safety/blocker.go:20-20 |
| Task Requirements | models.TaskRequirements |
internal/models/requirements.go:1-10 |
| Local Runner | RunLocalShell |
internal/execution/guarded.go:171-171 |
| Remote Runner | transport.SSHExecutor |
internal/transport/ssh.go:1-20 |
| Resource Tracking | reservation.Ledger |
internal/reservation/ledger.go:1-20 |
Execution Data Flow
sequenceDiagram
participant CLI as "axis task run"
participant GE as "execution.RunGuarded"
participant SE as "safety.Evaluator"
participant PE as "placement.Engine"
participant RL as "reservation.Ledger"
participant TR as "transport.SSHExecutor"
CLI->>GE: Request(Cmd)
GE->>SE: Evaluate(Cmd)
SE-->>GE: Verdict (Allow)
GE->>PE: SelectNode(Requirements)
PE-->>GE: TargetNode
GE->>RL: Reserve(RAM)
RL-->>GE: ReservationID
GE->>TR: Execute(Cmd)
TR-->>GE: Output / ExitCode
GE->>RL: Release(ReservationID)
GE-->>CLI: GuardedExecutionResult
Sources: internal/execution/guarded.go:217-400, internal/safety/blocker.go:20-40, internal/reservation/ledger.go:1-50
The Execution Engine emits events to the internal event bus at critical transitions. These events allow the Advisory Layer: AI Surfaces and external webhooks to monitor cluster activity without interfering with the execution itself.
-
task.placement.requested: Fired before node selection. -
task.execution.reserved: Fired once RAM is committed in the ledger internal/execution/guarded.go:42. -
task.execution.started: Fired when the process is spawned. -
task.execution.finished: Fired after cleanup and observation recording internal/execution/guarded.go:43.
Sources: internal/events/events.go:20-35, internal/execution/guarded.go:42-50