Repository navigation
Kernel and Agents
shoal-kernel is the long-lived, per-user daemon that hosts shared sessions: a human TUI, one or
more agents over MCP, and an LSP client can all attach to the same session and see the same
transcript, cwd, env, and task tree. This page covers the kernel's architecture and the wire
protocol it actually speaks today. For the agent-facing doctrine and value encoding built on top of
this protocol, see Agent-Surface; for what gates a exec call before it's allowed to run, see
Leash-and-Security.
The evaluator, value model, and exec engine live in an embeddable core (shoal-eval). shoal script.shl and shoal -c run kernel-less — an in-process journal buffer, flushed to a file or
discarded, no socket, no daemon. Interactive and agent sessions instead attach to shoal-kernel,
which hosts the identical core long-lived behind a Unix socket. Per CONTRACTS.md's dependency DAG,
shoal (the binary) never depends on or spawns shoal-kernel — they are alternative hostings of
the same engine, not a client/server pair at the Cargo level. shoal-mcp and shoal-lsp are
spawned by shoal as companion subprocesses (shoal mcp, shoal lsp) that then talk to a
running shoal-kernel's socket purely over the wire protocol — shoal-mcp has zero shoal-*
Cargo dependencies.
// crates/shoal-kernel/src/lib.rs — the real shape
struct Session {
id: String,
evaluator: Mutex<Evaluator>,
transcript: Mutex<HashMap<Ref, Value>>, // out:n -> Value, session-wide
client_it: Mutex<HashMap<u64, Ref>>, // per-CLIENT `it` cursor (TDD §1.11)
next_value: AtomicU64,
}A session holds one Evaluator (cwd, env, lexical scope), a shared transcript mapping refs
(out:1, out:2, …) to values, and background task state. Multiple clients attach to the same
session concurrently; the kernel spawns a thread per accepted connection. Crucially, it is
per-client (TDD §1.11) — each attached client has its own cursor into the shared transcript, so
a human typing at the prompt and an agent driving shoal_exec in the same session don't stomp each
other's "the last thing I ran" — while out[n] indexes the transcript globally for everyone.
Authentication is SO_PEERCRED same-uid by default; agent principals additionally authenticate
with bearer tokens minted via shoal-auth's TokenStore (shoal token create --caps <leash-profile>), each carrying a principal name, capability list, and profile that
session.attach resolves into the attached client's identity.
Transport: JSON-RPC 2.0, newline-delimited, over a Unix socket at
$XDG_RUNTIME_DIR/shoal/<session>.sock (created 0600). Frames are capped at 16 MiB.
Methods actually implemented in crates/shoal-kernel/src/lib.rs's dispatcher today:
| Method | Notes |
|---|---|
session.attach |
{session?, token?, client} → {session, principal, caps, cwd, env_hash, ast_version}. caps honestly reports {enforced: false, tier: "D", ...} until an OS backend is wired at this layer — see Leash-and-Security. |
parse |
{src} → {ast_version, ast} or a structured parse diagnostic. |
exec |
{src, mode:"run"|"plan"|"approved", position, asynchronous, elide?}. mode:"plan" derives a Plan (effects, plan_ref) and stops before spawn; mode:"run" evaluates the policy verdict first (Deny/ApprovalRequired/Allow) and only then executes; asynchronous:true returns a task ref immediately and runs on a worker thread. |
value.get |
{ref, path?, slice?, elide?} — re-query any transcript value by field path/slice, no re-execution. |
task.list / task.await / task.cancel / task.suspend
|
Task lifecycle. task.suspend returns an honest "unavailable for evaluator-owned processes" error rather than pretending to suspend. |
plan.apply |
{plan_ref} → re-dispatches as exec {mode:"approved"} once a plan is approved. |
cap.request |
{plan_ref, effects?} → approves a stored plan if policy doesn't outright deny it. |
journal.query |
{since?, head?, principal?, ok?, limit} → structured entry rows. |
Every value on the wire uses the $-tagged encoding from Agent-Surface §2 — {"$":"int","v":3},
{"$":"path","v":"/tmp","raw":null}, and so on; refs are stable (out:12, val:blake3:…,
task:7, plan:8f2c).
exec's side effects on every real run: a journal entry is appended before evaluation (so a
crash mid-run still leaves a row), the resulting value is stored in the session transcript under a
fresh out:n ref, the per-client it cursor is updated, and the journal entry is finished with
status/duration and linked output blobs (value, render, and — for an Outcome — stdout/
stderr) once evaluation completes. A failing eval still finishes the journal row (ok=false) and
records the diagnostic's stderr if present.
Backing store: SQLite in WAL mode at ~/.local/share/shoal/journal.db, plus a content-addressed
blob store at ~/.local/share/shoal/cas/<hex[0..2]>/<hex[2..4]>/<hex>.zst (shoal-journal).
entry(id INTEGER PK, session TEXT, principal TEXT, ts INT, dur_ns INT,
cwd BLOB, env_hash BLOB, ast BLOB, effects TEXT, status INT, ok BOOL, opaque BOOL)
output(entry_id INT, kind TEXT, hash BLOB, len INT, meta TEXT)
undo(entry_id INT, op TEXT, inverse TEXT)
pin(hash BLOB PK)Every entry records src, the canonical AST as JSON, the concrete effects derived for the call
(["opaque"] for T0/sh{}), and links to zstd-compressed, deduplicated output blobs by blake3
hash. GC (shoal-history) is TTL + LRU with pins exempt — entry rows outlive their blobs, so
history is never silently incomplete, only occasionally light on payload. Redaction by
construction: secrets are journaled as names only, never material (see Leash-and-Security).
Undo is wired into the tree-walk evaluator today: fs-mutating builtins (rm, cp, mv, save, …)
record a typed inverse before the mutation (a trash-CAS move for delete, the prior-content hash
for overwrite), and undo (bare, replays the last reversible entry) / undo <entry-id> (an
explicit journal row id) replay inverses, refusing cleanly when the target has gone stale
(stale_undo) or when no journal is active. out[n] → entry-id addressing (so undo out[7] reads
as naturally as out[7] itself) is planned (roadmap wave R3) but not wired yet — today you undo by
bare undo or by the numeric entry id from journal/history.
journal.query (both the kernel RPC and shoal-history's Rust API) filters by since_ts_ns,
head (first word of src), principal, and ok — the same query surface backs both a human
typing journal at the prompt and an agent calling shoal_journal.
shoal-mcp is a standalone stdio-speaking binary — zero shoal-* Cargo dependencies — that
translates MCP's initialize/tools/list/tools/call into calls against a running kernel's
socket. It attaches once at startup (session.attach with client: {kind:"mcp", tty:false}) and
then maps each of the seven tools in Agent-Surface §5 (shoal_exec shoal_plan shoal_apply shoal_get shoal_journal shoal_cancel shoal_cap_request) onto exactly one kernel method call,
wrapping the kernel's JSON result as {content, structuredContent, isError} per the MCP tool-result
shape — every result also carries a resource_link back to the value's ref and caps its inline
text content at a hard 64 KiB regardless of the value's real size. Errors from the kernel (a
policy denial, a parse error, an unknown ref) are surfaced as isError: true results carrying the
kernel's structured error — never a bare exception, never free text the agent has to pattern-match.
resources/subscribe on a shoal://events/{channel} or shoal://task/{id}/out URI is also wired:
the bridge opens a dedicated events.subscribe connection to the kernel and re-emits each event as
an MCP notifications/resources/updated push — a real implementation of Agent-Surface §6, not
just a spec.
- Agent-Surface — the value encoding, elision rule, and event/channel model layered over this socket.
-
Leash-and-Security — the policy engine
exec/plan.apply/cap.requestconsult before doing anything with side effects. -
Developer-Guide — where
shoal-kernelsits in the crate dependency DAG.