Skip to content

Execution Model

Allie Coleman edited this page Jul 16, 2026 · 4 revisions

Execution Model

This page covers what happens once source is parsed: how a call becomes a process, what a command call is as a value, the streaming/reactive model, task lifecycle, and session-scoped state. See Language-Reference for the grammar that produces the AST this page executes, and Agent-Surface for how the same execution surfaces over the wire.

1. The PTY position rule

One rule, keyed to syntax position (TDD §1.2):

  • Statement position, in an interactive session — the call is the whole statement — the child gets a real PTY, passed through raw: colors, progress bars, cursor tricks, everything, byte-identical to bash — while the byte stream is simultaneously teed into the CAS. it binds to the resulting outcome afterward, with structured parsing applied lazily to the teed bytes on first structured access. cargo build at the prompt looks exactly like it does today, and is still a queryable value afterward.
  • Value position — the call feeds a chain, an assignment, an argument, or any expression — the child gets pipes and no controlling tty. It self-detects non-interactive, emits clean machine output, and shoal captures it structurally.

Scripts and non-tty sessions are all value-position. interact <cmd> forces a PTY anywhere; adapters can declare class = "tui" to get PTY behavior automatically. Redirects (>, >>) force capture regardless of position, because the tee makes it work either way.

R0 ergonomics fix. An outcome now carries streamed: bool — true only when its bytes actually hit the real terminal via the PtyTee path. The REPL's render_result re-renders any outcome whose streamed is false, which fixes a real defect: statement-position builtins (echo hello, ls, cat) never went through PtyTee (they have no external process to hand a PTY to) and so used to render nothing at the prompt until you explicitly pulled .out — only genuine external commands in statement position (already visible live on the real tty) correctly skipped the double-render. exit [code: int = 0] / quit are now REPL builtins too — they set a host-level exit flag (Flow/pending_exit) rather than calling std::process::exit directly, so the same code works identically whether the host is the interactive REPL, a script, or an embedded kernel session.

Capture (value position) PtyTee (statement position)
Streams stdout/stderr piped, drained concurrently raw bytes to the real terminal, simultaneously teed
Process group own group (setpgid(0,0)), no controlling tty session leader on a real PTY
stdin per StdinSpec (Null/Inherit/Bytes/File) real stdin forwarded, terminal in raw mode for the duration
stderr separate buffer merged into stdout (PTY semantics — this is why a live run can't split .stderr back out; only a captured run can)

Implemented at shoal-exec (crates/shoal-exec), which is deliberately blocking/thread-based — no tokio. Cancellation escalates SIGINT → the process group, SIGTERM after 3s, SIGKILL after 3 more; signal deaths surface honestly as signal: Some("SIGSEGV") rather than bash's 128+n encoding, and children are always reaped (no zombies). When the host process itself has no tty (CI), PTY mode degrades gracefully: a PTY is still allocated for the child, but raw mode and stdin forwarding are skipped.

2. Outcomes — every command yields one

Under outcome unification, every command call — builtin or external — evaluates to a single outcome value: {status: int, ok: bool, out, err, dur, pid, cmd}. .out is the outcome's structured payload, parsed lazily. This means echo, ls, and an arbitrary external binary are all the same shape of result, just with different fidelity depending on whether an adapter is present (Ecosystem-and-Tooling).

# spec/cases/outcome.toml
(echo hi).out                 # -> "hi"
(echo hi).ok                   # -> true
(echo hi).status                # -> 0
echo hi                          # -> outcome(status: 0, ok: true)   (bare call at value position)

echo a && echo b && echo c        # -> outcome(status: 0, ok: true)  (chain returns the deciding operand verbatim)
echo hi && false                    # -> false   (lhs truthy, && proceeds; rhs is a plain bool)
true && (echo hi)                    # -> outcome(status: 0, ok: true)

A builtin's .out is exactly its own structured return value — echo's .out is the joined argument string verbatim, never re-sniffed as JSON even if it looks like [1,2,3]. A real subprocess with no parsed preset does auto-upgrade bracket/brace-shaped stdout to a structured list/record on first access — the T1 sniffing tier described in Ecosystem-and-Tooling. An adapter-backed call (T2) skips guessing entirely: .out is whatever output.parse promised, typed.

ok_codes (adapter field, default {0}) decides what "success" means per tool — non-zero raises stays the default without pretending every tool agrees that 0 is the only success code.

3. The value model

The complete type list (TDD §4.1): null bool int(i64) float(f64) str path glob regex size(u64 bytes) duration(i64 ns) datetime time bytes list<T> record table stream<T> error outcome task plan cmd secret.

  • path is bytes-backed (OsString) — path → str is fallible (.str() errors on invalid UTF-8; .display() is lossy-with-replacement). Non-UTF-8 filenames are handled end-to-end as bytes, never mangled.
  • secret is opaque: renders as secret(NAME), cannot be interpolated into a str (a type error), is injected by the kernel at spawn time, and only its name — never its material — reaches the journal.
  • table is semantically list<record> — there's no separate table type at the language level, just a rendering/storage optimization over the same shape.
  • Equality is structural for data types, identity for task/stream — comparing two streams is a runtime error, not a silent false.

The real Value enum (crates/shoal-value/src/lib.rs) additionally carries Range, Closure, and CmdRef (an alias/partial-application node, TDD §1.8 — alias gs = git status binds gs to a partial call AST node; invoking gs -sb appends to that AST, never splices text).

4. Streams

stream<T> is lazy (constructing one does no work until it's driven), single-consumption (TDD §1.9 — a second .each/.collect/render on an already-driven stream is a runtime error, code stream_consumed), pull-based with bounded backpressure, and cancellable sink-to-source. Comparing streams is an error; there is no stream == stream.

Implemented (roadmap wave R1 — crates/shoal-eval/src/streams.rs + channels.rs; corpus in spec/cases/streams.toml, 33 cases). This is the honest replacement for tail -f | grep, fswatch | xargs, and sentinel-file polling loops: every source and sink shares the same combinators as an ordinary collection — a live filesystem watch and a static list<record> are driven with the identical .where/.map vocabulary.

Sources: .stream() promotes any collection or string (line-split) to a finite stream; watch(path | glob) (via notify, cross-platform) → stream<{path, kind}>; tail(file, from_start: bool=false) → stream<str>, following external appends; every(duration) → stream<datetime> ticks; channel(name).events() → stream<event>.

Combinators: .where .map .scan(init, f) .window(n) .dedupe .distinct .flat_map .enumerate .buffer(n) .zip(other) .merge(other) .take(n) .take_until(pred) .throttle(dur) .debounce(dur).

Sinks: .collect() (finite only — an unbounded stream errors stream_unbounded, use .take(n) first), .each(f), .sum() .first() .len() .sort() (collection sinks fall through on a finite stream), .into(channel(name)), .save(path), .feed(cmd).

# spec/cases/streams.toml — real, passing cases
[1, 2, 3, 4, 5, 6].stream().where(x => x % 2 == 0).map(x => x * 10).collect()
                                                          # -> [20, 40, 60]
[1, 2, 3, 4].stream().scan(0, (a, x) => a + x).collect()  # -> [1, 3, 6, 10]
[1, 1, 2, 2, 2, 3, 1].stream().dedupe().collect()          # -> [1, 2, 3, 1]
(1..5).stream().map(x => x * x).collect()                   # -> [1, 4, 9, 16]
"error: a\ninfo: b\nerror: c".stream()
    .where(l => l.starts_with("error")).collect()            # -> ["error: a", "error: c"]

let s = [1, 2, 3].stream()
s.collect()
s.collect()                                                    # -> error, stream_consumed

channel("x").events().collect()                                 # -> error, stream_unbounded
# the tail/watch/every sources are non-deterministic (real time, real fs events) so they're
# covered by shoal-eval's own unit tests rather than the corpus, but the shape is real and runs:
tail("app.log").where(.contains("ERROR")).each(render)
watch("src/**/*.rs").debounce(200ms).each(_ => cargo test)
every(1s).map(_ => ps.where(.name == "myapp")).each(render)

In-language channel(name) (emit/latest/events/take, ring-buffered with a seq cursor) is the same substrate the wire's user.{name} events use — see Agent-Surface §7 for the pair-shelling use case and the exact method list.

5. Tasks, spawn, and jobs

spawn { … } produces a task (structured concurrency — children are cancelled with it). task.await() / .cancel() / .suspend() / .resume(); parallel(a, b, …) and .parallel(f, max: n) gather with fail-fast by default (settle: true collects errors instead). jobs is the live task table. Ctrl-C cancels the foreground task tree (SIGINT to the process group, escalating); Ctrl-Z suspends and returns to the prompt; fg <task> re-fronts it with a PTY. The kernel is the parent of every job — detaching a client never kills what it started; background output is buffered per-task and rendered as discrete blocks on resume.

At the kernel level (see Kernel-and-Agents), exec {asynchronous: true} returns a task ref immediately and the kernel runs the same evaluation on a worker thread; task.list / task.await / task.cancel operate on it. task.suspend is implemented as an explicit, honest error today ("task suspension is unavailable for evaluator-owned processes") rather than a silent no-op.

6. cwd, env, and with — no ambient mutation

The session owns cwd and env. cd and env.NAME = v are legal at session top level (journaled) and illegal inside fn bodies — the error names with cwd: / with env: as the fix, because a function silently mutating session-global state is exactly the ambient-mutation disease export causes in bash. with cwd: p, env: {…} { … } scopes both dynamically and restores on any exit path — including an error thrown inside the block.

# spec/cases/reef.toml — cwd/env purity is a load-bearing regression fence,
# because reef's project-scope resolution (see [[Reef]]) depends on it.
let before = pwd
with cwd: "sub" { }
let after = pwd
before == after                          # -> true

let before = pwd
try { with cwd: "sub" { 1 / 0 } } catch { null }
let after = pwd
before == after                          # -> true   (restores even when the body raises)

fn f() { cd "sub" }
f()                                       # -> custom error, "with cwd:" in the message

env.SHOAL_REEF_ENV_ASSIGN = "via-assign"
sh { printf '%s' "$SHOAL_REEF_ENV_ASSIGN" }.out    # -> "via-assign" (session env write, child observes it)

with env: {SHOAL_REEF_MARKER2: "set"} { (env SHOAL_REEF_MARKER2).out }   # -> "set"

with reef: {tool: "version"} { … } is the same dynamic-scoping primitive applied to tool resolution — see Reef.

Session-scoped cwd navigation beyond plain cd, all top-level-only for the same reason cd is (and all routed through one change_cwd choke point so directory history keeps recording):

  • cd - returns to the previous directory via a session-scoped OLDPWD; erroring custom ("OLDPWD" in the message) if nothing has been recorded yet.
  • pushd dir / popd / dirs — a directory stack. pushd dir cds and pushes; no-arg pushd swaps the top two instead; popd pops and cds there (custom error, "empty", on an empty stack); dirs returns the stack as a list<path>, current directory first, so it dot-chains (dirs.len(), dirs.first()).
  • jump/j — frecency-ranked cd: ranks previously-visited directories by a recency+frequency score and cds to the best match for a fuzzy query.
# spec/cases/dir-stack.toml
let start = pwd
pushd "sub"
dirs.len()                                # -> 2
popd
start == pwd                              # -> true

Cross-links

  • Language-Reference — the grammar and coercion rules feeding every call here.
  • Reef — how the command head resolves to a binary before spawn ever happens.
  • Leash-and-Security — what gates a spawn before it's allowed to happen at all.
  • Agent-Surface — shoal_exec {position: "stmt"|"value"} is this exact rule, over the wire.

Clone this wiki locally