Self Coding

Self-Coding

shelldon can call real tools mid-conversation, write brand-new tools (code plus a test) for itself, and grow its own toolbox over time — all behind owner approval, resource-capped, on a 416MB Raspberry Pi Zero 2W.

This page documents the system as built: the native function-calling loop, the two safety tiers (FREE and RISKY), the owner-approval flow over Telegram, persistent self-coded tools and their gate, and the layered safety model that keeps it from wedging or bricking the Pi.

See also: The Brain, Architecture, Memory, Telegram Transport.

The shape of it

The pet's brain is a text-in/text-out LLM reached through a broker. Self-coding adds a bounded agentic tool loop that runs entirely inside the per-turn fork worker:

core --fork--> worker
                 │  assemble prompt + tool schemas
                 ▼
        ┌──────► Job(messages, tools) ──► broker ──► provider.complete_with_tools
        │                                                  │
        │        Completion(text | tool_calls) ◄───────────┘
        │                 │
        │          tool_calls? ──no──► parse ops ──► Result ──► core (replies + applies)
        │                 │ yes
        │            tier check
        │             ├─ FREE  ─► run in worker, feed ToolResult back ─┐
        │             └─ RISKY ─► emit RequestToolApproval, end turn   │  (2-phase)
        └────────────────────────────────────────────────────────────┘

Three invariants shape every design choice here:

1. Core stays LLM-free. The loop and the tools live in the worker, never in core/. Import-linter enforces this. Core's only role in self-coding is to persist, gate, and route — it never imports model code or an LLM SDK.
2. The broker executes nothing. It only normalizes each provider's native tool-call format to/from shelldon's own ToolCall/ToolResult contracts. It shuttles requests and results; it never calls a tool function.
3. The fork accumulates nothing. The worker forks per turn and dies. The whole loop runs inside one fork. A turn that needs human approval ends the worker and resumes later in a fresh fork from persisted state — no worker ever blocks waiting on a human.

The function-calling foundation

The provider seam gained one method alongside the existing text-only complete():

# shelldon/broker/provider.py
complete_with_tools(messages, tools) -> Completion

Each provider adapter implements it and normalizes its native shape into closed contracts (shelldon/broker/anthropic_provider.py, shelldon/broker/openai_provider.py). GLM on Z.ai uses the Anthropic-compatible endpoint, so the Anthropic normalizer covers it. The worker loop sees only shelldon's own vocabulary — no provider format ever leaks into it.

The contracts (shelldon/contracts/__init__.py) that carry the loop:

• ToolDefinition(name, description, params_schema, tier) — the serializable tool spec that travels to the broker.
• ToolCall(id, name, args) and ToolResult(id, ok, content) — the SDK-agnostic tool-call vocabulary the broker normalizes.
• Message(role, content, tool_calls, tool_call_id) — a multi-turn conversation message (user / assistant / tool).
• Completion.tool_calls — a completion is either final text or a set of tool-calls to continue the loop.

The bounded loop

shelldon/worker/worker.py::_agentic_loop assembles the prompt plus the tool schemas, sends a Job, and reads a Completion. If it carries tool_calls, the worker runs each FREE tool, appends a ToolResult message, and loops. If it's text, the worker parses the ops block and returns a Result.

Two hard ceilings bound it:

• _MAX_TOOL_EXECUTIONS = 6 — at most six tool runs before the model must answer with text.
• _COMPLETION_TIMEOUT_S = 25.0 — the whole turn's budget. Each iteration tracks remaining time and bails (with a best-effort reply) if under 2s are left.

These are also safety controls: they cap the model-call-per-turn cost and stop any infinite tool-to-model ping-pong.

Everything fails soft. A tool that raises, an unknown tool name, a malformed call — all caught and fed back as ToolResult(ok=False, content=<error>) so the model can recover. The turn never crashes (execute_tool in shelldon/worker/tools.py).

The FREE tier — inline, no friction

FREE tools run synchronously in the loop with zero approval. They are read-only or pure-compute, so there's nothing to gate.

Tool	What it does
get_time	Current local date/time as ISO-8601. The trivial tool that proved the loop end-to-end.
read_file(path)	Read a text file inside the workspace jail (capped at 64 KB).
list_dir(path)	List a directory inside the jail.
python_eval(code)	Evaluate one pure-Python expression in a restricted namespace, wall-clock bounded (~2s).

The path jail

All FREE file tools are jailed to a single WORKSPACE_ROOT (default ~/.shelldon/workspace). _resolve_in_jail (shelldon/worker/tools.py) resolves the path to its real location — dereferencing symlinks before the containment check — and rejects anything that escapes the root. An absolute path drops the root in the join and so also fails. A symlink inside the workspace pointing out is caught.

On top of the jail, _deny_sensitive refuses the secrets tree and credential-shaped files regardless of tier: the vault/ directory (at <memory_root>/vault), .env / *.env / .env.*, key/cert suffixes (.pem, .key, .crt, .p12, .pfx, .htpasswd, ...), and any id_* private-key file. This is defense in depth — the uid-drop is a no-op on the non-root Pi, so the deny list is the real boundary.

python_eval is a restricted namespace, not a sandbox

_python_eval parses the snippet in eval mode (so a statement or import is a syntax error), runs _assert_eval_safe against the AST, then evaluates with only a curated _SAFE_BUILTINS map. The guard:

• Omits every side-effecting builtin — no open, __import__, eval, exec, compile, globals, getattr/setattr, input. A snippet reaching for the filesystem raises NameError and fails closed.
• Blocks the restricted-eval escapes — any attribute or name starting with _ (covers all dunders and the ().__class__.__mro__[-1].__subclasses__() graph walk), plus str.format/format_map (the "{0.__class__}".format(obj) getattr trick).
• Wall-bounds via SIGALRM in the worker's main thread; the timer is disarmed before the handler is restored so no stale handler is ever left installed.

This is explicitly not a true sandbox — it makes the common, model-likely escapes fail closed. Real isolation is the RISKY-tier and resource-cap concern below. The 25s loop ceiling and the worker's RLIMIT_AS backstop a runaway compute.

The RISKY tier — owner approval over Telegram

RISKY tools change something: write_file, run_shell, http_get, git. Each requires an explicit owner tap before it runs.

The central tension: the worker is fork → one turn → die in <25s, but human approval over Telegram is async and may take minutes. The worker cannot block on it. So RISKY tools use a two-phase resumable flow, not an in-loop await.

Phase 1 — pause and park

When the loop hits the first RISKY call, it does not execute it. It ends the turn emitting:

• RequestToolApproval(call, summary, messages) — a proposed op carrying the pending call and the running message list.
• A user-facing reply ("I'd like to run X — approve?").

The worker dies normally (fork reaped, arbiter slot freed). Core encodes (messages, call) as a msgpack blob and parks it in the sqlite pending_approvals table, keyed by turn id with an expires_at (default 1h TTL). The approval now waits out-of-band in sqlite — no process is alive during the human's think-time, so the coherent-timeout invariant is untouched.

If a completion mixes FREE and RISKY calls, the worker runs the FREE prefix, keeps the assistant message to [FREE prefix + the risky call] (so every tool_use block gets a tool_result and no provider rejects the next request), and parks. Calls after the first risky one are dropped — the model re-requests them on resume.

The Telegram surface

The transport (shelldon/transport/telegram.py) renders the approval as an inline keyboard:

✅ Approve     ❌ Deny

with callback_data carrying the turn id. Tool output is sent with parse_mode="HTML" and wrapped in <pre> blocks. A callback_query clears the client spinner via answerCallbackQuery. setMyCommands registers the command set on startup. The ALLOWED_USERS gate applies to callbacks too. (See Telegram Transport.)

Phase 2 — tap and resume

The owner's tap arrives as InboundMessage(approval_turn_id, approved). Core routes it — before the arbiter, since a decision is not chat — to _handle_approval_decision. Core takes the parked blob and spawns a fresh worker (spawn_resume) that rebuilds the loop:

• On Approve, it executes the pending call.
• On Deny, it synthesizes ToolResult(ok=False, content="denied by owner").

Either way it appends the ToolResult and continues the bounded loop to a final reply. The flow is re-entrant: a resumed loop that hits another RISKY call parks again.

A resume is a turn (it needs the fence, arbiter, reap). If a worker is already in flight, _handle_approval_decision leaves the approval parked and replies "tap again in a moment" — the sqlite row is consumed only when about to spawn. An expired or unknown decision is dropped (never executes), logged, and the owner gets a brief "that approval expired" note. Malformed frames (e.g. approved=None) fail safe to deny.

Persistent self-coded tools — the actual self-coding

This is the headline capability: the pet writes a new named tool that joins its permanent toolbox and is live on the next turn.

Propose

The model emits a ProposeTool(name, code, test) op in its reply — the tool implementation and its pytest test together. It rides the existing ops-block wire (the same path as remember/add_face), so there is no worker-loop change. A self-coded tool module follows a simple convention with no shelldon imports needed:

# module level — what discovery looks for
def run(**kwargs) -> str: ...
DESCRIPTION: str = "..."
PARAMS_SCHEMA: dict = {...}

Stage and gate (core, LLM-free)

shelldon/core/selfcode.py owns the workspace tool directories and is the sole writer of them (AD-5):

<workspace>/tools/             # live — discovered FREE by the worker
<workspace>/tools-staging/     # staged + gated, not yet live
<workspace>/tools-quarantine/  # repeatedly-bad tools, out of discovery's reach

stage() writes <stem>.py plus test_<stem>.py to the staging dir (source size capped at 64 KB; a Python-keyword stem like class is suffixed to class_tool so it stays importable; a stale prior pair is cleared first).

Then run_gate() runs two checks, and only on a pass does the tool advance:

1. An AST import-check (_forbidden_import). It scans the staged source and rejects any import of an LLM SDK (openai, anthropic, google, litellm, zhipuai, ollama) or shelldon.core. This is how core stays LLM-free while running model-written code: it never imports the tool — it AST-scans the text, then runs the test as a subprocess. It also catches __import__("anthropic") / importlib.import_module("openai") with string-literal args; a non-literal (dynamic) arg is logged as unverifiable (owner approval is the backstop).
2. A bounded pytest subprocess. asyncio.create_subprocess_exec(sys.executable, "-m", "pytest", "-q", ...) from the staging cwd, with a wall-clock timeout (default 30s — a timeout kills the subprocess and is a fail) and an RLIMIT_AS/RLIMIT_CPU preexec_fn. The combined output is captured and capped.

The gate runs untrusted, model-written code before the owner approves it — running the test inherently imports and calls the tool. This is accepted for a single-owner device: it's bounded by the subprocess timeout and resource caps, and the owner still reviews the code at the Approve step. A failed gate discards the staged pair and tells the owner why; it is never promoted.

Approve and promote

A passing gate parks a pending_promotions row (parallel to pending_approvals, keyed by turn id) and sends the same Story 9.3 inline keyboard ("I wrote a tool name and it passed its test — add it?").

• Approve → promote() moves <stem>.py from staging to the live dir and drops the staged test. The tool is now live.
• Deny → discard() deletes the staged pair.

Promotion needs no worker slot (it's a file move plus a reply), so _handle_approval_decision checks the promotion table before the RISKY-resume path.

Discover — free on the next turn, no restart

The worker discovers live tools the way the plugin host discovers plugins. discover_self_coded_tools (shelldon/worker/tools.py) imports each *.py in the live dir via importlib.util.spec_from_file_location, builds a FREE-tier ToolSpec from its run/DESCRIPTION/PARAMS_SCHEMA, and merges it into the registry. A discovered tool may not shadow a built-in (built-ins win, the collision is logged). A module that fails to import or breaks the convention is skipped and logged — it never wedges the worker.

Because every turn forks a fresh worker that re-imports, the next turn picks up the new tool automatically — no process restart. This fork-reimport property is the whole reason self-coding is cheap in Python. Self-coded tools are portable data: they live in the workspace, version with memory, and travel across bodies. Core stays pure; the worker is the mutable surface.

Self-coded tools run FREE once promoted — the one-time review at promotion (the gate plus the owner approving the code) is the safety boundary. A self-coded tool that itself needs a risky action calls the built-in RISKY tools, which still gate per-call — so it can't silently escalate.

Safety hardening — keeping the Pi alive

Three pillars (shelldon/core/limits.py, shelldon/core/selfcode.py, shelldon/core/history.py) keep broken or runaway tools from wedging or bankrupting the pet.

Quarantine — strike ledger

A live tool that errors on import or raises on a call is skipped and logged (the turn survives). But repeated failures get it removed. The pattern is worker detects, core decides and moves:

• The worker reports failing self-coded tool names on Result.tool_failures (additive field — discovery skips plus run-failures; a self_coded flag on ToolSpec attributes a run-failure to a self-coded tool, never a built-in).
• Core debits a per-tool strike count in the tool_health sqlite table (atomic UPSERT).
• At the threshold (default 3 strikes), core moves the live module to tools-quarantine/ via selfcode.quarantine — so the next fork's discovery no longer sees it (the faces-registry single-writer-plus-move pattern). Restore is manual; there is no auto-rehabilitation.

Resource caps — RLIMIT_AS / RLIMIT_CPU

core/limits.py sets address-space and CPU caps so a tool can't OOM or CPU-peg the 416MB Pi:

• On the worker fork (in the fork child, before the turn runs) — bounds everything the turn runs: python_eval, any FREE self-coded tool, the loop. A breach raises a clean MemoryError/SIGXCPU caught fail-soft, and the fork dies each turn so the cap is per-turn-clean.
• On spawned children via a preexec_fn — the run_shell/git runner and the gate pytest subprocess set the same caps, so a child can't escape the worker's bound.

Defaults are RLIMIT_AS ≈ 1 GiB (a gross-runaway catcher well above the ~244MB RSS peak) and RLIMIT_CPU ≈ 30s (above the 25s loop ceiling). Each setrlimit is guarded — an unsupported limit or an over-hard-limit value logs and continues. The layering: python_eval's SIGALRM trips at Python-bytecode boundaries; RLIMIT_AS is the C-level/memory backstop SIGALRM can't give (a tight C call like bytearray(10**10) never yields to SIGALRM); the systemd MemoryMax=400M cgroup (Deployment) is the hard physical backstop. Linux/the Pi is the enforcement target.

Credit / loop gating

The loop ceiling (_MAX_TOOL_EXECUTIONS) is the hard per-turn model-call cap, so worst-case self-driven spend = daily_turn_budget × loop-ceiling. Any scheduler-initiated turn that can run the loop carries a turn-count cost weight, so the daily budget bounds total self-driven spend. Owner-initiated turns stay un-budgeted but loop-ceiling-bounded. This reuses the existing budget gate verbatim — no new accounting.

Tool-policy hardening — raising the floor under RISKY tools

The RISKY tools already gate on the owner tap, but a benign-looking approval shouldn't be turnable against the pet. These guards (all worker-side, in shelldon/worker/tools.py) make the dangerous-but-plausible commands fail closed. They change only what runs after Approve — the approval flow is untouched.

http_get — SSRF protection + streaming cap

http_get follows redirects manually (follow_redirects=False, max ~5 hops), re-validating each hop's host. _assert_host_allowed resolves the host via socket.getaddrinfo and checks the resolved IPs (so evil.com → 10.0.0.1 is caught, not just string-matched):

• Every hop: loopback / link-local (covers the 169.254.169.254 cloud-metadata IP) / unspecified / multicast → rejected.
• Redirect hops only: private and reserved ranges → rejected. The initial owner-approved URL is exempt (the owner explicitly approved that host, so a LAN fetch they typed is allowed; a redirect into internal space is the attack).

The body is streamed with a pre-read byte cap (iter_bytes to _MAX_TOOL_OUTPUT_CHARS) so a multi-MB response never fully buffers into RAM. http(s)-only, no URL-embedded credentials (creds are broker-only). An unresolvable host fails closed.

git — subcommand allowlist

_git splits args safely (no shell) and requires the subcommand to be in a closed allowlist of read and local-history verbs:

status log diff show add commit branch checkout switch restore
stash init fetch pull push remote tag rev-parse reset mv rm

Anything else (clone, submodule, daemon, archive, ...) is rejected. The exec/pack/repo-redirect specifiers (--upload-pack, --receive-pack, --exec, --namespace, --git-dir, --work-tree, --exec-path, --config-env) are rejected anywhere in the args; the -c/-C config-injection and chdir short flags are rejected as global flags. config is deliberately excluded — git config core.sshCommand=… would persist a hook that turns a later approved fetch/commit into code execution.

run_shell — process-group cleanup

_run_subprocess runs the child in its own session/process group (start_new_session=True). On timeout and on normal exit, the whole group is SIGKILLed (os.killpg) so a backgrounded child (cmd &, disown, a daemon) can't outlive the turn. The original timeout still raises → ToolResult(ok=False). The 9.5 RLIMIT preexec_fn is preserved.

Credential blocklist

_deny_sensitive refuses, case-insensitively: .env / *.env / .env.* variants, the vault/ tree, key/cert suffixes (.pem, .key, .crt, .cer, .p12, .pfx, .htpasswd, .jks, .ppk), and any id_* private-key file. This also tightens the FREE read tools — intended defense in depth.

The layered safety model, at a glance

Everything below the conversation runs behind owner approval, resource-capped, on the Pi:

Layer	Guard
Tier boundary	FREE = read-only/pure-compute, inline. RISKY = writes/shell/network/git, owner-tap-gated.
Owner approval	RISKY calls and tool promotions both pause for a Telegram Approve/Deny tap; the owner sees the exact command/URL/code.
Path jail + deny list	File tools jailed to one workspace root; vault and credential files always denied.
Restricted eval	python_eval blocks side-effecting builtins, dunder/MRO escapes, and str.format.
Gate	Self-coded tools pass an AST import-check (no LLM/core imports) + a bounded pytest subprocess before promotion.
Quarantine	A tool failing 3 times is moved out of discovery.
Resource caps	RLIMIT_AS/RLIMIT_CPU on the fork and every spawned child; systemd MemoryMax=400M is the physical backstop.
Loop ceiling	6 tool executions / 25s budget per turn caps cost and stops infinite loops.
Tool policy	SSRF block, git allowlist, shell process-group cleanup, credential blocklist.
Fail-soft	Every guard raises → ToolResult(ok=False); the turn survives.

Key files

File	Role
shelldon/worker/tools.py	Tool registry, FREE + RISKY tools, the path jail, discovery, the policy guards.
shelldon/worker/worker.py	The bounded agentic loop and the RISKY pause/resume.
shelldon/core/selfcode.py	Stage / gate / promote / quarantine / discard — the workspace owner (LLM-free).
shelldon/core/limits.py	RLIMIT_AS/RLIMIT_CPU caps for the fork and spawned children.
shelldon/core/history.py	sqlite tables: pending_approvals, pending_promotions, tool_health.
shelldon/core/runtime.py	Routes proposals, decisions, resumes, and the strike ledger.
shelldon/broker/provider.py	complete_with_tools — the provider seam that normalizes tool-calls.
shelldon/contracts/__init__.py	ToolCall, ToolResult, ToolDefinition, Message, RequestToolApproval, ProposeTool.