Repository navigation
v0.5.0
Fixed
-
One slow MCP tool call bricked the whole scheduler until a manual restart —
a tool call that consumed the run budget and expired mid-read poisoned the
shared MCP client (internal/mcp), and nothing ever recovered it: the
registry held one client per server for the process lifetime, so every
subsequent tool call from every agent short-circuited with
client poisoned by prior read timeout(a 7.5-hour outage in production).
Registrynow serializes an on-demand recreate path (Registry.Recreate);
onmcp.ErrPoisonedthe executor recreates the client once and retries, so
a poisoned transport self-heals on the next call instead of requiring a
leather serverestart. -
Failed MCP tool calls were silently treated as successes —
cmd/shell-mcp
returned execution failures as ordinaryerror: ...text with noisError
flag, sointernal/mcpcouldn't distinguish a failed call from a real
result. The failure survived three layers as plain text and reached the
runner's dedupe map as a success, permanently blocking retries of a call
that never actually happened — the root cause of a production incident
where IP ban deployments were silently dropped for 6+ hours.shell-mcp
now setsisError: trueon execution failure;internal/mcp.Callreturns
a typed*ToolError; the executor stops retrying deterministic tool
errors instead of burning its retry budget; and the runner's dedupe map is
never populated by a failed call. -
Duplicate tool calls asserted unverifiable success — the dedupe map
stored only a boolean, so a repeated identical call was answered with
"already completed successfully earlier in this run" and no result
content, letting the model narrate success it never observed. Duplicate
calls now replay the cachedmodel.ToolResultwith a
[replay: identical call completed earlier this run; result repeated below]prefix. Zero-argument write tools could also never re-run within
a run regardless of intervening world-state changes;ToolDefinition
gains a per-toolmax_repeats(skill yaml), with-1disabling dedupe
entirely. -
An out-of-scope tool call was run-fatal — a tool call outside the
current turn's scope (hallucinated, prompt-injected, or recalled from the
system prompt after a context clear) was correctly never executed, but the
rejection failed the entire run, and the temperature-0 retry reproduced it
deterministically — one recoverable model mistake became a dead work item.
On the sig-triage eval a 4B model re-called a system-prompt-mentioned tool
on atools: []turn in 435/471 decide rounds, dead-lettering 214/250
issues; a 35B never attempted the call on the identical config, which is
why the failure mode was invisible until a tool-happy small model probed
it. The executor now answers such calls with a tool-result error
(tool X is not available on this turn), still never executing them; a
model that keeps calling anyway is bounded by max tool rounds, and refused
calls never populate the dedupe cache. -
shell-mcpemitted invalid JSON Schema for zero-argument tools — a
tool declaring no arguments serialized its schema as"required": null
instead of"required": []. Invisible undertool_choice: auto, but any
backend that compiles the schema into a grammar rejects every call with
400 Grammar error: Expected array for 'required', got null. The field
now always marshals as an array. -
Editor JSON schemas flagged valid, runtime-supported fields — the
hand-maintainedschemas/*.schema.yamlhad drifted from the runtime
validators ininternal/schema/defs.go:toolsetswas missing from
agent-1(while present inlifecycle-1),persist_runs_detail/
persist_runs_tool_capfromconfig-1, and per-toolmax_repeatsfrom
skill-1, so editors marked working configs as errors. The missing fields
are now declared; full codegen fromdefs.goremains an open follow-up. -
shell-mcpper-tool limits were silently ignored — per-tool
timeout_secondsandoutput_cap_bytesin shell-tools config were parsed
but never applied, so every tool ran at the global defaults (30 s /
4000 bytes), silently truncating large results mid-output. Both are now
enforced, and a call missing a required argument returns a proper tool
error instead of executing the command with literal{{placeholder}}
argv. -
go installbinaries self-identified asdev (none)(#49, #50) — the
version stamp came only from the Makefile's-ldflags, which plain
go install github.com/TGPSKI/leather/cmd/{leather,shell-mcp}@<tag>never
applies (the installs themselves work as of the v0.4.1 module-path fix;
verified against the module proxy). Both binaries now fall back to the
embedded Go build info — the module version for@taginstalls, the VCS
revision for plaingo buildfrom a checkout.shell-mcp --version(also
-v/version) now prints it too, instead of failing with
read config --version: no such filebecause the argument was taken for a
config path. -
queue_input:in agent frontmatter was accepted but silently ignored —
bothAgentFrontmatterSchemaandagent-1.schema.yamladvertised the
field ("a paired lifecycle takes precedence"), andleather validate
passed it, but only the lifecycle parser actually read it, so a
frontmatter-only agent never drained its queue and nothing said why. The
frontmatter parser now honors it; lifecycle still wins when both are set.
Added
-
Per-tool-call timeout — a single run-level timeout previously governed
every LLM call and every tool call, so one slow tool (e.g. an 89-repo
fetch) would consume the entire run budget and expire the shared MCP
transport read mid-call. Newtool_timeoutconfig (global, default600s;
envLEATHER_TOOL_TIMEOUT; flag--tool-timeout), overridable per agent via
front-matter / lifecycletool_timeout:, wraps each tool call in its own
child deadline. A tool that exceeds it fails a single call cleanly
(tool X exceeded tool_timeout …) while the run continues; the run-level
timeoutremains the outer budget.0disables the per-tool deadline. -
Structured tool-call traces in run records —
.state/runs/*.jsonl
previously recorded only turn-level text, making tool-call forensics
(exact argv, result content, error, timing) unrecoverable without source
archaeology.Turngains an opt-intool_callsfield (ToolTrace: name,
redacted args, capped content, error, replay flag, duration) controlled by
newpersist_runs_detail: none|toolsconfig (defaultnone, byte-identical
legacy output) andpersist_runs_tool_cap(default 2048 bytes/field). Args
are redacted viainternal/secretbefore persisting. -
Per-turn context clearing —
clear: trueon a turn section — leather
had per-turn tool scoping (tools:/skills:/toolsets:per turn)
but no per-turn context bounding:Session.Resethad zero production
callers, so context only ever grew, and every turn inherited everything —
including the model's own intermediate speculation (measured on the
sig-triage eval: a three-turn agent grew 1206 → 2828 prompt tokens across
its turns and lost ~9 points to its two-turn sibling on paired per-issue
comparison). A turn may now
declareclear: truealongsidetools:/skills:/toolsets:: the
conversation is reset before that turn's prompt is added; the system
message and turn variables survive, because skillextract:captures live
outside the session. That pairing is the point — distil a large tool
result into{{key}}, then discard the raw blob. The alternative
(splitting into two curings for a fresh session) was measured too and
costs information: the handoff discarded the correct answer on 9.6% of
issues; per-turn clear keeps one agent reasoning across the boundary. -
examples/14-sig-triage— assign a Kubernetes SIG to unlabeled issues
with a small local model, with the accuracy claims measured rather than
asserted: the same frozen 4B model scores 62.4%–81.6% on a 250-issue gold
corpus depending only on the runtime design around it. The example ships
the eval harness that measured it (tiered corpus, ablation-arm registry,
paired per-issue verdicts, evidence archives with replayable analysis)
alongside the winning curing design; method and verdicts are documented in
docs/LEP-0006-group-evals.mdanddocs/LEP-0008-conditional-routing.md. -
doclint — a documentation-consistency gate (
scripts/doclint, CI
workflowdoclint.yml) that cross-checks doc claims against the source,
with per-linedoclint:allowescapes and a file-level
doclint:disable-filedirective for documents that quote drift by design
(audit reports); a missing docs root fails closed. Env vars leather sets
for child processes (os.Setenv, e.g.LEATHER_INTAKE_URL) count as
referents. The full doc set — subagent guides,docs/GUIDE.md, schemas —
was reconciled against the code to bring the gate to zero violations:
phantom flags/env/endpoints removed or marked planned, the stale
exec.*shell-tools form replaced with the realcommand/argsschema,
tool-name casing corrected to snake_case,/metricsdocumented as the
JSON it actually returns, and UI docs repointed from/runsto the real
/jobs+/history. -
Offline LLM fixture:
llm_record/llm_fixture— there was no
supported way to run a full pipeline end-to-end without a live model:
MockLLMwas reachable only fromtest-agent, so proving wiring in CI
meant hand-rolling a scripted OpenAI-compatible server.--llm-record capture.jsonlnow wraps the live client and captures every completion
(including tool calls) to JSONL;--llm-fixture capture.jsonlreplays it
instead of calling a model — one recorded completion per call, in order,
failing loudly with the call index and last message when a run diverges
from its recording. serve, run, and workflow-run share one client per
process so replay order spans jobs.make 06-smokeis the working proof:
the full ingest → triage → summarize → artifact pipeline of example 06,
modelless, failing the target if no artifact is produced. -
leather validatecoversshell-tools.json— the most format-fiddly
hand-edited artifact was the only one with no schema and no validate
coverage; a malformed tools file passedleather validateand failed only
at runtime inside shell-mcp, as a silently tool-less agent.
schema.ValidateShellToolsJSONnow validates every*.jsonreferenced
from anmcp-servers.yamlcommand line: required fields, the removed
exec.*/argvforms, unknown fields, RE2 pattern compilation, snake_case
and duplicate names. A matching editor schema ships as
schemas/shell-tools-1.schema.yaml. First catch: example 09 declared
"timeout": N, a field shell-mcp silently ignores — its tools had been
running at the 30 s default since they were written (now
timeout_seconds). -
Schema ↔ runtime parity is now enforced — a test
(internal/schema/parity_test.go) asserts every runtime validator field
indefs.goappears in the correspondingschemas/*.schema.yamland
vice-versa (nested-only blocks declared explicitly), so the editor-schema
drift class cannot recur; the YAML schemas stay hand-written because their
descriptions carry operator guidance codegen would flatten. First catches:
agent frontmatter acceptedtoolsets,tool_timeout, andthinking
while bothdefs.goandagent-1.schema.yamlomitted them. -
make new-example NAME=<slug>— adding an example was tribal
knowledge (pick the next index, hand-register Makefile targets, copy
pretty.sh, sourcepreflight.sh). The scaffolder allocates the index,
creates the standard tree, appends theNN/NN-livetargets, and prints
the two hand-written registrations; the convention itself is now
documented inexamples/README.md. -
docs/CONVENTIONS.md— central environment-variable reference (name,
default, scope, effect) covering the binary's load-bearing vars, shell-mcp,
and the example-shell contract (LEATHER_DEMO_MODEand the dry-mode
idiom, example 13's git vars, webhook/GitHub tokens). The GUIDE config
reference now also documentsllm_endpoint/llm_api_key,
mcp_servers_file(without it, agents run tool-less with no error),
queue_patternroutes, and theworkflow runwebhook-secret coupling.