You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Performance improvements specifically with Bob (#106)
* Improve sub agents with bob. Add harbor tests for bob
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000068_task-goal-validation: per-task goals checked by an independent reviewer
Each implement task's acceptance criteria and scope limits become a goal.
After verification, a separate read-only sub-agent with a fresh context
judges every criterion as met, not yet met or impossible, with evidence.
Only met criteria are ticked; not-yet-met work is retried up to twice and
then handed to the user; impossible stops at once. The reviewer runs on a
cheaper model tier where the agent allows (Claude, Codex, Copilot, omp),
on the default model for Bob, is on by default, and can be switched off
or given a per-agent model override. References the plan-task-graph
design, whose ticking rule this changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000069_compact-agents-md: short rules in AGENTS.md, full guidance on demand
Spektacular's eight managed AGENTS.md sections (~22 KB) load into every
session. Each shrinks to a short binding rule plus its trigger and a link
to full guidance in files under .spektacular/, read only when needed,
targeting under 5 KB. init installs the compact form for every agent,
migrate converts existing projects and keeps the guidance current, user
content is left untouched, CLAUDE.md's import is unchanged, omp's rule
files keep the same binding rules, and the docs site is updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000069_compact-agents-md: compact AGENTS.md rules with on-demand guidance
Plans shrinking Spektacular's eight standing rules in AGENTS.md (~22 KB) to
short statements under 5 KB, with each rule's full guidance installed under
.spektacular/guidance/ and read only when needed. A shared table-driven
installer serves all five agents; init and migrate install, convert and
refresh it. Includes a new harbor standing-rules suite for a baseline, Go
tests for conversion, idempotency, refresh and the size budget, and docs
site updates.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000068_task-goal-validation: independent goal check per task
Plans a new implement step, check_goal, between verify and update_plan.
A fresh-context, read-only reviewer judges each acceptance criterion
met, not_met or impossible with evidence. The CLI builds each task's goal,
validates and records verdicts in the plan's goals.json, and bounds
retries at two per round before asking the user. update_plan ticks only
met criteria, and implement.goal_check can switch the check off.
The plan has 12 agent tasks over 4 milestones:
- plan reader, goal package, `implement goal record|decide|show` commands,
config, FSM step and templates;
- per-agent spek-goal-reviewer installs on cheaper models (Bob uses
explore), and an epic merge guard while a goal-check question is open;
- implement/implement-epic harbor updates and a new goal-check harbor
suite;
- the plan-task-graph design tick rule and the docs site pages.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Add plans
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000069_compact-agents-md: Milestone 1, standing-rules harbor suite
Add tests/harbor/standing-rules, an end-to-end suite whose image runs
`spektacular init <agent>` at build time, so Spektacular's standing rules
in AGENTS.md are the agent's real project instructions from its first turn.
The project registers a code repository outside /app and seeds a final
spec. The session-log verifier checks that the agent consulted the repo
registry and answered from the registered code, read the spec only through
the CLI and left it unchanged, and offered rather than started a spec. A
local self-check proves each rule passes and fails in both directions.
Make targets run it for Claude, Codex and Bob. The Bob harbor agent moves
to tests/harbor/agents/, where the Makefile already looks, so Bob runs of
every suite can start.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000069_compact-agents-md: Milestone 1, standing-rules harbor suite
Add tests/harbor/standing-rules, an end-to-end suite whose image runs
`spektacular init <agent>` at build time, so Spektacular's standing rules
in AGENTS.md are the agent's real project instructions from its first turn.
The project registers a code repository outside /app and seeds a final
spec. The session-log verifier checks that the agent consulted the repo
registry and answered from the registered code, read the spec only through
the CLI and left it unchanged, and offered rather than started a spec. A
local self-check proves each rule passes and fails in both directions.
Make targets run it for Claude, Codex and Bob. The Bob harbor agent moves
to tests/harbor/agents/, where the Makefile already looks, so Bob runs of
every suite can start.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000069_compact-agents-md: Milestone 2, compact standing rules with guidance on demand
Every standing rule Spektacular installs into AGENTS.md is now a short
statement under its unchanged heading: the obligation that always applies,
when to read further, and a link to the rule's full guidance in
.spektacular/guidance/<slug>.md. Spektacular's part of AGENTS.md drops from
about 22 KB to 4,254 bytes for every agent.
- One table-driven installer (internal/agent/standing_rules.go) replaces the
five agents' hand-copied lists of eight section installers, and omp's rule
writer reads the same table. The historical-artifacts rule's private copy
of the placement logic is gone.
- New short templates in templates/agents/short/. The existing full
templates become standalone guidance documents, written before the section
that links to them and refreshed on every init and migrate.
- The store-access rule now exempts .spektacular/guidance/ in both forms:
agents read those files with their own tools and never edit them.
- Existing projects convert in place through migrate or init, and the
team's own AGENTS.md content stays where it was. omp rule files carry the
same short text, and CLAUDE.md still imports AGENTS.md.
- Every pre-existing phrase test is kept on the surface that must carry it.
New tests pin the size and line budget, conversion of a pre-change
AGENTS.md, idempotency and guidance refresh for all five agents.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000069_compact-agents-md: Milestone 2, compact standing rules with guidance on demand
Every standing rule Spektacular installs into AGENTS.md is now a short
statement under its unchanged heading: the obligation that always applies,
when to read further, and a link to the rule's full guidance in
.spektacular/guidance/<slug>.md. Spektacular's part of AGENTS.md drops from
about 22 KB to 4,254 bytes for every agent.
- One table-driven installer (internal/agent/standing_rules.go) replaces the
five agents' hand-copied lists of eight section installers, and omp's rule
writer reads the same table. The historical-artifacts rule's private copy
of the placement logic is gone.
- New short templates in templates/agents/short/. The existing full
templates become standalone guidance documents, written before the section
that links to them and refreshed on every init and migrate.
- The store-access rule now exempts .spektacular/guidance/ in both forms:
agents read those files with their own tools and never edit them.
- Existing projects convert in place through migrate or init, and the
team's own AGENTS.md content stays where it was. omp rule files carry the
same short text, and CLAUDE.md still imports AGENTS.md.
- Every pre-existing phrase test is kept on the surface that must carry it.
New tests pin the size and line budget, conversion of a pre-change
AGENTS.md, idempotency and guidance refresh for all five agents.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000069_compact-agents-md: Milestone 3, document the compact AGENTS.md layout
The configuration reference gains a "Standing rules: AGENTS.md and
.spektacular/guidance/" section, which explains:
- what each short statement in AGENTS.md holds;
- the one-file-per-rule guidance folder, which is committed with the
project and read only when a rule applies;
- that init and migrate regenerate the guidance files, so they are not
edited by hand;
- how each agent reads AGENTS.md.
The migrate section says an upgrade converts the standing rules to their
short form and refreshes the guidance, leaving the team's own AGENTS.md
content in place. The omp plugin and getting-started text say the omp
rules mirror the short sections and link to the guidance.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000069_compact-agents-md: implementation records, test plan and spec reconciliation
Implementation of 000069_compact-agents-md is complete across both repos.
- spektacular: each of the eight standing rules is a short section in
AGENTS.md (4,254 bytes for every agent, down from about 22 KB), linking
to its full guidance in .spektacular/guidance/. A single table-driven
installer serves all five agents. init and migrate convert existing
projects in place and refresh the guidance. A new standing-rules harbor
suite checks agent behaviour, and the Bob harbor agent moved to
tests/harbor/agents/.
- docs: the configuration reference describes the compact layout, the
guidance folder and how migrate converts a project.
This commit records the plan changelog, the manual test plan (harbor pass
rates before and after, a Claude /context token check, Copilot and omp
sessions, a rule-by-rule review and a docs review), the project and
per-repo changelog records, and the spec reconciliation. Two acceptance
items stay open until the live agent runs in the test plan are done:
agents in every supported tool still following every rule, with no drop
in harbor pass rates.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000068_task-goal-validation: Milestone 1, implement runs check every task's goal before ticking it
Implement runs no longer tick a task's acceptance criteria straight from
passing tests. After verify, the new check_goal step renders the current
task's goal (its criteria and scope limits) and has the agent stage the
work's diff and launch a separate fresh-context, read-only reviewer that
judges each criterion met, not_met or impossible with evidence.
- internal/plantask keeps criterion text and parses legacy Phase N.M items.
- implement.goal_check config: on by default, per-agent reviewer models.
- internal/goal builds goals, validates replies, applies the retry rule
(two retries per round, impossible stops at once) and keeps goals.json.
- spektacular implement goal record|decide|show record verdicts and the
user's decisions, fingerprint the work and print the outcome.
- skill review-task-goal serves the shared reviewer brief.
- update_plan refuses until the goal check is settled and ticks only the
criteria judged met; with the check off every step renders as before.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000068_task-goal-validation: Milestone 1, implement runs check every task's goal before ticking it
Implement runs no longer tick a task's acceptance criteria straight from
passing tests. After verify, the new check_goal step renders the current
task's goal (its criteria and scope limits) and has the agent stage the
work's diff and launch a separate fresh-context, read-only reviewer that
judges each criterion met, not_met or impossible with evidence.
- internal/plantask keeps criterion text and parses legacy Phase N.M items.
- implement.goal_check config: on by default, per-agent reviewer models.
- internal/goal builds goals, validates replies, applies the retry rule
(two retries per round, impossible stops at once) and keeps goals.json.
- spektacular implement goal record|decide|show record verdicts and the
user's decisions, fingerprint the work and print the outcome.
- skill review-task-goal serves the shared reviewer brief.
- update_plan refuses until the goal check is settled and ticks only the
criteria judged met; with the check off every step renders as before.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000068_task-goal-validation: Milestone 2, a cheap read-only reviewer for every agent and safe epic stops
spektacular init now installs a spek-goal-reviewer helper in each agent's
own sub-agent format, embedding the shared reviewer brief: Claude
(.claude/agents, haiku, Read/Grep/Glob), Codex (.codex/agents TOML,
gpt-6-luna, read-only by instruction), Copilot (.github/agents, Claude
Haiku 4.5, read/search) and omp (.omp/agents, @smol, read/grep/glob).
implement.goal_check.models overrides the model for one agent only; Bob
uses its built-in explore helper. The README documents the helpers.
epic merge refuses a spec whose implement lane is stopped at check_goal
(goal_check_open), and the spek-implement-epic skill treats an exhausted
or impossible goal check as a genuine question to put to the user.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000068_task-goal-validation: Milestone 2, a cheap read-only reviewer for every agent and safe epic stops
spektacular init now installs a spek-goal-reviewer helper in each agent's
own sub-agent format, embedding the shared reviewer brief: Claude
(.claude/agents, haiku, Read/Grep/Glob), Codex (.codex/agents TOML,
gpt-6-luna, read-only by instruction), Copilot (.github/agents, Claude
Haiku 4.5, read/search) and omp (.omp/agents, @smol, read/grep/glob).
implement.goal_check.models overrides the model for one agent only; Bob
uses its built-in explore helper. The README documents the helpers.
epic merge refuses a spec whose implement lane is stopped at check_goal
(goal_check_open), and the spek-implement-epic skill treats an exhausted
or impossible goal check as a genuine question to put to the user.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000068_task-goal-validation: Milestone 3, prove the goal check end to end with real agents
The implement suite's step order and seeded state include check_goal. The
implement-epic solution walks every task of every spec through the goal
check and records its verdicts, and its verifier fails a run in which any
task has no recorded verdicts or reaches update_plan without one. Goal
commands naming a spec are now filed under that spec's lane in the session
log.
A new goal-check-workflow suite seeds a one-task plan whose README
criterion is missing while every test passes (unmet) or whose criteria all
hold (met), resumes the implement run at verify under Claude, Codex or Bob
(make harbor-test-goal-check-<agent> SCENARIO=unmet|met), and verifies the
verdicts, the retry, that nothing is ticked before the check or unless met,
that the reviewer changed nothing and saw no conversation, and on Claude
that the reviewer was a separate helper on haiku. A local self-check proves
every verifier rule can fail.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000068_task-goal-validation: Milestone 3, prove the goal check end to end with real agents
The implement suite's step order and seeded state include check_goal. The
implement-epic solution walks every task of every spec through the goal
check and records its verdicts, and its verifier fails a run in which any
task has no recorded verdicts or reaches update_plan without one. Goal
commands naming a spec are now filed under that spec's lane in the session
log.
A new goal-check-workflow suite seeds a one-task plan whose README
criterion is missing while every test passes (unmet) or whose criteria all
hold (met), resumes the implement run at verify under Claude, Codex or Bob
(make harbor-test-goal-check-<agent> SCENARIO=unmet|met), and verifies the
verdicts, the retry, that nothing is ticked before the check or unless met,
that the reviewer changed nothing and saw no conversation, and on Claude
that the reviewer was a separate helper on haiku. A local self-check proves
every verifier rule can fail.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000068_task-goal-validation: Milestone 4, document the goal check and update the task design
The docs site's workflow reference describes the implement goal check: a
separate fresh-context, read-only reviewer judges each acceptance
criterion met, not yet met or impossible, with evidence; only met criteria
are ticked, not yet met work is retried up to twice before you choose to
keep fixing, accept or skip, and an impossible criterion stops the run.
The plan tasks page states the new tick rule, and the configuration
reference documents implement.goal_check.enabled and
implement.goal_check.models.<agent> with each agent's default reviewer
model and limits. The docs changelog gains an entry.
The plan-task-graph design's acceptance-criteria rule now says criteria
are ticked only when the goal check judged them met, or from verification
with the check switched off.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000068_task-goal-validation: independent goal check for every implement task
Every implement task's acceptance criteria and scope now form a goal that a
separate, fresh-context, read-only reviewer judges met, not_met or
impossible, with evidence, after verification and before anything is
ticked. Only met criteria are ticked; not-yet-met work is retried up to
twice per round, then the user chooses to keep fixing, accept the shortfall
or skip, and an impossible criterion stops the run at once. Every check is
recorded in the plan's goals.json and read back with
`spektacular implement goal show`.
Each coding agent gets a spek-goal-reviewer helper on a cheaper model
(Bob uses its explore helper); implement.goal_check switches the check off
and picks the reviewer's model per agent. epic merge refuses a spec whose
goal-check question is open. The harbor suites, the plan-task-graph design
and the documentation site follow the change, and the implement run's
test plan, changelog records and spec reconciliation are written.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Spec 000070_harbor-run-metrics: standard performance metrics for harbor runs
Specifies a standard metrics record for every harbor run (wall and model
time, turns, cost, tokens, per-step duration and output volume, sub-agent
counts, and per-group check scores, with unknowns never recorded as zero)
for Claude, Codex and Bob, plus a local report that lists runs by suite,
shows one run's step breakdown, compares a baseline set of job directories
with a candidate set, and flags dominant steps and regressions. The 0/1
reward keeps its meaning. Sourced from issue #90.
This commit also carries the working tree's earlier uncommitted changes
(epic skill and test updates, Makefile, removal of tests/harbor/.build).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000070_harbor-run-metrics: standard metrics record and local report for harbor runs
Plans a recorder that runs in every harbor suite's verifier after the
reward is written and writes metrics.json beside it, in one shape for
every agent and suite:
- totals from each agent's own log (Claude result events, or the ATIF
trajectory for Bob and Codex)
- per-step duration from Spektacular's session log
- per-step characters written
- sub-agent counts and durations
- the JUnit check score, overall and per test class or module
- build identity from a Makefile-written build.json
Missing measures are recorded as null with a reason, never as zero.
Also plans a local report under tests/harbor/shared that reads only
verifier/metrics.json and result.json, so it needs no sudo. It shows:
- a table of runs grouped by suite
- one run's step breakdown, flagging a step over a third of the run
- a baseline-versus-candidate comparison by median, flagging 2x
regressions in time, turns, tokens or cost and check-score drops of
0.1 or more
Seven agent tasks across three milestones. They also wire the shared
helpers and python3 into the spec, plan, repo and implement suites,
move harbor-test-implement onto a .build render, and document the
record and report in the README's harbor section.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000070_harbor-run-metrics Milestone 1: every harbor run leaves a metrics record
Every harbor suite's verifier now writes metrics.json beside reward.txt,
in one shape for Claude, Codex, Bob and oracle runs:
- run identity from a Makefile-written build.json (agent, suite,
scenario, git-described version, commit, dirty)
- totals from the agent's own log: Claude's result events (single vs
several-result rule), harbor's ATIF trajectory for Bob and Codex, or
Codex's rollout as a fallback
- per-step durations from Spektacular's session log (spec, plan and
implement lanes) and characters written per step, joined by time, by
goto order for Bob, or by launch prompt across orchestrated lanes
- sub-agent counts and durations
- the check score overall and per test class or module, from pytest's
JUnit XML
Anything a log does not supply is null with a reason under `unknown`.
The recorder (tests/harbor/shared/metrics.py) runs after the reward is
written, with its failure ignored, so a run's pass-or-fail result is
unchanged. All eight render targets copy the shared helpers and write
build.json; harbor-test-implement now renders into .build like the
rest; the spec, plan, repo and implement images gain python3.
Self-checks under tests/harbor/shared/tests cover the readers, step
windows, the recorder's every failure path and the suite wiring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000070_harbor-run-metrics Milestone 1: every harbor run leaves a metrics record
Every harbor suite's verifier now writes metrics.json beside reward.txt,
in one shape for Claude, Codex, Bob and oracle runs:
- run identity from a Makefile-written build.json (agent, suite,
scenario, git-described version, commit, dirty)
- totals from the agent's own log: Claude's result events (single vs
several-result rule), harbor's ATIF trajectory for Bob and Codex, or
Codex's rollout as a fallback
- per-step durations from Spektacular's session log (spec, plan and
implement lanes) and characters written per step, joined by time, by
goto order for Bob, or by launch prompt across orchestrated lanes
- sub-agent counts and durations
- the check score overall and per test class or module, from pytest's
JUnit XML
Anything a log does not supply is null with a reason under `unknown`.
The recorder (tests/harbor/shared/metrics.py) runs after the reward is
written, with its failure ignored, so a run's pass-or-fail result is
unchanged. All eight render targets copy the shared helpers and write
build.json; harbor-test-implement now renders into .build like the
rest; the spec, plan, repo and implement images gain python3.
Self-checks under tests/harbor/shared/tests cover the readers, step
windows, the recorder's every failure path and the suite wiring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000070_harbor-run-metrics Milestone 2: one command shows where runs spend their time
Adds tests/harbor/shared/report.py and `make harbor-report`, a local,
stdlib-only report over collected harbor jobs. It reads only each
trial's verifier/metrics.json and result.json, so it needs no sudo for
the root-owned files runs leave under agent/.
- The default view prints one row per recorded run, grouped by suite
and scenario: wall time, turns, output and cache-read tokens, cost,
check score and pass/fail. Trials without a record are skipped.
- `--run <trial dir, name or prefix>` prints that run's step breakdown
(duration, share of the run, characters written) and flags any step
over a third of the run's wall time, then the sub-agents and the
unknown measures with their reasons.
Unknown values print as unknown, never zero. Self-checks cover the
table, the breakdown, the dominant-step threshold and unreadable agent
files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000070_harbor-run-metrics Milestone 2: one command shows where runs spend their time
Adds tests/harbor/shared/report.py and `make harbor-report`, a local,
stdlib-only report over collected harbor jobs. It reads only each
trial's verifier/metrics.json and result.json, so it needs no sudo for
the root-owned files runs leave under agent/.
- The default view prints one row per recorded run, grouped by suite
and scenario: wall time, turns, output and cache-read tokens, cost,
check score and pass/fail. Trials without a record are skipped.
- `--run <trial dir, name or prefix>` prints that run's step breakdown
(duration, share of the run, characters written) and flags any step
over a third of the run's wall time, then the sub-agents and the
unknown measures with their reasons.
Unknown values print as unknown, never zero. Self-checks cover the
table, the breakdown, the dominant-step threshold and unreadable agent
files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000070_harbor-run-metrics Milestone 3: compare runs before and after a change, and document the metrics
Adds the report's comparison view: `--baseline` and `--candidate` each
take job directories or globs, and for every suite (and scenario) in
both sets the report prints the median wall time, turns, output and
total tokens, cost and check score before and after, with the change.
A measure is flagged when the candidate median is at least twice the
baseline's, or when the check score falls by 0.1 or more; medians use
known values only, and an unknown measure is never flagged. Suites in
only one set are listed as unmatched.
Documents the run metrics record and the report in the README's harbor
section: the new verifier files in the job layout, every field of the
record, which fields Bob, Codex and oracle runs leave unknown, how
step windows and Claude's totals are derived, and the report's table,
single-run view, comparison and self-checks. A doc-sync self-check
keeps the field table in step with the record.
Also: `--run <trial dir>` no longer reads the default jobs directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000070_harbor-run-metrics Milestone 3: compare runs before and after a change, and document the metrics
Adds the report's comparison view: `--baseline` and `--candidate` each
take job directories or globs, and for every suite (and scenario) in
both sets the report prints the median wall time, turns, output and
total tokens, cost and check score before and after, with the change.
A measure is flagged when the candidate median is at least twice the
baseline's, or when the check score falls by 0.1 or more; medians use
known values only, and an unknown measure is never flagged. Suites in
only one set are listed as unmatched.
Documents the run metrics record and the report in the README's harbor
section: the new verifier files in the job layout, every field of the
record, which fields Bob, Codex and oracle runs leave unknown, how
step windows and Claude's totals are derived, and the report's table,
single-run view, comparison and self-checks. A doc-sync self-check
keeps the field table in step with the record.
Also: `--run <trial dir>` no longer reads the default jobs directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000070_harbor-run-metrics: test plan, changelog and spec reconciliation
Closes the implement run for harbor run metrics. Adds the manual test
plan: eight procedures covering one real run of every suite and agent
target, Claude totals and sub-agents against claude-code.txt, Bob and
Codex unknowns (settling the open question about Codex's trajectory at
verify time), the report's single-run and comparison views on real
runs, complete Claude records, the report as the normal user, and a
README review. Writes the project and spektacular changelog records,
and reconciles the spec: every requirement and all but two acceptance
criteria are met. Still open, pending the manual harbor runs in the
test plan: a record from one run of every suite with every agent, and
Codex unknowns on a real Codex run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Update knowledge
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Save uncommitted changes before spec workflow for 000071_harbor-child-agents
These are the user's changes from before the spec workflow for 000071_harbor-child-agents started. Spektacular committed them separately at the user's request so they are not mixed with the agent's work.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000071_harbor-child-agents: child agents in harbor runs
Harbor suites will exercise child agents the way a real session does and
record every child agent, with the context it was given, in each run's
metrics. Required delegation and child context are checked, epic children
run in the background once the headless ten-minute limit is verified, timed
wake-ups while waiting on children fail epic runs, idle time is measured,
and two verifier checks stop misreading how a command was written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000071_harbor-child-agents: child agents in harbor runs
Plan the work to make harbor end-to-end runs exercise and record child
agents as real sessions do:
- Record every child agent per run (who started it, full context, background
or foreground, timing, ending, hand-back, models, output), placed by spec
and step, plus per-step child counts and run idle time (metrics schema 2),
shown in the report with a --prompts option.
- Judge the plan walkthrough and per-repo changelog checks from the session
log instead of command text; make plan discovery delegation record-only.
- Enforce required delegation and child context for Claude runs (spec review,
implement test/verify/check_goal, one child per epic spec, no wake-ups),
strengthening the step and epic skill instructions to match.
- Settle the ten-minute headless background-child claim with a probe run
before rewriting the epic skills' headless guidance.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000071_harbor-child-agents Milestone 1: record child agents and idle time
Every harbor run's metrics record (now schema version 2) lists each child
agent the run started, nested ones included. Each entry carries its parent,
kind, description, full launch prompt and its size, and its real background
state. It also has start and end times, the agent's own reported duration,
models, output, ending and first hand-back line, and the spec and workflow
step it was started in. Every step entry counts the children started in it,
and the run records its idle time, the longest stretch with no activity.
- transcripts.py: Child, Turn and Wakeup readers for Claude stream-json, and
ATIF children for Bob (kind, description, prompt) and Codex (unknown).
- metrics.py: child placement by spec and step, per-step counts, idle time,
each in its own never-fail section.
- report.py: idle column and mark, a children table in the run view, and
--run <trial> --prompts for every child's full context.
- Trimmed 2026-10-23 reference runs (m70/s55 epic planning, s55 plan and
implement) checked in under tests/harbor/shared/tests/data/, produced by
trim_reference.py. The script refuses to write a trim that changes any
recorded value.
- README run-metrics reference updated for schema version 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000071_harbor-child-agents Milestone 1: record child agents and idle time
Every harbor run's metrics record (now schema version 2) lists each child
agent the run started, nested ones included. Each entry carries its parent,
kind, description, full launch prompt and its size, and its real background
state. It also has start and end times, the agent's own reported duration,
models, output, ending and first hand-back line, and the spec and workflow
step it was started in. Every step entry counts the children started in it,
and the run records its idle time, the longest stretch with no activity.
- transcripts.py: Child, Turn and Wakeup readers for Claude stream-json, and
ATIF children for Bob (kind, description, prompt) and Codex (unknown).
- metrics.py: child placement by spec and step, per-step counts, idle time,
each in its own never-fail section.
- report.py: idle column and mark, a children table in the run view, and
--run <trial> --prompts for every child's full context.
- Trimmed 2026-10-23 reference runs (m70/s55 epic planning, s55 plan and
implement) checked in under tests/harbor/shared/tests/data/, produced by
trim_reference.py. The script refuses to write a trim that changes any
recorded value.
- README run-metrics reference updated for schema version 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000071_harbor-child-agents Milestone 2: judge what ran from the session log
The plan suite's walkthrough-before-finished check and the implement suite's
project and per-repo changelog checks now read Spektacular's own session
log, so a correct run no longer fails because its commands were piped,
chained, looped, built from shell variables or continued over lines.
- sessionlog.py: plan_steps_entered (the step each successful plan new/goto
left the workflow at) and changelog_writes_by_repo (successful changelog
writes per --repo; help requests and failed writes excluded).
- plan verifier: walkthrough check judged from the session log, and the
discovery spawn check dropped, because discovery only suggests a child and
the metrics record now counts it. The stray temp.py is deleted.
- implement verifier: the three changelog checks judged from the session
log instead of command substrings.
- plan and implement tasks gain the "running non-interactively" section
every other suite carries, and the suite-wiring self-check pins it in all
of them. The plan task's status wording now matches its verifier.
- The trim script's self-verification covers both new rules, proven on the
s55 plan and implement reference runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000071_harbor-child-agents Milestone 2: judge what ran from the session log
The plan suite's walkthrough-before-finished check and the implement suite's
project and per-repo changelog checks now read Spektacular's own session
log, so a correct run no longer fails because its commands were piped,
chained, looped, built from shell variables or continued over lines.
- sessionlog.py: plan_steps_entered (the step each successful plan new/goto
left the workflow at) and changelog_writes_by_repo (successful changelog
writes per --repo; help requests and failed writes excluded).
- plan verifier: walkthrough check judged from the session log, and the
discovery spawn check dropped, because discovery only suggests a child and
the metrics record now counts it. The stray temp.py is deleted.
- implement verifier: the three changelog checks judged from the session
log instead of command substrings.
- plan and implement tasks gain the "running non-interactively" section
every other suite carries, and the suite-wiring self-check pins it in all
of them. The plan task's status wording now matches its verifier.
- The trim script's self-verification covers both new rules, proven on the
s55 plan and implement reference runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000071_harbor-child-agents Milestone 3: enforce required delegation
Claude test runs now fail when a step whose instructions require a child
agent completes without one, when a required child is not given the context
its instructions say to pass, or when an epic orchestrator sets a timed
wake-up while children run. Codex and Bob runs are recorded but never failed
by these rules.
- Step templates: spec verification and implement test/verify state that an
agent that can start sub-agents must delegate, and that doing the step in
its own context "does not complete this step". Pinned by Go tests.
- Epic skills: never set a wake-up, timer, recurring job or sleeping command
while children run; spek-implement-epic names each spec's worktree paths
in its child's prompt. Pinned by Go tests.
- tests/harbor/shared/delegation.py: the hand-kept oracle (required,
suggested, per-spec children) and pure rules returning named failures
(missing required child, missing per-spec child, epic and review context
gaps, wake-ups while children run), tied to the templates' wording by a
self-check.
- spec and goal-check verifiers require the review child and the goal-check,
test and verify children; the epic and implement-epic verifiers gain a
Claude-only TestChildren class.
- Knowledge: testing-architecture updated for the Claude-only child checks
and the delegation oracle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000071_harbor-child-agents Milestone 3: enforce required delegation
Claude test runs now fail when a step whose instructions require a child
agent completes without one, when a required child is not given the context
its instructions say to pass, or when an epic orchestrator sets a timed
wake-up while children run. Codex and Bob runs are recorded but never failed
by these rules.
- Step templates: spec verification and implement test/verify state that an
agent that can start sub-agents must delegate, and that doing the step in
its own context "does not complete this step". Pinned by Go tests.
- Epic skills: never set a wake-up, timer, recurring job or sleeping command
while children run; spek-implement-epic names each spec's worktree paths
in its child's prompt. Pinned by Go tests.
- tests/harbor/shared/delegation.py: the hand-kept oracle (required,
suggested, per-spec children) and pure rules returning named failures
(missing required child, missing per-spec child, epic and review context
gaps, wake-ups while children run), tied to the templates' wording by a
self-check.
- spec and goal-check verifiers require the review child and the goal-check,
test and verify children; the epic and implement-epic verifiers gain a
Claude-only TestChildren class.
- Knowledge: testing-architecture updated for the Claude-only child checks
and the delegation oracle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000071_harbor-child-agents Milestone 4: settle the headless background-child limit
A recorded harbor probe settled the claim that headless Claude Code stops
long background children. With Claude Code 2.1.310 under `claude --print`, a
background child was killed about 600 s after the main agent ended its turn,
and no completion notice woke the session. The epic skills and suites now
follow that outcome.
- tests/harbor/background-child: the probe task and its outcome-neutral
verifier, with `make harbor-test-background-child`. The recorded run is
trimmed and checked in under
tests/harbor/shared/tests/data/probe-background-child/, with an OUTCOME
line.
- Epic skills keep the foreground workaround and state the recorded outcome.
Being told no one answers is no longer by itself the stated reason.
Pinned by Go tests.
- delegation.py: PROBE_OUTCOME = "stopped" selects the foreground rule. Each
spec's child runs in the foreground and no turn ends while one runs; the
background rule is kept for a future outcome. Both epic verifiers'
TestChildren check it.
- transcripts.py: turns are delimited by mid-stream system/init events,
because a result can be written late. A turn ends at the main agent's own
last message. A stopped child has no hand-back.
- README documents the probe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Implement 000071_harbor-child-agents Milestone 4: settle the headless background-child limit
A recorded harbor probe settled the claim that headless Claude Code stops
long background children. With Claude Code 2.1.310 under `claude --print`, a
background child was killed about 600 s after the main agent ended its turn,
and no completion notice woke the session. The epic skills and suites now
follow that outcome.
- tests/harbor/background-child: the probe task and its outcome-neutral
verifier, with `make harbor-test-background-child`. The recorded run is
trimmed and checked in under
tests/harbor/shared/tests/data/probe-background-child/, with an OUTCOME
line.
- Epic skills keep the foreground workaround and state the recorded outcome.
Being told no one answers is no longer by itself the stated reason.
Pinned by Go tests.
- delegation.py: PROBE_OUTCOME = "stopped" selects the foreground rule. Each
spec's child runs in the foreground and no turn ends while one runs; the
background rule is kept for a future outcome. Both epic verifiers'
TestChildren check it.
- transcripts.py: turns are delimited by mid-stream system/init events,
because a result can be written late. A turn ends at the main agent's own
last message. A stopped child has no hand-back.
- README documents the probe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Finish 000071_harbor-child-agents: test plan, changelog and spec reconciliation
Wraps up the implementation of child-agent recording and delegation checks
in the harbor suites:
- The test plan sets out the manual checks only a real Claude batch can
settle: every child recorded across all nine Claude suites, no required
delegation skipped and no idle flag, epic-planning idle under a quarter,
no command-text misreads, the report answering "which children, told
what". It also lists the reviews of the probe outcome, the epic runs'
foreground children, Codex and Bob unknowns, and the README.
- The project-level and spektacular repo changelog records describe what
was built and its deviations from the plan.
- The spec's requirements and acceptance criteria are reconciled. Two
criteria stay open until a real post-change batch runs: epic children in
the background (conditional, and the probe found background children
stopped), and no passing run ending on a question.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Save uncommitted changes before spec workflow for 000072_knowledge-access-index-and-prune
These are the user's changes from before the spec workflow for 000072_knowledge-access-index-and-prune started. Spektacular committed them separately at the user's request so they are not mixed with the agent's work.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000072_knowledge-access-index-and-prune: self-written knowledge with usage index and prune
Specifies that coding agents write durable knowledge to the project's
knowledge base without asking first, and tell the user what they wrote. To
keep that safe:
- A JSON usage index inside the knowledge base, changed only by
Spektacular's knowledge commands, records each entry's discoveries (search
hits) and retrievals (reads and always-applied loads, with the active
workflow and spec), and stays exact under parallel runs.
- Every entry gets a creation date, backfilled for existing entries.
- `knowledge prune` acts without asking and reports every entry's verdict
with evidence; `--dry-run` previews it without changing anything.
- Pass 1 removes entries older than a configurable age that were never
retrieved. Entries are not pruned for non-use until tracked for the full
threshold, and prune's own reads do not count.
- Pass 2 has the agent check each remaining entry: wrong ones are
corrected, ones whose subject is gone are deleted, and unmet targets are
kept under the binding knowledge rule.
- The website docs are updated. No split offered: the parts form one
capability, self-written knowledge made safe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Add performance improvements
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Save uncommitted changes before spec workflow for 000073_log-mode-and-logger-providers
These are the user's changes from before the spec workflow for 000073_log-mode-and-logger-providers started. Spektacular committed them separately at the user's request so they are not mixed with the agent's work.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000073_log-mode-and-logger-providers: always-on trace logging of every CLI call
First spec of epic 000002_workflow-tracing-and-run-stats. Debug mode becomes
a log mode that is on by default:
- Every command is recorded once as a span in the OpenTelemetry trace data
model, carrying the configured agent name, workflow kind, name and step.
The run's trace identifier comes from the workflow state, kept across
resumes, new on a restart, separate for concurrent orchestrated runs, and
given to runs already in progress at upgrade.
- Levels: the default records timing, exit status and errors only; verbose
keeps today's full responses and state snapshots; logging can be switched
off, and one command can be left out or made verbose from the environment.
- The logger is a provider setting (file logger, JSON Lines, first), with a
configurable directory defaulting to the logs folder, ignored by git through
init and migrate.
- Retention keeps the last 10 workflow runs by default (configurable);
standalone commands are grouped per day.
- migrate converts debug settings; old debug files are left to the user.
- Harbor keeps its session reporting on the new logs until spec 4 replaces
it; the docs debugging page and configuration reference describe logging.
Drafted without per-section confirmation at the user's request; fresh-eyes
review findings applied. No split offered: the requirements form one
capability within the epic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000074_trace-spans-for-agent-work: step, task, helper and user-wait spans
Second spec of epic 000002_workflow-tracing-and-run-stats, building on
000073's per-command spans:
- Workflow step changes open and close step spans themselves; commands nest
under their step. Implement records each plan task as a span with its
steps nested under it. Agents do nothing extra for either.
- Agents mark only what the CLI cannot see: helper-agent launches
(sub-agent spans) and questions to the user (user-wait spans, their own
type so timings can exclude them), through new trace start and stop
commands with a default or named parent, so concurrent helpers nest
correctly.
- Interrupted steps are closed on resume, restarted ones are abandoned, and
spans left open close with their parent; trace commands never change
workflow state and follow log mode.
- Every workflow's instructions tell agents what to mark; the docs logging
page describes span types and commands.
Drafted without per-section confirmation at the user's request; fresh-eyes
review findings applied. Depends on 000073. No split offered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000075_run-stats-command: spektacular stats over the run logs
Third spec of epic 000002_workflow-tracing-and-run-stats. A report command
reads the spans 000073 and 000074 record:
- List held runs; select by name, run identifier, kind or agent, combined.
- One run: wall and working time (user waits excluded), user-wait and
helper-wait totals, per-step and per-task breakdowns, each helper launch,
and the longest gaps with a settable count and threshold. Interrupted and
unfinished work is flagged and kept out of the totals it would distort.
- Compare runs side by side (difference from the first, time and percent)
and summarise many (median and 90th percentile).
- JSON in the shared envelope with stable field names for Harbor, plus a
table; works on a collected logs directory; read-only; damaged lines are
skipped and counted. Docs describe the report.
Drafted without per-section confirmation at the user's request; fresh-eyes
review findings applied. Depends on 000074. No split offered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000076_harbor-on-built-in-logs: Harbor measures from Spektacular's logs
Fourth spec of epic 000002_workflow-tracing-and-run-stats. Harbor stops
parsing each agent's transcript and the old debug log for timing:
- Step timings, helper agents and where they started (including Codex's,
invisible in its transcript), idle, gap and user-wait time, epic run
timelines and delegation placement come from Spektacular's logs through
the run statistics report, at the default log level.
- Metrics records report total user wait, total gaps and a per-step split
instead of the single longest gap; the cross-run report compares them.
- Tokens, models and cost still come from transcripts, unchanged.
- The debug-log reader and transcript-based timing are removed; trials
collect the logs directory; missing data stays unknown, and measuring
never fails a run.
Drafted without per-section confirmation at the user's request; fresh-eyes
review findings applied. Depends on 000075 (and 000074). No split offered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000077_html-log-viewer: a run as a self-contained HTML waterfall
Fifth spec of epic 000002_workflow-tracing-and-run-stats. Renders one
recorded run, chosen the same ways as the run statistics report, as a
single offline HTML page:
- A waterfall of every span on a shared timeline, nested by parent, with
each span type distinct and user waits set apart; failed, interrupted and
abandoned work stands out.
- Details on selection, run totals matching the stats report, gaps above a
settable threshold (5 minutes by default) marked, zoom and collapse.
- Stays responsive at several thousand spans, renders collected logs, and
the docs describe it.
Drafted without per-section confirmation at the user's request; fresh-eyes
review findings applied. Depends on 000075. No split offered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000073_log-mode-and-logger-providers: always-on trace logging replaces debug mode
Plans the replacement of debug mode with an always-on log mode. Every
command writes one OpenTelemetry-model span record (trace and span ids,
timing, status, agent, workflow kind and name, step and step moved to)
through a provider-chosen logger, `file`, to a git-ignored
.spektacular/logs/ directory. Each workflow run gets its own file, with
one file per UTC day for standalone commands, and count-based retention
keeps 10 runs by default.
The run id lives in workflow state (`run_id`), is minted on fresh state,
and is backfilled for runs that were in progress before the upgrade.
Config `log: {provider, level: off|standard|verbose, retention,
config.directory}` replaces `debug`. Settings format 5 adds a migration
step that maps debug on to verbose and off to standard, and adds
`logs/` to .spektacular/.gitignore. SPEKTACULAR_SESSION_LOG keeps its
name and is remapped onto the levels.
Harbor gets a dual-format reader and verbose images. The docs site's
debugging page, the configuration reference, the README and the Harbor
knowledge entries are updated.
The plan has 4 milestones and 9 agent tasks across the spektacular and
docs repos.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000072_knowledge-access-index-and-prune: usage index and knowledge prune
Plans self-written knowledge for the workflow-tracing epic. Each knowledge
store gains a JSON usage index, written through a new atomic store update
guarded by a portable lock file. Searches record discoveries. Reads and
always-applied loads record retrievals, attributed to the workflow and spec
they ran under. Creation dates live in the index and are backfilled from git.
A new `knowledge prune` command removes old, unread entries, then walks the
agent through every remaining entry with verdicts checked against the store.
It has a dry-run preview and a `knowledge.prune.max_age_days` setting
(default 90). Installed instructions switch from "offer and wait" to "write,
then report", and the README, maintainer docs and website are updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000074_trace-spans-for-agent-work: step, task and agent spans in run logs
Plans the trace spans built on 000073's log records. The workflow engine
marks every step and every implement task as a span itself, keeping the open
step and task spans in state beside run_id and reporting transitions to a sink
that writes span_start/span_end records. Commands nest under their step.
`spektacular trace start|stop` lets agents mark helper agents and user waits,
using the run's log as the registry of open spans and a <trace>-<span> id that
routes to orchestrated lanes. One shared instruction block, appended to every
step and included in every workflow skill, tells agents what to mark. Harbor
learns to skip and check span records, and the logging docs gain a spans
section.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000075_run-stats-command: spektacular stats over the run logs
Plans `spektacular stats list|run|compare|summary`, a report read from the
logs 000073 and 000074 write. A new pure package, internal/runstats, loads a
logs directory read-only (damaged lines skipped and counted), groups records
into runs by trace id, pairs span starts and ends, and derives each run's
wall and working time, helper and user waits (interval unions, unfinished
spans flagged and left out), per-step and per-task figures, helper launches,
and the longest gaps labelled by step. Comparisons and summaries key steps by
name with per-run deltas or count, median and p90. Output is a readable table
or the standard JSON envelope with field names fixed for Harbor (000076) and
the HTML viewer (000077). `--dir` reports on copied logs, and the stats group
is exempt from logging so reporting never changes a log. A docs page and the
README describe the commands and figures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000077_html-log-viewer: a run as a self-contained HTML waterfall
Plans `spektacular stats view`, a fifth stats form that writes one recorded
run as a single HTML file. A new pure package, internal/runview, builds a
compact columnar view model from 000075's runstats (tree-ordered rows,
offsets, problem flags, details, every gap down to 1s, and the report's
totals). It splices the model as HTML-safe JSON into an embedded shell
whose inline script draws the visible rows on a canvas, with zoom,
collapse, a gap threshold control and a longest steps and gaps summary.
The command reuses the stats group's selection, --dir and logging
exemption, adds --out, and refuses to write into the logs directory. Docs
gain a Run viewer page and the README a stats view line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000076_harbor-on-built-in-logs: Harbor measures runs through spektacular stats
Plans moving the Harbor harness onto Spektacular's own logs. `stats` gains
`commands` and `spans` listings (including records written outside any
run, such as an epic orchestrator's), and a new shared `runlog.py` becomes
Harbor's only route to the logs. The metrics record (schema 3), epic
timeline, delegation checks and every verifier rule read through it at the
default log level, with no session-log override. Records gain total user
wait, helper wait and gap time and a per-step working/helper/user split;
tokens, models and cost stay on transcripts. The old debug-log reader and
transcript step, placement and idle code are removed, with the shared
scripts held to 2,000 lines. 14 tasks over three milestones, including a
person-run baseline of Harbor pass rates before the change.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Save uncommitted changes before spec workflow for 000078_token-use-in-run-stats
These are the user's changes from before the spec workflow for 000078_token-use-in-run-stats started. Spektacular committed them separately at the user's request so they are not mixed with the agent's work.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Specify 000078_token-use-in-run-stats: tokens and cost beside run timings
Seventh spec of epic 000002_workflow-tracing-and-run-stats, added at the
end-of-planning review so time can be judged against work done:
- Run reports give input, output and cache tokens and cost for the run,
each step, each task and each helper agent, with the models used.
- Main-agent requests are attributed by time to the step and task running;
helper tokens go to their launch and its step, even if it outlives it.
- A time-per-token figure (working time per thousand output tokens) sets
each part's duration against the work it did; comparisons and summaries
include tokens, cost and that figure.
- Token use is read from the agent's session data with no extra agent step
(Claude Code first; other agents unknown with a reason). Prices come
from an optional project price table; partial totals are marked
incomplete; missing data never fails a report.
Drafted without per-section confirmation at the user's request; fresh-eyes
review findings applied. Depends on 000075. No split offered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* Plan 000078_token-use-in-run-stats: tokens and cost beside run timings
Plans token use in `spektacular stats`. A new read-only leaf package,
internal/agentusage, reads Claude Code's own session transcripts (main and
helper files, plus Claude Code's session totals). Pure attribution in
internal/runstats finds each run's driving transcript from its own
`new`/`goto` commands, with no hook and no agent action. It places every
request in the step and task it was made in, or outside any step, and
puts helper tokens on their launch and launching step. Requests Claude
Code counted but never recorded are added as session overhead. Tokens are
priced from an optional `stats.prices` table (no format bump, no built-in
prices) and set against working time per thousand output tokens. The run
report, compare and summary only gain fields, checked against 000075's
golden files. Other agents and missing sessions report unknown with a
reason. 8 agent tasks over three milestones, including docs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000072_knowledge-access-index-and-prune: Milestone 1, the knowledge base records how its entries are used
Implemented Milestone 1 of 000072_knowledge-access-index-and-prune:
- A shared portable file lock (internal/filelock), which the commit lock now
delegates to, and an atomic read-modify-write Update on the store contract.
- A per-store usage index (.usage-index.json) in internal/knowledge recording
creation dates, tracking start, discoveries and retrievals, reconciled with
the store on every write and hidden from listings, search and entry commands.
- knowledge search/read/always-applied/conventions record usage (with workflow
and spec attribution), write and delete keep records in step, and the new
read-only `knowledge usage` command reports the index.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000072_knowledge-access-index-and-prune: Milestone 1, the knowledge base records how its entries are used
Implemented Milestone 1 of 000072_knowledge-access-index-and-prune:
- A shared portable file lock (internal/filelock), which the commit lock now
delegates to, and an atomic read-modify-write Update on the store contract.
- A per-store usage index (.usage-index.json) in internal/knowledge recording
creation dates, tracking start, discoveries and retrievals, reconciled with
the store on every write and hidden from listings, search and entry commands.
- knowledge search/read/always-applied/conventions record usage (with workflow
and spec attribution), write and delete keep records in step, and the new
read-only `knowledge usage` command reports the index.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000072_knowledge-access-index-and-prune: Milestone 2, a prune command removes unused knowledge and checks the rest
Implemented Milestone 2 of 000072_knowledge-access-index-and-prune:
- An optional knowledge.prune.max_age_days project setting (default 90,
negative refused), with no settings-format bump or migration.
- `knowledge prune` removes entries created and tracked for at least the age
and not retrieved within it, then hands the agent one instruction per
remaining entry (body inlined, verdict rules: an unmet standard is current,
only a vanished subject is deleted, evidence for every change).
- `knowledge prune verdict` checks each claim against the store and returns the
next entry or a final report of every entry; --dry-run changes nothing,
an unfinished run resumes, and --force restarts.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000072_knowledge-access-index-and-prune: Milestone 2, a prune command removes unused knowledge and checks the rest
Implemented Milestone 2 of 000072_knowledge-access-index-and-prune:
- An optional knowledge.prune.max_age_days project setting (default 90,
negative refused), with no settings-format bump or migration.
- `knowledge prune` removes entries created and tracked for at least the age
and not retrieved within it, then hands the agent one instruction per
remaining entry (body inlined, verdict rules: an unmet standard is current,
only a vanished subject is deleted, evidence for every change).
- `knowledge prune verdict` checks each claim against the store and returns the
next entry or a final report of every entry; --dry-run changes nothing,
an unfinished run resumes, and --force restarts.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000072_knowledge-access-index-and-prune: Milestone 3, agents write knowledge themselves and say what they wrote
Implemented Milestone 3 of 000072_knowledge-access-index-and-prune:
- The knowledge-trigger standing rule, memory-context, the plan discovery and
walkthrough steps, the implement update_changelog and finished steps, the
orchestrated-stop partial and both epic skills now write durable knowledge
through spek-knowledge and report the entry (tier, store, path, why) instead
of offering and waiting; the plan discovery step names its spec on reads.
- The spek-knowledge skill's contribute and update intents write then report,
and its maintenance review is replaced by a prune intent driving
`knowledge prune` (with --dry-run as the preview); the tag audit is unchanged.
- Template-contract tests pin the new wording and ban the old offer phrases,
and the Harbor suites drop their obsolete "decline the offer" line.
Signed-off-by: Nic Jackson <jackson.nic@gmail.com>
* 000072_knowledge-access-index-and-prune: Milestone 3, agents write knowledge themselves and say what they wrote
Implemented Milestone 3 of 000072_knowledge-access-index-and-prune:
- The knowledge-trigger standing rule, memory-context, the plan discovery and
walkthrough steps, the implement update_changelog and finished steps, the
orchestrated-stop partial and both epic skills now write durable knowledge
through spek-knowledge and report the entry (tier, store, path, why) instead
of offering and waiting; the plan discovery step names its spec on reads.
- The spek-knowledge skill's contribute and update intents write then report,
and its maintenance review is replaced by a prune intent driving
`knowledge prune` (with --dry-run as the preview); the tag audit is unchanged.
- Template-contract tests pin…
