Skip to content

v0.14.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 07 Aug 23:00
· 19 commits to main since this release
v0.14.0
fdd7d9b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Ask the pipeline how it's doing 🐝📊

Apiary has always recorded how its agents perform — timings, tokens, cost,
turns, failure kinds, a transcript of every session — and then only ever used
that to answer what happened to this task. This release adds the cross-cutting
question: across the last N runs, which steps waste money, which agent
instructions keep producing rework, and what should change.

apiary improve mines the history, has an agent reason over it, and emits an
evidence-backed report plus a validated diff over your config, workflows, souls
and skills — or applies it. Alongside it, apiary profile answers where a
step's minutes actually went, and three monitoring-source features that had been
sitting unreleased finally ship.

apiary improve

apiary improve                    # analyse; print findings and a diff
apiary improve --effort deep --since 30d
apiary improve --apply            # write the accepted changes
apiary improve --dump-evidence    # just the metrics, as JSON, no model

The evidence pack is computed entirely in Go — no model is involved in
producing any number, so the same database and window always give the same
result, and --dump-evidence is a free diagnostic. It covers per-step rates,
percentiles, cost, cache reuse, prompt weight and failover; per-workflow terminal
states, cost per completed instance and rework loops (the same step running
twice in one instance — the signature of an on_fail/goto cycle, priced);
max_turns saturation; wait polling; clustered failures; and dead config that
never ran.

Every proposal passes a five-stage gate before you see it: path → applies
cleanly → the patched config parses and validates → expressions still lint → new
warnings surfaced. Patch application does no fuzzy matching: a hunk whose
context does not match is rejected, because a patch landing in the wrong place is
worse than one that does not land. Rejected proposals are shown with the reason,
since the observation behind them is often still worth acting on by hand.

Instruction files are labelled honestly. Souls and skills clear "the patch
applies" and nothing else — nothing in a markdown file can be validated
mechanically — and the diff says so per file. That is the common case, not the
edge case, which is why apiary improve effect exists: it recomputes the metrics
that justified an applied change and reports the delta. For a prose edit, those
numbers are the first evidence either way.

Applying leaves undo to git. No backups, no snapshots, no revert command —
git diff and git checkout do that better. What apply owes you is an accurate
account of what it touched, including which files nothing could check and whether
a daemon restart is needed.

Full documentation: Self-Improvement.

Where the minutes went

  • Per-step wall-clock attribution (#399) — thinking, writing, tool waits and
    background tasks recorded alongside the token columns, plus the slowest
    individual calls. New apiary profile <instance-id> [--json].
  • A step that is 80% tool waits has a different problem from one that is 80%
    thinking, and until now nothing distinguished them.

Monitoring sources

  • ack_via_silence (#362) — silence a Prometheus alert while an agent
    investigates, so a firing alert does not page twice.
  • interrupt_on_resolve (#362) — stop a still-running instance when the
    alert that triggered it stops firing. Off by default: an investigation's
    findings usually outlive the alert.
  • dispatch_by: group (#362) — one task per Alertmanager group instead of
    per alert.

Timeouts

settings.task_timeout was documented as defaulting to 30m in two places while
the code has always used 2h (#411). The docs are now correct. The default did
not change: implementation steps legitimately run for an hour, so a shorter
default would kill working pipelines rather than runaway ones.

That makes a total timeout a poor hang detector, so there is now a second bound:

settings:
  task_timeout: 2h      # total — the backstop
  stall_timeout: 20m    # no output at all for this long — the hang detector

stall_timeout measures silence rather than duration. A step streaming tool
calls for ninety minutes survives; one silent for twenty does not. It is
off by default — it only makes sense for runners that stream as they work,
and a runner buffering until exit would look permanently stalled.

Both bounds are now logged at step start, and a bound that fires is recorded as
a timeout naming the bound and the elapsed time, instead of a bare
signal: killed that was indistinguishable from a crash.

Fixes

  • Transcript sampling read the wrong file — it picked each task's most
    recently modified transcript rather than the one for the step being analysed,
    so every hotspot got the final step's session. Excerpt budgets raised
    accordingly (24 KB standard, 40 KB deep).
  • The thinking/writing split double-counted backgrounded calls (#399).
  • Daemon test teardown is deterministic — dispatch goroutines are tracked, so
    a prior test's workers can no longer race the next one.
  • The shipped example config validates again — spec-decompose used
    materialize: sub_issue on a source that cannot host sub-issues.

Upgrading

No breaking changes. Two new optional settings — settings.improve and
settings.stall_timeout — both absent by default.

To use apiary improve you need an agent for it to run as: define one with id
improver, set settings.improve.agent, or pass --advisor / --runner +
--model on the command line. It errors rather than guessing, because model is
required per agent and there is no global default. --dump-evidence needs none
of that.

A note on maturity: the deterministic half of improve — the evidence pack, the
validation gate, the ledger — is well covered by tests. The advisor's judgement
has had little real-world exposure so far. Diff is the default and applying is
opt-in for that reason; read what it proposes before you take it.

Changelog

Contributors

orlandoburli
Assets 13
Loading