Skip to content

v1.1.0

Choose a tag to compare

@Cloflin Cloflin released this 02 Sep 07:02
· 273 commits to main since this release

Changed

The window grid is 408, and the eval figures are gated. docs/guide/evaluation.md
said the window search had 272 candidates; spec 006 widened it to 408 two releases
ago, and nothing caught it — the existing figures gate covers only the index
numbers, and its repair script cannot reach a constant inside calibrate.js. Both
mentions are corrected against WINDOWS.length, the sample output block is a real
transcript of this repository's own run, and three new checks derive the printed
window grid, the printed tune grid (11 λ × 9 k = 99) and the probe count from the
code and the artefacts that produce them.

Both shipped indexes carry a measured guard again. They were on corpus
e9985350 against a calibration measured on ab42d56c, so both shipped
guard.source: "provisional" — a tau of 0.3 that nothing measured on this corpus.
The loop was re-run: docs/public/rag-local is calibrated-reduced and
docs/public/rag is transferred-window, both on corpus 08e7a87e, tau 0.69.

One embedding purchase per run. eval, tune and bench emit bought one
vector per request — 58 requests for this repository's golden set, against a
free tier that allows fifty a day. They now use the same batcher calibrate has
used since 1.0.0, which is 2. The batcher moved out of calibrate.ts into
src/eval/prefetch.ts with its behaviour unchanged to the letter — same batch of
32, same retryable statuses, same refusal to guess at a short batch, same
retry-after with a twenty-second ceiling, same silent fallback to the per-text
path — and gained the direct tests none of that had. It is a cache and never a
second code path: a provider that will not batch degrades to the loop that
already worked.

The eval report names its witnesses, and the seed is finally sent. meta
carried eighteen fields and every one described the input — the corpus hash,
the prompt hash, the levers, the flags. A report taken against a metered
provider and one taken against a laptop's Ollama were byte-identical, and a
rerun a week later was indistinguishable from the one it overwrote. It now also
carries ranAt, chatBase, embedBase (origins only — never a path, never a
key), node, package, goldenSha, temperature and seed.
calibration.json gains ranAt, embedBase, probeSha and embedRequests —
the last was already printed during the run and then thrown away, and it is the
one number that says what a rerun costs. tuning.report.md prints sweptAt.

goldenSha is a new incomparability marker. records is a count, so a
question rewritten inside an existing record moved nothing and two different
sets of 56 were identical to every check. Absent in an older report reads as
unknown, not as changed — the rule meta.level already established — so
upgrading does not announce a golden-set change to everybody once.

The seed reaches the call that writes the answer. Every answer metric was
one unseeded sample: the transport has accepted a seed all along, and
config.llm simply carried no tuning, so the guard never fired. It does now,
at 20260829 — the constant calibrate already draws its anchors with. A
related trap is fixed with it: the harness passed the whole tuning record
through its can-this-model-think filter, so on a model without thinking the
seed, topP and verbosity vanished from the final call. "Cannot think" and
"does not take a seed" are separate claims; only the first is that filter's to
make. Sent, not guaranteed — Anthropic's API has no such parameter, and
bench runs remains the arbiter of answer reproducibility.
DOCPILOT_EVAL_SEED="" restores the previous behaviour. The first seeded run is
not comparable with unseeded history on the answer half; meta.seed is where
that line falls.

A rerun no longer erases the run before it. The report name is a pure
function of the inputs, so a rerun with nothing changed wrote over the very file
it was being diffed against. The existing file is now copied into
docpilot/reports/history/ first, under its own ranAt. latest.json,
reportName and the pairing are untouched, and the directory is gitignored:
long-term history is git, this covers one working cycle.

A transferred calibration stops looking measured. --transfer runs no
sweep, and both sweep sections printed a table header with no rows under it —
which reads as a measurement that found nothing. They now say what was inherited
and from where. retrievalMisses printed 0/0 (bound 5%) on a probe set where
not one probe carries gold_page; it now says the bound is not armed, and
writes armed: false.

doctor --json and lint --json, docpilot help <command>, and one law for
.env.local.
doctor is documented as a CI gate and answered only in prose,
so gating on it meant grepping wording that is not a contract; --json puts one
object on stdout — the same facts the rows carry, plus the installed version and
ready as a boolean — while the diagnosis still goes to stderr, so
doctor --json | jq .ready shows the report to the operator and the answer to
the script. The exit code is unchanged in both commands.

.env and .env.local are now read once, by the launcher, and the existing
environment wins
. That law was written down in cli-context.ts and followed by
three of seven readers: bin/docpilot.js and vocabulary inverted it, so a
one-off OPENROUTER_API_KEY=… npx docpilot … was overruled by a checked-in
.env; index put the file in a private object that never reached
process.env; and bench and lint did not read it at all, while this CLI's
own help told you to put your key there. All six copies are gone and every
command sees one environment.

Help gained docpilot help <command> as a synonym for --help, --version in
the global block and in the reference, and the short spelling of a flag that has
one (--yes, -y). feedback's hand-written usage block is deleted — it is
rendered from the flag table like every other command's — and the "Full
reference" footer takes its address from one named constant instead of a string
copied into the renderer.

npx docpilot <command> is correct once the package is installed and wrong as a
one-off, because the unscoped name on npm is not this package. The help and the
reference now say so and show npx @cloflin/docpilot.

Four exit codes, and diagnostics on stderr. There were two, and cancelling
was one of them: Ctrl-C or Ctrl-D at either of the package's prompts returned
0, telling a script that the thing it asked for had been done. It returns
130 now. A command line this package cannot parse returns 2 — an unknown
command, flag, mode or value — which separates a typo from 1, the code a run
that was attempted and failed returns. The scale is documented at
Exit codes; before this it was promised nowhere,
which is why this is a minor and not a major.

CALIBRATION FAILED and doctor's ready NO block moved to stderr, so
docpilot doctor | … no longer swallows the only sentence explaining the
refusal. Progress counters redraw in place only when stderr is a terminal —
piped to a file they print whole lines instead of one line holding every value
the counter ever had. Every error is one line prefixed [docpilot] , the shape
import, feedback and vocabulary already used; the five entry modules' FAIL frame is gone, and so is the stack trace four of them printed by
default. DOCPILOT_DEBUG=1 brings the stack back.

doctor and init are TypeScript, and doctor checks its own flags. Both
were written directly in bin/docpilot.js — 670 lines outside the type checker
and outside the run* contract. doctor --proxyy used to produce no output
about itself and exit on project readiness; it exits 2 and names the flag. A
chat.provider the resolver refuses used to end the command in a stack trace
before it printed the block that would have named the fault; it is now a row in
the report and the diagnosis continues. Its stdout is byte-for-byte what it was.

One flag reader, and one dash is a short option. src/cli-flags.js exported
flagValue and flagGiven and nothing in production called them: seven
hand-written copies read the flags instead, and they had drifted — lint
truncated --file=a=b.jsonl at the first =, index returned '' where it had
been handed a default, vocabulary returned true where --out had been given
no path. All seven are gone. Three command lines that used to be accepted in
silence are now refused, because none of them ever meant what they looked like:
-level=low (one dash on a long name — the table accepted it and every reader
ignored it, so the widest tier ran), --level=low --level=high (every reader
takes the first and drops the rest), and --html-select main (the space form, on
the one command whose other parser has only ever taken =; import and
feedback keep it, because their pages have always shown it). -- now ends the
options instead of being reported as an unknown flag. docpilot bench runs
nothing when it is imported rather than run, and its usage line names the command
rather than answer-bench.js.

The empty state holds five questions, not three. The panel shows what you
configure, up to five; anything past that is still dropped and still named on
stdout. The built-in fallback is still three, so a site that configures none pays
nothing new. Five is not free: five embedding requests plus, with
suggestions.answers on, five model calls — and they are spent again whenever the
corpus hash moves or you edit one opener, because the bundle is fingerprinted
over the whole list rather than per question. On a fifty-a-day free tier that is
ten. The number itself lived in two places, SUGGESTION_LIMIT and a literal 3
in DocPilot.vue, so the warning an author read and the list a reader saw were
free to disagree; it now lives in one, and a test says so.

Added

opener-candidates.js — the openers a corpus with no readers would ask for.
docpilot feedback faq clusters real reader votes, which a site that has not
shipped does not have. This new script in the docs-rag skill reads the index
instead: an <FaqAccordion> question the author already typed, a heading that is
already a question, and — for everything else — the one template the panel
already ships for follow-ups, over a page title. It ranks them, keeps at most one
per section, refuses a pair the panel would refuse to match, cross-scores against
calibration.jsonl so a candidate cannot swallow an unrelated probe, and runs
the real retriever in its lexical-only mode so every candidate carries a real
gate score for zero requests. That score is a floor and not a pass — the
shipped gate is hybrid and the lexical channel cannot clear it alone — so it
proposes and never writes, and the verdict is still the openers block of
npx docpilot index.

Fixed

SuggestionsSettings declared three of its six keys. precomputed,
answers and matchTau shipped in 1.0.1, were resolved, were documented on the
config reference — and were missing from the public .d.ts, so a consumer
writing TypeScript could not set them without a cast. The conformance gate could
not have caught it and never would: excess-property checking applies only to
fresh object literals, so a wider generated object stays assignable to a narrower
interface with optional keys, in both directions, forever. A new rule 11f walks
DEFAULTS against the declaration file instead. The same three keys were also
missing from the pre-configure copy in session.js, under a comment claiming
it equalled its own resolver.

Full Changelog: v1.0.1...v1.1.0