Skip to content

0.114.0: Reviewer notes, expected analysis cost, percent-encoded UTF-8 in verify

Choose a tag to compare

@danielgwilson danielgwilson released this 08 Oct 01:17
· 244 commits to main since this release
19a5fea

humanish 0.114.0 lets a reviewer note a moment of a recorded run, admits post-run analysis on its
expected cost, and reads percent escapes as UTF-8 when it checks a run for secrets. Pause the study
timeline in an Observer served on 127.0.0.1 and choose "Add a note at 02:31", or run
humanish notes <run> --add --at 02:31 "text". Each note is saved as its own file under notes/
in the run directory, marked on the study timeline and listed under "Reviewer notes" in Findings,
and verify, HTML export and feedback drafts read it. Analysis admission now prices input at 3
bytes per token and output at 12,000 tokens plus 1,000 per participant, and runs an analysis when
that expected cost plus 10% is within the cap, so the default $3 cap no longer refuses analyses
billed at $1 to $2. A refusal names the expected cost, the worst case, the cap and the
humanish analyze --max-cost command that admits it, and study check and live starts give the
expected cost range for the study's participant count. In analyze --json,
admission.estimatedCostUsd is now the expected cost. OpenAI computer-use traces list the
impressions request in conversation.requests with kind: impressions. verify and bundle
export also read percent escapes as UTF-8, so a percent-encoded password that starts with a
non-ASCII letter grades its run blocked, and the known-value scrub finds a percent-encoded value
with non-ASCII characters. The scrub also checks its matches against redaction markers by binary
search, so 1 MiB of markers and values takes well under a second instead of about 5 s.

This is a minor release because three defaults change and one JSON field changes meaning.
Post-run analysis and humanish analyze admit an analysis on its expected cost, so analyses that
0.113.0 refused now run and are billed under the same cap. admission.estimatedCostUsd in
analyze --json, and usage.estimatedAdmissionUsd in a new analysis receipt, hold the expected
cost where 0.113.0 held a conservative estimate. A run whose evidence holds a percent-encoded
secret with a non-ASCII character grades blocked where 0.113.0 graded it share_ready. The
served Observer takes one kind of write, a reviewer note. It removes and renames no command, flag,
study field, bundle field or library export, and the package exports the same 13 values and 31
types as 0.113.0. It adds the notes command, the run file notes/<id>.json
(humanish.run-note.v1), humanish.notes-result.v1, POST /api/notes on a loopback Observer,
admission.worstCaseCostUsd and admission.maxCostUsd in analyze --json, admission in a
cost-refused analysis-automatic/job.json, analysis.participants and
analysis.expectedCostUsd in study check --json, kind on a conversation.requests entry, and
reviewer_notes in feedback drafts. Nine outputs change for the same input, which a script could
notice:

# 0.113.0 0.114.0 What to do
1 Post-run analysis and analyze were admitted when a conservative estimate was at or under the cap: every UTF-8 byte of the instructions, evidence and schema counted as an input token, and the whole output allowance was charged. In the smoke, a four-participant analysis estimated $3.66 and would have been refused under the default $3. Admission compares the expected cost plus 10%, or the worst case when that is lower, with the cap. Input is the text bytes divided by 3 plus framing and image tokens; output is 12,000 tokens plus 1,000 per participant, at most the allowance; the worst case spends the whole allowance. The same analysis was expected at $2.24, admitted and billed $2.11. (#1667) If you relied on the old refusal to keep large analyses from running, lower review.analysis.maxCostUsd or set review.analysis: false.
2 analyze --json admission.estimatedCostUsd, and a receipt's usage.estimatedAdmissionUsd, held the conservative estimate. Both hold the expected cost, which a bill can exceed. admission adds worstCaseCostUsd and maxCostUsd. (#1667) Read worstCaseCostUsd for an upper bound. Admission passes when min(worstCaseCostUsd, estimatedCostUsd × 1.1) is at or under maxCostUsd.
3 analyze --dry-run printed "Admission estimate: $2.313313; output allowance: 32768 tokens including reasoning. No request sent." A refusal read "The conservative admission estimate exceeds --max-cost. No provider request was sent." "Expected cost: $1.01. Worst case: $1.99, if the analyst writes its whole 32768-token output allowance, reasoning included. Admitted under the $3 cap. No request sent." A refusal reads "The expected cost is $1.01 and the worst case is $1.17. With a 10% margin the expected cost is over the $0.01 cap, so no request was sent. To run it, raise the cap: humanish analyze --run --max-cost 2". The code is still analysis_budget_exceeded. (#1667) Match a refusal by error.code. Read costs from --json.
4 An automatic analysis refused for its cost: the run's analysis: line read "not started because its configuration or question was refused"; review said to run a dry run, with next: npx humanish analyze --run <id> --max-cost 3 --dry-run; the Observer said "Analysis was refused before dispatch. Check the CLI admission details. ..." Each gives the expected cost, the worst case, the cap and humanish analyze --run <id> --max-cost <n>, with n the worst case rounded up to a whole dollar; review --json analysis.next is that command. analysis-automatic/job.json adds admission (expectedCostUsd, worstCaseCostUsd, maxCostUsd). 0.113.0's review reads such a job as AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN, "could not be read or checked". (#1667) Run the named command after agreeing to the spend. Read a 0.114.0 run with 0.114.0 or later.
5 study check and the live-start line said "refused before it starts if its estimate is over $3", and doctor --study's post-run analysis row said the same. study check and the live-start line say "expected $0.75 to $3.79 for 1 participant, depending on how much evidence the run keeps · refused before it starts if the expected cost plus a 10% margin is over $3"; the doctor row says "if its expected cost plus a 10% margin is over $3". study check --json and study summaries add analysis.participants and analysis.expectedCostUsd {low, high}. (#1667) Read analysis.expectedCostUsd. Its high end is over the default cap for every participant count.
6 After dispatch, an analysis billed above its admission estimate was marked partial with analysis_admission_estimate_exceeded. Only a bill above the worst case or the cap is marked. (#1667) Nothing.
7 An OpenAI computer-use trace's conversation.requests listed participant requests only. It ends with the impressions request, {"mode": ..., "kind": "impressions"}. In explicit-context mode that entry carries the same exchanges, screenshots, token estimate and summarized-turn accounting as a turn, and summarizedTurns counts a cut it causes. (#1666) Count turns from counts.turns, or skip entries that have kind.
8 verify and bundle export read each percent escape as one character, so a percent-encoded ?next= redirect holding a password whose first letter is à graded a run share_ready. The known-value scrub missed a percent-encoded value with a non-ASCII character. They also read percent escapes as UTF-8: that run grades blocked with PUBLIC_SAFETY_FINDINGS, and the scrub replaces that value. The byte-by-byte reading is still scanned. (#1670) Run verify again on a run you plan to share.
9 observe --run <id> --serve printed "Observer is served read-only over loopback http on 127.0.0.1; only this run's bundle directory is exposed." Its server answered 404 to a POST or DELETE at /api/notes. humanish --help listed no notes. The warning reads "... only this run's bundle directory is exposed, and the one change the page can make is adding a reviewer note." The one-run server and observe --all answer POST /api/notes: 201 with the token of the page they served from that page's own address, 403 otherwise, 415 for a body that is not JSON; the one-run server answers 405 to other methods. A served page's run-notes slot carries the token; saved pages and exports carry notes without one. notes is listed under Commands. (#1668) Nothing. An Observer shared with --expose takes no note.

Added

  • Reviewer notes. Pause the study timeline in an Observer served on 127.0.0.1 and choose "Add a
    note at 02:31" to write what you saw at that moment, for the open participant or the whole
    study. Notes show as marks on the study timeline and in a "Reviewer notes" list in Findings,
    marked as written by a person, and reloading the Observer shows them at the same moment. Each
    note is saved in the run directory as its own file, notes/<id>.json (humanish.run-note.v1),
    and never rewritten, so notes added at the same time from the Observer and the CLI are all
    kept. humanish notes <run> lists them and
    humanish notes <run> --add --at <mm:ss> [--participant <id>] "text" adds one; --json works
    for both, as humanish.notes-result.v1. --participant takes the participant's id from the
    study, such as power-user, or its stream id, and listings name each participant by its
    caption. A run with no capture times, such as a dry run, takes no note
    (HUMANISH_NOTE_NO_CLOCK). The server takes a note only with the token in the page it rendered
    and from that page's own address, and an Observer shared with --expose takes none (row 9).
    Text that looks like a secret or a local path is replaced before it is saved, and verify scans
    the note files like other run text. Feedback drafts include the notes under "Reviewer notes"
    (reviewer_notes, with each note's participant_caption); a draft or an HTML export checks the
    notes again as it reads them and refuses ones that look like a secret. An HTML export shows the
    notes and offers no way to add one, and a bundle export copies notes/. (#1668)

Changed

  • study check and live starts give the expected analysis cost range for the study's
    participant count: $0.75 to $3.79 for one participant and $1.00 to $4.04 for six with the
    default gpt-6-astra analysis (row 5). analyze --dry-run prints the expected cost, the worst
    case and the cap (row 3). In analyze --json, admission.estimatedCostUsd is now the expected
    cost, and admission adds worstCaseCostUsd and maxCostUsd (row 2). (#1667)
  • An analysis billed above its expected cost is no longer marked partial; only a bill above
    the worst case or the cap is (row 6). (#1667)
  • The packed skill tells an agent that a refused analysis prints its costs and the command that
    runs it, and to run that command only once the user agrees to the spend. (#1667)
  • The one-run Observer server answers 405 to methods other than GET and HEAD, and observe's
    loopback warning names the reviewer note as the one change the page can make (row 9). (#1668)
  • In the repository's benchmark (pnpm bench, which the package does not ship), each analysis's
    cap follows its run's admission estimate with 10% headroom within the remaining budget.
    --analysis-max-usd still sets a fixed cap. Results list cost refusals with their estimate and
    cap, and state that those analyses are excluded from recall. (#1665, #1667)

Fixes

  • Post-run analysis is no longer refused on an estimate several times its real cost. Admission
    now prices input at 3 bytes per token and output at the size expected for the participant
    count, and compares that expected cost plus a 10% margin with the cap. On 148 billed analyses
    the expected cost was at or above the bill every time. The eight-participant run that the old
    estimate put at $11.38 and that was billed $2.29 is now expected at $2.72 and admitted under the
    default $3 cap (row 1). (#1667)
  • A refused analysis says what it would cost and how to run it. The run output, humanish review
    and the Observer give the expected cost, the worst case, the cap and
    humanish analyze --run <id> --max-cost <n>, with n the worst case rounded up (rows 3 and 4).
    (#1667)
  • OpenAI computer-use traces now list the impressions-only request in conversation.requests,
    marked with kind: impressions. In explicit-context mode its record includes the carried
    exchanges, screenshots and estimated input tokens, using the same context budget and summaries
    as a participant turn (row 7). (#1666)
  • verify and bundle export also read percent escapes as UTF-8, and the known-value scrub finds a
    percent-encoded value with non-ASCII characters. Each escape was read as one character, so
    caf%C3%A9 read as café, and a percent-encoded redirect holding a password that starts
    Voilà graded a run share-ready. The byte-by-byte reading is still scanned too: read as UTF-8,
    %E2%80%80 is a space that can end a match the byte reading keeps whole (row 8). (#1670,
    closes #1645)
  • The known-value scrub checks each match against the [REDACTED_...] markers in the text by
    binary search. It compared every match with every marker, so 1 MiB of alternating markers and
    values took about 5 s; it now takes well under a second. (#1672, part of #1646)

Known gaps

  • The contained-file helpers have three races that need a second local process, running as the
    same user, to swap a directory in the run for a symlink or FIFO between two filesystem calls:
    temporary-file cleanup unlinks by path, contained reads have no byte cap and can block on a
    FIFO, and the notes listing can follow a swapped notes/. Forged web requests reach none of
    them. (#1669)
  • A process killed between a note's link and the unlink of its temporary name leaves that
    note with two links. Readers skip it and name it, and verify reports it.
  • Nothing Mac-specific from 0.113.0 has run on a Mac: the Lima VM sizing, runtime setup --memory --cpus and the local capacity refusal.
  • The calibration behind the expected cost is 148 billed gpt-6-astra analyses, mostly of
    English evidence; other models' tokenizers and non-English evidence were not measured. In the
    smoke, the four-participant analysis billed $2.11 against an expected $2.24, 6% under it.
  • The high end of study check's expected cost range is over the default $3 cap for every
    participant count, so a run that keeps evidence near the limits is refused by default and needs
    the --max-cost its refusal names.
  • An older humanish reads a 0.114.0 job recorded for a cost refusal as an unknown outcome (row 4).
  • A study whose explicit review.analysis is refused for its cost still ends with live run failed and exit 2 while run.json's outcome.ok is true, as in 0.113.0. (#1673)
  • Every impressions request in the smoke read no cached input in threaded mode, so its input is
    billed at the uncached rate.
  • Design findings cannot say that text or a click target is too small by measure. (#1638)
  • The shared secret scrubber leaves any [REDACTED_...]-shaped span alone, so a value encoded
    inside one, such as its hex form, survives, and it has no property-based tests. (#1646)
  • The impressions ask describes its kinds in visual terms, which can steer a participant toward
    visual critique whatever its persona. (#1654)
  • Reviewer notes ran in Chromium only. Quoted-printable =XX escapes are still read byte by byte.

Package

humanish 0.114.0 packs to 2.5 MB, 7.7 MB unpacked, 942 files (npm pack on release/0.114.0 at
7c81b32). Against 0.113.0 it adds dist/analysis/admission, dist/cli/commands/notes,
dist/observer/notes-route, dist/run/note-files and dist/run/notes (.js and .d.ts each)
and drops nothing. The package exports only ., with the same 13 values and 31 types as 0.113.0
(pnpm api:proof). It needs Node 22.19 or newer, as before, and @e2b/desktop 2.3.2 or newer as
an optional peer.

The published tarball's shasum is 0640fb28751a8f870ae0dc3750cd737ee733b33b and its integrity
sha512-N7U+EKqHfEcZ/LyjitqbiKzQk/w34pV0YJ8b231BDjfBg3UudT6LDzXDrl0XyaHMEV90GULf0BVX5HUWBsIomA==,
the same as npm pack on the release head 7c81b32, which passed release:check.

Checked before release

A live smoke on a tarball packed from main dabf656, before #1672, in fresh projects under /tmp
with a fresh HOME and npm cache, DO_NOT_TRACK=1, and @e2b/desktop 2.4.0 beside humanish. Keys
came from the user key store through key discovery, with XDG_CONFIG_HOME pointing at it; no
--dotenv was passed. The store's OpenAI organization accepts server-side conversation state,
so every participant ran threaded. Dry runs, refusals and the same CLI commands also ran on npm
0.113.0.
Priced spend: $5.28 for the two live runs, $2.71 of it in analysis, then $6.19 for the
benchmark, $4.28 of it in analysis, and $0.45 for the two dogfood runs.

  • try-live, as init writes it: reached the goal in 9 turns, input growing every turn from 2,888
    to 16,498 tokens, then 16,843 for the impressions request; conversation.requests has 10
    entries, the last {"mode":"threaded","kind":"impressions"}. $0.23 run. Its automatic analysis
    was admitted at an expected $1.01 (worst case $1.99, cap $3) and billed $0.60. 5 impressions;
    the analysis's one finding cites 2.
  • A computer-use fan-out, 4 participants on drawDB, all reached the goal: 20, 21, 24 and 16
    participant turns, input growing every turn (2,958 to 35,752 for the first), and an impressions
    request after each, listed last in conversation.requests with kind: impressions; 6
    impressions each. $2.35 run. Its automatic analysis was admitted under the default $3 cap at an
    expected $2.24 (worst case $3.08) and billed $2.11; npm 0.113.0's analyze --dry-run on a copy
    estimated $3.66 and refused. 7 findings, all citing an impression, and 8 design findings
    (partial: the 40-capture limit).
  • study check: "expected $0.75 to $3.79 for 1 participant" (try-live), "$0.90 to $3.94 for 4"
    and "$1.00 to $4.04 for 6", --json analysis.participants 6 and expectedCostUsd
    {low: 1.0032, high: 4.0392}. npm 0.113.0 printed no range.
  • analyze --dry-run --max-cost 3 and --json on the finished try-live run: expected $1.01,
    worst case $1.99, cap 3, estimatedCostUsd 1.005438, worstCaseCostUsd 1.993838,
    maxCostUsd 3. --max-cost 0.01: exit 2, the expected cost, the worst case and humanish analyze --run <id> --max-cost 2; no attempt added. A local scripted run with
    review.analysis.maxCostUsd: 0.01 showed the automatic refusal in the run output, review
    (next: the command), the Observer panel and job.json admission.
  • Reviewer notes on the try-live run: notes <id> --add --at 0:05 with and without
    --participant lane-01, listed by notes <id> and --json (humanish.notes-result.v1, one
    file each under notes/); a note holding a synthetic key and a local path was saved with both
    replaced and a warning. In headless Chrome on observe --run <id> --serve, Findings listed the
    notes, a note added through the page's form appeared on the timeline and in the list and after
    a reload, and there were no page errors. POST /api/notes without the token, with a foreign
    Origin, with no Origin or with a rebound Host answered 403, text/plain 415 and DELETE 405,
    and none added a file. verify graded the run local_only (RAW_SCREENSHOTS) before and after
    the notes. The HTML export carries the 4 notes with no write token and offers no "Add a note";
    the redacted bundle export is share_ready and holds notes/.
  • A percent-encoded ?next= redirect holding password= and a value starting with à (built at
    run time) in a copy of a share_ready bundle: main grades it blocked with
    PUBLIC_SAFETY_FINDINGS, npm 0.113.0 share_ready. The control with an ASCII first letter is
    blocked on both.
  • npm 0.113.0's review prints the two live runs exactly as main does, and its verify grades
    them the same.
  • After every run, reclaim --check exited 0 with clean. Raw sandbox ids appear only in
    sandbox-receipts.ndjson, E2B listed 0 tagged sandboxes after the runs, and no key value from
    the store appears in any run or kit file, as written, base64 or hex.

On the release branch:

  • pnpm release:check on the release head 7c81b32, which adds one CHANGELOG sentence for #1672:
    exit 0, 7,571 tests passed and 12 skipped, TUI 158 tests. On the merge head 1bb47e7 before it:
    exit 0 in 3.5 min, 480 test files passed and 10 skipped, 7,571 tests passed and 12 skipped; TUI
    158 tests; lint 439 warnings at its cap of 439; api:proof "13 values and 31 types match";
    public-surface scan of 2,947 text files and 523 binary assets. pnpm docs:generate on that head changed
    nothing. Earlier, on the release commit 8d949de before the benchmark commit and the merge: exit
    0 in 3.8 min with 7,570 tests passed and a scan of 2,945 text files; pnpm docs:generate there
    wrote 0.114.0 into cli.mdx.
  • pnpm release:dogfood on 1bb47e7 (the script run with Node, keys read from the user store by
    Node's --env-file into its environment, and a temporary HOME): "release:dogfood ok", verdict
    blocked (blocked_approval), no-spend satisfied, $0.20. The participant ran init and the
    free dry run on the installed 0.114.0 and reported "Dry run: 4 participants; evidence passed all
    16 checks and was share_ready." and "It recommends humanish run try-live." It stopped at the
    credentials the gate withholds. verify on the run: share_ready, 16 of 16; reclaim --check
    clean. A first dogfood run on 8d949de ($0.25) reached the same verdict, 16 of 16.
  • After publish, the post-publish check against the npm registry (TMPDIR=/tmp verify.sh 0.114.0, with the registry and unpkg rows): 98 of 98 passed.
  • The post-publish check on the release tarball from 1bb47e7: 92 of 92 passed. Its 5 new rows
    cover notes help, the dry-run refusal, three notes added on a local scripted run (one with a
    synthetic key and a path, saved replaced) and --json, with verify grading the run the same
    before and after (#1668); the served Observer's POST /api/notes answering 403 without the
    token, with a foreign Origin or another Host, 415 for text/plain, 405 for DELETE and 201
    with the token (#1668); analyze --dry-run --json with worstCaseCostUsd and maxCostUsd and
    the --max-cost 0.01 refusal (#1667); the study check range (#1667); and a percent-encoded
    password starting with U+00E0 graded blocked (#1670). The same script on npm 0.113.0 passed
    87 and failed exactly those 5 rows, and on the 8d949de tarball it passed 92 of 92.
  • The consumer check: the 0.113.0 check's library and CLI consumer, plus nine CLI lines for this
    release. Typecheck 0 errors on both versions. Every output line from the 0.113.0 check is the
    same on npm 0.113.0 and the release tarball from 1bb47e7, and from 8d949de. The new lines:
    notes --help and notes under Commands, a note added and listed with --json on the check's
    local scripted run, the dry-run refusal HUMANISH_NOTE_NO_CLOCK, analyze --dry-run --json
    with worstCaseCostUsd and maxCostUsd, the refusal naming the costs and the command, and
    study check with 6 participants giving $1.00 to $4.04.
  • pnpm bench on 8d949de with its defaults (3 runs per arm, openai-computer-use, neutral
    mission), committed in 2ec7460: report recall 11/15, analysis recall 9/15, 0 invented on either
    arm, $6.19 estimated. 0.113.0 scored 10/15 and 6/10, and 11/15 and 11/15 read by hand. Each
    analysis's cap came from its run's admission estimate plus 10% ($1.14 to $1.30 against bills of
    $0.59 to $0.88), and no analysis was refused by its cost cap, so analysis recall has all 15
    planted chances. Two planted analysis lines the rubric left unresolved are D1 ("Completed tasks
    remained after the clear attempt") and D5 ("Edited wording reverted despite repeated Save
    attempts"), which puts analysis recall at 11/15 read by hand. No planted participant typed more
    than 30 characters in one action, so none met D2. Every participant ran threaded and its input
    grew every turn. E2B listed 0 tagged sandboxes after it. The benchmark ran before the merge of
    #1672, which changes only the scrub's marker check.

Not verified

  • macOS and Windows. Every check ran on Linux x64 with Node 24.12.
  • A live run on the published package: the live smoke ran on a main tarball, and the post-publish
    rows are offline dry runs, refusals and local scripted runs.
  • The impressions request record in explicit-context mode: the store's OpenAI organization
    accepts server-side state, so every live participant ran threaded, whose entries carry only
    mode and kind.
  • A feedback draft with reviewer notes, a draft refused for a secret-shaped note, and an export
    blocked by one: their tests in release:check cover them, and no check here ran them.
  • Notes on the run library (observe --all), and the Observer's note form at phone width.
  • #1672 in a live run or the benchmark: both ran before the merge. release:check, the consumer
    check, the post-publish check and the second dogfood run ran with it.
  • The eight-participant figures in Fixes ($11.38, $2.29, $2.72) come from #1667; no run here
    reproduced them.
  • The local VM, email capture and receiving, local-agent participants, terminal participants and
    the shared-world route: not run live on 0.114.0.