Repository navigation
0.114.0: Reviewer notes, expected analysis cost, percent-encoded UTF-8 in verify
humanish 0.114.0 lets a reviewer note a moment of a recorded run, admits post-run analysis on its
expected cost, and reads percent escapes as UTF-8 when it checks a run for secrets. Pause the study
timeline in an Observer served on 127.0.0.1 and choose "Add a note at 02:31", or run
humanish notes <run> --add --at 02:31 "text". Each note is saved as its own file under notes/
in the run directory, marked on the study timeline and listed under "Reviewer notes" in Findings,
and verify, HTML export and feedback drafts read it. Analysis admission now prices input at 3
bytes per token and output at 12,000 tokens plus 1,000 per participant, and runs an analysis when
that expected cost plus 10% is within the cap, so the default $3 cap no longer refuses analyses
billed at $1 to $2. A refusal names the expected cost, the worst case, the cap and the
humanish analyze --max-cost command that admits it, and study check and live starts give the
expected cost range for the study's participant count. In analyze --json,
admission.estimatedCostUsd is now the expected cost. OpenAI computer-use traces list the
impressions request in conversation.requests with kind: impressions. verify and bundle
export also read percent escapes as UTF-8, so a percent-encoded password that starts with a
non-ASCII letter grades its run blocked, and the known-value scrub finds a percent-encoded value
with non-ASCII characters. The scrub also checks its matches against redaction markers by binary
search, so 1 MiB of markers and values takes well under a second instead of about 5 s.
This is a minor release because three defaults change and one JSON field changes meaning.
Post-run analysis and humanish analyze admit an analysis on its expected cost, so analyses that
0.113.0 refused now run and are billed under the same cap. admission.estimatedCostUsd in
analyze --json, and usage.estimatedAdmissionUsd in a new analysis receipt, hold the expected
cost where 0.113.0 held a conservative estimate. A run whose evidence holds a percent-encoded
secret with a non-ASCII character grades blocked where 0.113.0 graded it share_ready. The
served Observer takes one kind of write, a reviewer note. It removes and renames no command, flag,
study field, bundle field or library export, and the package exports the same 13 values and 31
types as 0.113.0. It adds the notes command, the run file notes/<id>.json
(humanish.run-note.v1), humanish.notes-result.v1, POST /api/notes on a loopback Observer,
admission.worstCaseCostUsd and admission.maxCostUsd in analyze --json, admission in a
cost-refused analysis-automatic/job.json, analysis.participants and
analysis.expectedCostUsd in study check --json, kind on a conversation.requests entry, and
reviewer_notes in feedback drafts. Nine outputs change for the same input, which a script could
notice:
| # | 0.113.0 | 0.114.0 | What to do |
|---|---|---|---|
| 1 | Post-run analysis and analyze were admitted when a conservative estimate was at or under the cap: every UTF-8 byte of the instructions, evidence and schema counted as an input token, and the whole output allowance was charged. In the smoke, a four-participant analysis estimated $3.66 and would have been refused under the default $3. |
Admission compares the expected cost plus 10%, or the worst case when that is lower, with the cap. Input is the text bytes divided by 3 plus framing and image tokens; output is 12,000 tokens plus 1,000 per participant, at most the allowance; the worst case spends the whole allowance. The same analysis was expected at $2.24, admitted and billed $2.11. (#1667) | If you relied on the old refusal to keep large analyses from running, lower review.analysis.maxCostUsd or set review.analysis: false. |
| 2 | analyze --json admission.estimatedCostUsd, and a receipt's usage.estimatedAdmissionUsd, held the conservative estimate. |
Both hold the expected cost, which a bill can exceed. admission adds worstCaseCostUsd and maxCostUsd. (#1667) |
Read worstCaseCostUsd for an upper bound. Admission passes when min(worstCaseCostUsd, estimatedCostUsd × 1.1) is at or under maxCostUsd. |
| 3 | analyze --dry-run printed "Admission estimate: $2.313313; output allowance: 32768 tokens including reasoning. No request sent." A refusal read "The conservative admission estimate exceeds --max-cost. No provider request was sent." |
"Expected cost: $1.01. Worst case: $1.99, if the analyst writes its whole 32768-token output allowance, reasoning included. Admitted under the $3 cap. No request sent." A refusal reads "The expected cost is $1.01 and the worst case is $1.17. With a 10% margin the expected cost is over the $0.01 cap, so no request was sent. To run it, raise the cap: humanish analyze --run --max-cost 2". The code is still analysis_budget_exceeded. (#1667) |
Match a refusal by error.code. Read costs from --json. |
| 4 | An automatic analysis refused for its cost: the run's analysis: line read "not started because its configuration or question was refused"; review said to run a dry run, with next: npx humanish analyze --run <id> --max-cost 3 --dry-run; the Observer said "Analysis was refused before dispatch. Check the CLI admission details. ..." |
Each gives the expected cost, the worst case, the cap and humanish analyze --run <id> --max-cost <n>, with n the worst case rounded up to a whole dollar; review --json analysis.next is that command. analysis-automatic/job.json adds admission (expectedCostUsd, worstCaseCostUsd, maxCostUsd). 0.113.0's review reads such a job as AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN, "could not be read or checked". (#1667) |
Run the named command after agreeing to the spend. Read a 0.114.0 run with 0.114.0 or later. |
| 5 | study check and the live-start line said "refused before it starts if its estimate is over $3", and doctor --study's post-run analysis row said the same. |
study check and the live-start line say "expected $0.75 to $3.79 for 1 participant, depending on how much evidence the run keeps · refused before it starts if the expected cost plus a 10% margin is over $3"; the doctor row says "if its expected cost plus a 10% margin is over $3". study check --json and study summaries add analysis.participants and analysis.expectedCostUsd {low, high}. (#1667) |
Read analysis.expectedCostUsd. Its high end is over the default cap for every participant count. |
| 6 | After dispatch, an analysis billed above its admission estimate was marked partial with analysis_admission_estimate_exceeded. |
Only a bill above the worst case or the cap is marked. (#1667) | Nothing. |
| 7 | An OpenAI computer-use trace's conversation.requests listed participant requests only. |
It ends with the impressions request, {"mode": ..., "kind": "impressions"}. In explicit-context mode that entry carries the same exchanges, screenshots, token estimate and summarized-turn accounting as a turn, and summarizedTurns counts a cut it causes. (#1666) |
Count turns from counts.turns, or skip entries that have kind. |
| 8 | verify and bundle export read each percent escape as one character, so a percent-encoded ?next= redirect holding a password whose first letter is à graded a run share_ready. The known-value scrub missed a percent-encoded value with a non-ASCII character. |
They also read percent escapes as UTF-8: that run grades blocked with PUBLIC_SAFETY_FINDINGS, and the scrub replaces that value. The byte-by-byte reading is still scanned. (#1670) |
Run verify again on a run you plan to share. |
| 9 | observe --run <id> --serve printed "Observer is served read-only over loopback http on 127.0.0.1; only this run's bundle directory is exposed." Its server answered 404 to a POST or DELETE at /api/notes. humanish --help listed no notes. |
The warning reads "... only this run's bundle directory is exposed, and the one change the page can make is adding a reviewer note." The one-run server and observe --all answer POST /api/notes: 201 with the token of the page they served from that page's own address, 403 otherwise, 415 for a body that is not JSON; the one-run server answers 405 to other methods. A served page's run-notes slot carries the token; saved pages and exports carry notes without one. notes is listed under Commands. (#1668) |
Nothing. An Observer shared with --expose takes no note. |
Added
- Reviewer notes. Pause the study timeline in an Observer served on 127.0.0.1 and choose "Add a
note at 02:31" to write what you saw at that moment, for the open participant or the whole
study. Notes show as marks on the study timeline and in a "Reviewer notes" list in Findings,
marked as written by a person, and reloading the Observer shows them at the same moment. Each
note is saved in the run directory as its own file,notes/<id>.json(humanish.run-note.v1),
and never rewritten, so notes added at the same time from the Observer and the CLI are all
kept.humanish notes <run>lists them and
humanish notes <run> --add --at <mm:ss> [--participant <id>] "text"adds one;--jsonworks
for both, ashumanish.notes-result.v1.--participanttakes the participant's id from the
study, such aspower-user, or its stream id, and listings name each participant by its
caption. A run with no capture times, such as a dry run, takes no note
(HUMANISH_NOTE_NO_CLOCK). The server takes a note only with the token in the page it rendered
and from that page's own address, and an Observer shared with--exposetakes none (row 9).
Text that looks like a secret or a local path is replaced before it is saved, andverifyscans
the note files like other run text. Feedback drafts include the notes under "Reviewer notes"
(reviewer_notes, with each note'sparticipant_caption); a draft or an HTML export checks the
notes again as it reads them and refuses ones that look like a secret. An HTML export shows the
notes and offers no way to add one, and a bundle export copiesnotes/. (#1668)
Changed
study checkand live starts give the expected analysis cost range for the study's
participant count: $0.75 to $3.79 for one participant and $1.00 to $4.04 for six with the
default gpt-6-astra analysis (row 5).analyze --dry-runprints the expected cost, the worst
case and the cap (row 3). Inanalyze --json,admission.estimatedCostUsdis now the expected
cost, andadmissionaddsworstCaseCostUsdandmaxCostUsd(row 2). (#1667)- An analysis billed above its expected cost is no longer marked
partial; only a bill above
the worst case or the cap is (row 6). (#1667) - The packed skill tells an agent that a refused analysis prints its costs and the command that
runs it, and to run that command only once the user agrees to the spend. (#1667) - The one-run Observer server answers 405 to methods other than GET and HEAD, and
observe's
loopback warning names the reviewer note as the one change the page can make (row 9). (#1668) - In the repository's benchmark (
pnpm bench, which the package does not ship), each analysis's
cap follows its run's admission estimate with 10% headroom within the remaining budget.
--analysis-max-usdstill sets a fixed cap. Results list cost refusals with their estimate and
cap, and state that those analyses are excluded from recall. (#1665, #1667)
Fixes
- Post-run analysis is no longer refused on an estimate several times its real cost. Admission
now prices input at 3 bytes per token and output at the size expected for the participant
count, and compares that expected cost plus a 10% margin with the cap. On 148 billed analyses
the expected cost was at or above the bill every time. The eight-participant run that the old
estimate put at $11.38 and that was billed $2.29 is now expected at $2.72 and admitted under the
default $3 cap (row 1). (#1667) - A refused analysis says what it would cost and how to run it. The run output,
humanish review
and the Observer give the expected cost, the worst case, the cap and
humanish analyze --run <id> --max-cost <n>, withnthe worst case rounded up (rows 3 and 4).
(#1667) - OpenAI computer-use traces now list the impressions-only request in
conversation.requests,
marked withkind: impressions. In explicit-context mode its record includes the carried
exchanges, screenshots and estimated input tokens, using the same context budget and summaries
as a participant turn (row 7). (#1666) verifyand bundle export also read percent escapes as UTF-8, and the known-value scrub finds a
percent-encoded value with non-ASCII characters. Each escape was read as one character, so
caf%C3%A9read ascafé, and a percent-encoded redirect holding a password that starts
Voilàgraded a run share-ready. The byte-by-byte reading is still scanned too: read as UTF-8,
%E2%80%80is a space that can end a match the byte reading keeps whole (row 8). (#1670,
closes #1645)- The known-value scrub checks each match against the
[REDACTED_...]markers in the text by
binary search. It compared every match with every marker, so 1 MiB of alternating markers and
values took about 5 s; it now takes well under a second. (#1672, part of #1646)
Known gaps
- The contained-file helpers have three races that need a second local process, running as the
same user, to swap a directory in the run for a symlink or FIFO between two filesystem calls:
temporary-file cleanup unlinks by path, contained reads have no byte cap and can block on a
FIFO, and the notes listing can follow a swappednotes/. Forged web requests reach none of
them. (#1669) - A process killed between a note's
linkand theunlinkof its temporary name leaves that
note with two links. Readers skip it and name it, andverifyreports it. - Nothing Mac-specific from 0.113.0 has run on a Mac: the Lima VM sizing,
runtime setup --memory --cpusand the local capacity refusal. - The calibration behind the expected cost is 148 billed
gpt-6-astraanalyses, mostly of
English evidence; other models' tokenizers and non-English evidence were not measured. In the
smoke, the four-participant analysis billed $2.11 against an expected $2.24, 6% under it. - The high end of
study check's expected cost range is over the default $3 cap for every
participant count, so a run that keeps evidence near the limits is refused by default and needs
the--max-costits refusal names. - An older humanish reads a 0.114.0 job recorded for a cost refusal as an unknown outcome (row 4).
- A study whose explicit
review.analysisis refused for its cost still ends withlive run failedand exit 2 while run.json'soutcome.okis true, as in 0.113.0. (#1673) - Every impressions request in the smoke read no cached input in threaded mode, so its input is
billed at the uncached rate. - Design findings cannot say that text or a click target is too small by measure. (#1638)
- The shared secret scrubber leaves any
[REDACTED_...]-shaped span alone, so a value encoded
inside one, such as its hex form, survives, and it has no property-based tests. (#1646) - The impressions ask describes its kinds in visual terms, which can steer a participant toward
visual critique whatever its persona. (#1654) - Reviewer notes ran in Chromium only. Quoted-printable
=XXescapes are still read byte by byte.
Package
humanish 0.114.0 packs to 2.5 MB, 7.7 MB unpacked, 942 files (npm pack on release/0.114.0 at
7c81b32). Against 0.113.0 it adds dist/analysis/admission, dist/cli/commands/notes,
dist/observer/notes-route, dist/run/note-files and dist/run/notes (.js and .d.ts each)
and drops nothing. The package exports only ., with the same 13 values and 31 types as 0.113.0
(pnpm api:proof). It needs Node 22.19 or newer, as before, and @e2b/desktop 2.3.2 or newer as
an optional peer.
The published tarball's shasum is 0640fb28751a8f870ae0dc3750cd737ee733b33b and its integrity
sha512-N7U+EKqHfEcZ/LyjitqbiKzQk/w34pV0YJ8b231BDjfBg3UudT6LDzXDrl0XyaHMEV90GULf0BVX5HUWBsIomA==,
the same as npm pack on the release head 7c81b32, which passed release:check.
Checked before release
A live smoke on a tarball packed from main dabf656, before #1672, in fresh projects under /tmp
with a fresh HOME and npm cache, DO_NOT_TRACK=1, and @e2b/desktop 2.4.0 beside humanish. Keys
came from the user key store through key discovery, with XDG_CONFIG_HOME pointing at it; no
--dotenv was passed. The store's OpenAI organization accepts server-side conversation state,
so every participant ran threaded. Dry runs, refusals and the same CLI commands also ran on npm
0.113.0.
Priced spend: $5.28 for the two live runs, $2.71 of it in analysis, then $6.19 for the
benchmark, $4.28 of it in analysis, and $0.45 for the two dogfood runs.
- try-live, as
initwrites it: reached the goal in 9 turns, input growing every turn from 2,888
to 16,498 tokens, then 16,843 for the impressions request;conversation.requestshas 10
entries, the last{"mode":"threaded","kind":"impressions"}. $0.23 run. Its automatic analysis
was admitted at an expected $1.01 (worst case $1.99, cap $3) and billed $0.60. 5 impressions;
the analysis's one finding cites 2. - A computer-use fan-out, 4 participants on drawDB, all reached the goal: 20, 21, 24 and 16
participant turns, input growing every turn (2,958 to 35,752 for the first), and an impressions
request after each, listed last inconversation.requestswithkind: impressions; 6
impressions each. $2.35 run. Its automatic analysis was admitted under the default $3 cap at an
expected $2.24 (worst case $3.08) and billed $2.11; npm 0.113.0'sanalyze --dry-runon a copy
estimated $3.66 and refused. 7 findings, all citing an impression, and 8 design findings
(partial: the 40-capture limit). study check: "expected $0.75 to $3.79 for 1 participant" (try-live), "$0.90 to $3.94 for 4"
and "$1.00 to $4.04 for 6",--jsonanalysis.participants6 andexpectedCostUsd
{low: 1.0032, high: 4.0392}. npm 0.113.0 printed no range.analyze --dry-run --max-cost 3and--jsonon the finished try-live run: expected $1.01,
worst case $1.99, cap 3,estimatedCostUsd1.005438,worstCaseCostUsd1.993838,
maxCostUsd3.--max-cost 0.01: exit 2, the expected cost, the worst case andhumanish analyze --run <id> --max-cost 2; no attempt added. A local scripted run with
review.analysis.maxCostUsd: 0.01showed the automatic refusal in the run output,review
(next:the command), the Observer panel andjob.jsonadmission.- Reviewer notes on the try-live run:
notes <id> --add --at 0:05with and without
--participant lane-01, listed bynotes <id>and--json(humanish.notes-result.v1, one
file each undernotes/); a note holding a synthetic key and a local path was saved with both
replaced and a warning. In headless Chrome onobserve --run <id> --serve, Findings listed the
notes, a note added through the page's form appeared on the timeline and in the list and after
a reload, and there were no page errors.POST /api/noteswithout the token, with a foreign
Origin, with no Origin or with a rebound Host answered 403,text/plain415 andDELETE405,
and none added a file.verifygraded the runlocal_only(RAW_SCREENSHOTS) before and after
the notes. The HTML export carries the 4 notes with no write token and offers no "Add a note";
the redacted bundle export isshare_readyand holdsnotes/. - A percent-encoded
?next=redirect holdingpassword=and a value starting withà(built at
run time) in a copy of ashare_readybundle: main grades itblockedwith
PUBLIC_SAFETY_FINDINGS, npm 0.113.0share_ready. The control with an ASCII first letter is
blockedon both. - npm 0.113.0's
reviewprints the two live runs exactly as main does, and itsverifygrades
them the same. - After every run,
reclaim --checkexited 0 withclean. Raw sandbox ids appear only in
sandbox-receipts.ndjson, E2B listed 0 tagged sandboxes after the runs, and no key value from
the store appears in any run or kit file, as written, base64 or hex.
On the release branch:
pnpm release:checkon the release head 7c81b32, which adds one CHANGELOG sentence for #1672:
exit 0, 7,571 tests passed and 12 skipped, TUI 158 tests. On the merge head 1bb47e7 before it:
exit 0 in 3.5 min, 480 test files passed and 10 skipped, 7,571 tests passed and 12 skipped; TUI
158 tests; lint 439 warnings at its cap of 439; api:proof "13 values and 31 types match";
public-surface scan of 2,947 text files and 523 binary assets.pnpm docs:generateon that head changed
nothing. Earlier, on the release commit 8d949de before the benchmark commit and the merge: exit
0 in 3.8 min with 7,570 tests passed and a scan of 2,945 text files;pnpm docs:generatethere
wrote 0.114.0 into cli.mdx.pnpm release:dogfoodon 1bb47e7 (the script run with Node, keys read from the user store by
Node's--env-fileinto its environment, and a temporary HOME): "release:dogfood ok", verdict
blocked(blocked_approval), no-spend satisfied, $0.20. The participant raninitand the
free dry run on the installed 0.114.0 and reported "Dry run: 4 participants; evidence passed all
16 checks and wasshare_ready." and "It recommendshumanish run try-live." It stopped at the
credentials the gate withholds.verifyon the run:share_ready, 16 of 16;reclaim --check
clean. A first dogfood run on 8d949de ($0.25) reached the same verdict, 16 of 16.- After publish, the post-publish check against the npm registry (
TMPDIR=/tmp verify.sh 0.114.0, with the registry and unpkg rows): 98 of 98 passed. - The post-publish check on the release tarball from 1bb47e7: 92 of 92 passed. Its 5 new rows
covernoteshelp, the dry-run refusal, three notes added on a local scripted run (one with a
synthetic key and a path, saved replaced) and--json, withverifygrading the run the same
before and after (#1668); the served Observer'sPOST /api/notesanswering 403 without the
token, with a foreign Origin or another Host, 415 fortext/plain, 405 forDELETEand 201
with the token (#1668);analyze --dry-run --jsonwithworstCaseCostUsdandmaxCostUsdand
the--max-cost 0.01refusal (#1667); thestudy checkrange (#1667); and a percent-encoded
password starting with U+00E0 gradedblocked(#1670). The same script on npm 0.113.0 passed
87 and failed exactly those 5 rows, and on the 8d949de tarball it passed 92 of 92. - The consumer check: the 0.113.0 check's library and CLI consumer, plus nine CLI lines for this
release. Typecheck 0 errors on both versions. Every output line from the 0.113.0 check is the
same on npm 0.113.0 and the release tarball from 1bb47e7, and from 8d949de. The new lines:
notes --helpandnotesunder Commands, a note added and listed with--jsonon the check's
local scripted run, the dry-run refusalHUMANISH_NOTE_NO_CLOCK,analyze --dry-run --json
withworstCaseCostUsdandmaxCostUsd, the refusal naming the costs and the command, and
study checkwith 6 participants giving$1.00 to $4.04. pnpm benchon 8d949de with its defaults (3 runs per arm,openai-computer-use,neutral
mission), committed in 2ec7460: report recall 11/15, analysis recall 9/15, 0 invented on either
arm, $6.19 estimated. 0.113.0 scored 10/15 and 6/10, and 11/15 and 11/15 read by hand. Each
analysis's cap came from its run's admission estimate plus 10% ($1.14 to $1.30 against bills of
$0.59 to $0.88), and no analysis was refused by its cost cap, so analysis recall has all 15
planted chances. Two planted analysis lines the rubric left unresolved are D1 ("Completed tasks
remained after the clear attempt") and D5 ("Edited wording reverted despite repeated Save
attempts"), which puts analysis recall at 11/15 read by hand. No planted participant typed more
than 30 characters in one action, so none met D2. Every participant ranthreadedand its input
grew every turn. E2B listed 0 tagged sandboxes after it. The benchmark ran before the merge of
#1672, which changes only the scrub's marker check.
Not verified
- macOS and Windows. Every check ran on Linux x64 with Node 24.12.
- A live run on the published package: the live smoke ran on a main tarball, and the post-publish
rows are offline dry runs, refusals and local scripted runs. - The impressions request record in explicit-context mode: the store's OpenAI organization
accepts server-side state, so every live participant ranthreaded, whose entries carry only
modeandkind. - A feedback draft with reviewer notes, a draft refused for a secret-shaped note, and an export
blocked by one: their tests inrelease:checkcover them, and no check here ran them. - Notes on the run library (
observe --all), and the Observer's note form at phone width. - #1672 in a live run or the benchmark: both ran before the merge.
release:check, the consumer
check, the post-publish check and the second dogfood run ran with it. - The eight-participant figures in Fixes ($11.38, $2.29, $2.72) come from #1667; no run here
reproduced them. - The local VM, email capture and receiving, local-agent participants, terminal participants and
the shared-world route: not run live on 0.114.0.