Skip to content

0.115.0: Closing report limits, admitted analysis cost, concurrent reviewer notes, known-value scrub

Choose a tag to compare

@danielgwilson danielgwilson released this 08 Oct 05:06
· 220 commits to main since this release
62d05ce

humanish 0.115.0 asks the OpenAI computer-use participant for its closing report and impressions
within the limits its reply is checked against, keeps every reviewer note added at the same time,
and finds more encoded forms of a registered secret. The participant's request schema now carries
the limits the parser already enforced: a summary of up to 4,000 characters, up to 8 friction
reports of up to 2,000 characters and up to 6 impressions of up to 500 characters, so a longer
reply is no longer requested and then dropped. analyze --json adds admission.admittedCostUsd,
the smallest --max-cost that admits the analysis, and humanish analyze names its refusal
command as npx humanish analyze where humanish is a dev dependency. A reviewer note is created
under its own name, so notes added together from the Observer and humanish notes --add are all
kept. export --local-only keeps a blocked run blocked when its analysis fails the export's
own check. The known-value scrub exempts only the markers humanish writes, reads text in the same
five ways as verify, and finds a value that overlaps itself in time linear in the text. The TUI's
study screen shows the expected analysis cost line that study check prints, telemetry reads
CI=false as off, and humanish keys set constructor is refused with the list of provider keys.

This is a minor release because a JSON output gains a field and three behaviors change for the
same input. analyze --json adds admission.admittedCostUsd. The OpenAI computer-use
participant's closing report and impressions requests now send length and count limits, so the
provider is asked for a different reply shape. export --local-only can now write blocked where
0.114.0 wrote local_only. Telemetry's ci and studyParticipant properties read false as off.
The release removes and renames no command, flag, study field, bundle field or library export, and
the package exports the same 13 values and 31 types as 0.114.0. Twelve outputs change for the same
input, which a script could notice:

# 0.114.0 0.115.0 What to do
1 The OpenAI computer-use participant's participant_closing_report and participant_impressions schemas had no maxLength, minLength or maxItems. A reply over the limits the parser checks was requested, then dropped, and the debrief recorded failed. Both schemas carry the parser's limits: summary 1 to 4,000 characters, frictionReports at most 8 of 1 to 2,000, impressions at most 6 with text 1 to 500. Replies are parsed with the same zod schema. The Codex participant's final schema is unchanged keyword for keyword. The Observer reads a recorded closing account with more than 8 friction reports or 6 impressions as unreadable; no humanish version records more. (#1680) Nothing.
2 analyze --json admission had estimatedCostUsd, worstCaseCostUsd and maxCostUsd. It adds admittedCostUsd: the expected cost times 1.1, or the worst case when that is lower, rounded up at six places. Admission passes when it is at or under maxCostUsd. In the smoke, 1.131406 for a one-participant run and 2.24649 for four. (#1681) Read admittedCostUsd as the smallest --max-cost that admits the analysis.
3 A cost refusal from humanish analyze named a bare humanish analyze --run <id> --max-cost <n>. It names the command the way the install runs humanish, such as npx humanish analyze --run <id> --max-cost 2 in a project and npx humanish@0.115.0 analyze ... through npx --package. The run output, review and the Observer already did. (#1681) Match a refusal by error.code, analysis_budget_exceeded.
4 export --local-only of a run that verify graded blocked, with an analysis that failed the export's own check, wrote local_only into the result and the file's share record. It writes blocked: a failed export check only moves a run toward blocked, as in verify. (#1682) Read shareSafety.status. A run with a planted secret still fails verify first and is refused, as before.
5 Telemetry counted any CI or HUMANISH_STUDY_PARTICIPANT value other than empty or 0 as set, so CI=false sent ci: true. Unset, blank, 0 and false are off, as the update check and is-in-ci read them. HUMANISH_STRICT_KEYS keeps its =1 reading. (#1678) Nothing.
6 keys set constructor printed "Not a valid env name: function Object() { [native code] }" and keys set __proto__ "Not a valid env name: [object Object]", exit 2. Both print "The store holds provider keys only (OPENAI_API_KEY, ANTHROPIC_API_KEY, E2B_API_KEY, GH_TOKEN, GITHUB_TOKEN, CODEX_API_KEY, AGENTMAIL_API_KEY). For anything else, use an explicit --dotenv.", exit 2. (#1678) Nothing.
7 The TUI's study screen read "After live runs: analysis · gpt-6-astra · separate $3 admission estimate limit · not a billing cap". It shows the line study check prints: "After live runs: default analysis · gpt-6-astra · expected $0.75 to $3.79 for 1 participant, depending on how much evidence the run keeps · refused before it starts if the expected cost plus a 10% margin is over $3; this is not a billing cap. Set review.analysis: false to disable." (#1678) Nothing.
8 The warning for a participant whose persona has no background ended "Run humanish study show --json to see what the participant receives.", in study check output, study.warning events and the library's run result. It names the install's invocation, such as npx humanish study show <id> --json. (#1678) Match the warning by its first sentence.
9 A feedback draft for an amended finding said "This finding was corrected in human review. The reviewer's claim above replaces its original headline and account." It uses the sentence review and the Observer show: "Corrected in human review. The reviewer's claim replaces the original headline and account." (#1678) Nothing.
10 The reviewer-notes share-safety reason read "The reviewer notes being exported ..." in an HTML export and "The reviewer notes the draft would include ..." in a feedback draft. Both read "The reviewer notes being shared match secret, token or local-path patterns." or "... hold encoded text the scan cannot read." (#1682) Match by reason code.
11 notes --add on a run whose storage check failed said the run "has no run.json humanish can read safely" and suggested a bare humanish verify --run <id>. Two notes added at the same moment could get that refusal (#1679). A POST /api/notes body with atMs of 1e999 was refused as HUMANISH_NOTE_OUTSIDE_RUN. A storage refusal names what the check found, such as "... must not contain symbolic links.", with the same code HUMANISH_INVALID_RUN_BUNDLE; the run.json message names npx humanish verify in a project. Concurrent adds all succeed. The 1e999 body is HUMANISH_NOTES_BODY, still 400; the Observer page cannot send one. (#1682) Nothing.
12 The known-value scrub skipped every [REDACTED_...]-shaped span, picked the reading with the most finds, and could leave a value encoded twice encoded once. verify read base64 two levels deep only as written and decoded, did not read percent escapes that a \u0025 wrote as UTF-8, and expanded &constructor; to a function's source text. The scrub exempts only text wholly inside [REDACTED_SECRET], [REDACTED_LOCAL_PATH], [REDACTED_RUNTIME_PATH], [REDACTED_PROMPT_TEXT] and [REDACTED_LOBBY_CODE], so [REDACTED_373433393231] holding 743921 in hex becomes [REDACTED_[REDACTED_SECRET]]. It returns the decoded text with the values replaced, and replaces the whole text with [REDACTED_SECRET] when another reading of that result still holds a value. verify and bundle export read every text in five ways. &constructor; stays as written, so /home/&constructor; reads as clean. (#1674) Run verify again on a run you plan to share.

Added

  • analyze --json reports admission.admittedCostUsd: the expected cost plus the 10% margin, or
    the worst case when that is lower. Admission compares it with the cap, so it is the smallest
    --max-cost that admits the analysis (row 2). (#1681)

Changed

  • The OpenAI computer-use participant is now asked for its closing report and impressions within
    the limits its reply is checked against: a summary of up to 4,000 characters, up to 8 friction
    reports of up to 2,000 characters each, and up to 6 impressions of up to 500 characters each.
    Its request schema had no limits before, so a longer reply was asked for and then dropped as
    invalid. The Codex participant was already sent these limits. The Observer reads a recorded
    closing account with more than 8 friction reports or more than 6 impressions as unreadable; no
    humanish version records more (row 1). (#1680)
  • A feedback draft for an amended finding opens with the sentence humanish review and the
    Observer show: "Corrected in human review. The reviewer's claim replaces the original headline
    and account." It had its own wording of that sentence (row 9). (#1678)
  • An HTML export and a feedback draft give the same share-safety reason for reviewer notes: "The
    reviewer notes being shared match secret, token or local-path patterns." for a secret-shaped
    value, and "The reviewer notes being shared hold encoded text the scan cannot read." for text
    the scan cannot read (row 10). (#1682)
  • In the repository's benchmark (pnpm bench, which the package does not ship), an analysis's
    automatic cap is the dry run's admittedCostUsd, and the budget gate counts each analysis at its
    worst case; an analysis dispatched with no reported usage is charged its worst case. The
    benchmark README describes the expected-cost admission. (#1676, #1681)
  • Development dependencies: oxfmt 0.72, oxlint 1.87, knip 6.40, vite 8.3.3 (the Observer bundle is
    built with it), jsdom 30.1.2, axe-core 4.14, and next 16.4 and fumadocs 16.16 for the site; fast-check
    4.10.2 is new for the scrub's property tests. The package's own dependencies are unchanged.
    (#1671, #1674)

Fixes

  • The TUI's study screen shows the analysis budget line that humanish run and study check
    print: the expected cost range for the study's participants and the rule that refuses an
    analysis whose expected cost plus 10% is over the cap. It still showed the wording from before
    0.114.0, "admission estimate limit", with no expected cost (row 7). (#1678)
  • The warning for a participant with no persona background names study show the way the
    install runs humanish, such as npx humanish study show <id> --json where humanish is a dev
    dependency. It printed humanish study show, which reaches a stale global install or nothing
    (row 8). (#1678)
  • Telemetry reads CI=false and HUMANISH_STUDY_PARTICIPANT=false as unset, as the update check
    and is-in-ci already did. It counted any value other than empty or 0 as set, so CI=false
    marked an event as coming from CI (row 5). (#1678)
  • humanish keys set constructor and humanish keys set __proto__ now refuse the name with the
    list of provider keys the store holds. Both names matched a built-in object property in the alias
    table and printed Not a valid env name: function Object() { [native code] } (row 6). (#1678)
  • humanish analyze names its refusal command the way the rest of the CLI does, so a project
    that installs humanish as a dev dependency is told npx humanish analyze --run <id> --max-cost <n>
    (row 3). (#1681)
  • humanish export --local-only keeps a blocked run blocked when the export's own analysis
    check fails. It wrote local_only into the result and the file's share record, a better grade
    than verify gave the run. Export now applies verify's rule: a failed check can only move a
    run toward blocked (row 4). (#1682)
  • When humanish notes --add refuses a run with no readable run.json, the verify command it
    suggests names this install's invocation, such as npx humanish in a project (row 11). (#1682)
  • Adding reviewer notes at the same time, from the Observer or humanish notes --add, no longer
    refuses one of them. A note was published with a hard link, so for a moment it had two links,
    and another add's run directory check refused the run as having "no run.json humanish can read
    safely". A note is now created under its own name, and a refusal from that check names what it
    found (row 11). (#1682, closes #1679)
  • The known-value scrub finds a value inside a bracketed span such as [REDACTED_373433393231]
    (743921 in hex). It skipped every [REDACTED_...] span; it now keeps only the markers
    humanish writes ([REDACTED_SECRET], [REDACTED_LOCAL_PATH], [REDACTED_RUNTIME_PATH],
    [REDACTED_PROMPT_TEXT], [REDACTED_LOBBY_CODE]), and only text wholly inside one. It still
    returns the decoded text with the values replaced, and replaces the whole text when another
    reading of that result still holds a value: decoded once more, with percent escapes as UTF-8,
    with transfer escapes expanded, or with Latin-1 characters read as UTF-8 bytes. That covers a
    value encoded twice and a value with one of two non-ASCII characters percent-encoded, which came
    out encoded once or as éÃ\u00a0 for éà. It also finds a value such as xéz€ written as
    xéz€. A value that overlaps itself is found in time linear in the text; 1 MiB of one letter
    with a registered value half as long took 93 s (row 12). (#1674, part of #1646)
  • verify and bundle export read text in the same ways everywhere: as written, decoded, with
    percent escapes as UTF-8, with transfer escapes expanded and with Latin-1 characters read as
    UTF-8 bytes. Text decoded from base64 two levels deep was read only as written and decoded, and
    percent escapes that a \u0025 wrote were not read as UTF-8 (row 12). (#1674, part of #1646)
  • &constructor; and the other names only a JavaScript object's prototype defines stay as written
    in verify and the known-value scrub. They decoded to a JavaScript function's source text, so a
    made-up text such as /home/&constructor; graded as holding a local path; it now reads as clean
    (row 12). (#1674, part of #1646)

Known gaps

  • The contained-file helpers have three races that need a second local process, running as the
    same user, to swap a directory in the run for a symlink or FIFO between two filesystem calls:
    temporary-file cleanup unlinks by path, contained reads have no byte cap and can block on a
    FIFO, and the notes listing can follow a swapped notes/. Forged web requests reach none of
    them. (#1669)
  • A study whose explicit review.analysis is refused for its cost still ends with live run failed and exit 2 while run.json's outcome.ok is true. (#1673)
  • The 10-08 audit's "worth exploring" findings are open: one run clock for notes and the Observer,
    persona-background warnings rebuilt from the planner's list, the provider key list in five
    places, the Lima VM size rule in two, the Observer's analysis limits copied with no parity test,
    the closing ask's five participant_end branches, and files over the 700-line guideline. (#1677)
  • From #1646, after #1674: the scrub keeps no position mapping, accepts a registered value that is
    part of a marker, does not search short unpadded base64 under 8 characters, refuses the scope of
    a registered value of 32,768 characters, and does not read two further decodings at a marker
    edge. It shows a value holding an escape in its decoded spelling (pass%41word as passAword);
    the transient scrub's literal pass removes it. (#1646)
  • zod's .max counts UTF-16 code units and JSON Schema maxLength counts code points, so a
    closing report with characters outside the BMP can fit the provider's limit and fail the parse.
  • A closing report request after a stop rule is not listed in conversation.requests; its
    tokens are the last entry of tokenUsage.turns. The impressions request is listed with
    kind: impressions.
  • A reviewer note interrupted mid-write leaves an empty or short file that readers skip as
    unreadable.
  • Design findings cannot say that text or a click target is too small by measure. (#1638)
  • The impressions ask describes its kinds in visual terms, which can steer a participant toward
    visual critique whatever its persona. (#1654)
  • Nothing Mac-specific has run on a Mac: the Lima VM sizing, runtime setup --memory --cpus and
    the local capacity refusal from 0.113.0.

Package

humanish 0.115.0 packs to 2.5 MB, 7.7 MB unpacked, 952 files (npm pack on release/0.115.0 at
8b7620e, shasum 433d9384fde6d9ff88b61ece7cefedbbbc422b78). Against 0.114.0 it adds
dist/actors/closing-report, dist/actors/closing-report-limits, dist/analysis/finding-lead,
dist/cli/user-config and dist/run/note-shape (.js and .d.ts each) and drops nothing. The
package exports only ., with the same 13 values and 31 types as 0.114.0 (pnpm api:proof). Its
dependencies are the same as 0.114.0's. It needs Node 22.19 or newer, as before, and @e2b/desktop
2.3.2 or newer as an optional peer.

The published tarball's shasum is 433d9384fde6d9ff88b61ece7cefedbbbc422b78 and its integrity
sha512-mX1PggCFkWTEIy3klO+lrYAZb6mJCV4LwE4Tdj9Fqn/q05orrBGDKXOWTcjMRjYKufwi3Db9g5f+WvDQcVxD2Q==,
the same as npm pack on the release commit, and it carries a signed provenance statement from
the publish workflow.

Checked before release

A live smoke on a tarball packed from main 0b3df52 in fresh projects under /tmp with a fresh HOME
and npm cache, DO_NOT_TRACK=1, and @e2b/desktop 2.4.0 beside humanish. Keys came from the
user key store through key discovery, with XDG_CONFIG_HOME pointing at it; no --dotenv was
passed. The store's OpenAI organization accepts server-side conversation state, so every
participant ran threaded. Controls ran the same CLI commands on npm 0.114.0 against copies of
the main runs. Priced spend: $3.97 for the two live runs, $2.58 of it in analysis, then $6.20
for the benchmark, $4.33 of it in analysis, and $0.26 for the dogfood run.

  • After publish, the post-publish check against the npm registry (TMPDIR=/tmp verify.sh 0.115.0, with the registry and unpkg rows): 104 of 104 passed.
  • init --yes and doctor, and doctor --study try-live: the same output on main and npm
    0.114.0.
  • try-live, as init writes it: reached the goal in 10 participant turns, input growing every
    turn from 2,888 to 18,131 tokens, then 18,507 for the impressions request. $0.25 run. Its
    automatic analysis was admitted at an expected $1.03 (admittedCostUsd $1.13, worst case $2.02,
    cap $3) and billed $0.78. It ended on its own, so it was sent the impressions request and gave 6
    impressions (unclear, unfinished, unclear, unfinished, liked, liked; 105 to 166 characters).
  • A computer-use fan-out, 4 participants on drawDB, all passed. Newcomer and volunteer had a
    stopWhen rule for drawDB's first table and stopped after 4 turns, so each was sent the closing
    report request: summaries of 267 and 241 characters, 2 friction reports each (142 to 174
    characters) and 4 impressions each. Power-user and treasurer ended on their own after 17 and 15
    turns and gave 6 impressions each. Every report and impression is within the limits and parses
    with the package's closingReportSchema and participantImpressionsSchema; the request schema
    the package sends carries maxLength 4,000, 2,000 and 500 and maxItems 8 and 6, where npm
    0.114.0's carries none. Input grew every turn. $1.13 run. Its automatic analysis was admitted at
    an expected $2.04 (admittedCostUsd $2.25, worst case $2.88) and billed $1.80; 7 findings and 6
    design findings (partial: the 40-capture limit).
  • analyze --dry-run --max-cost 3 --json on both live runs: admittedCostUsd 1.131406 and
    2.24649; npm 0.114.0 on copies gives the same expected and worst-case costs and no
    admittedCostUsd. analyze --max-cost 0.01: exit 2, npx humanish analyze --run <id> --max-cost 2 on main, a bare humanish analyze on npm 0.114.0.
  • Reviewer notes added at the same time on the try-live run: 5 rounds of three notes --add
    processes started together and one POST /api/notes from the served Observer page sent the
    moment the first new note file appeared. All 15 CLI adds exited 0, all 5 POSTs answered 201, and
    notes --json listed all 20, each once; each round's notes were created within 13 to 24 ms. npm
    0.114.0 on a copy also kept all 20, so the smoke did not reproduce #1679; its regression test
    runs in release:check. The Observer's note form then added a 21st note, forged requests were
    refused as in 0.114.0, and export --local-only carried 21 notes with no write token.
  • export --local-only on a copy of the try-live run with a synthetic key planted in review.md:
    verify exit 2, blocked; export exit 2, HUMANISH_EXPORT_VERIFY_FAILED, share safety
    blocked, no file, on main and npm 0.114.0 alike.
  • keys set constructor and __proto__ with a value on stdin against an empty scratch config:
    the provider-key refusal on main, "Not a valid env name: ..." on npm 0.114.0; nothing written.
  • The TUI's try-live study screen under a pseudo-terminal sized on that pty only, at 100 and 45
    columns: the expected cost line on main, "separate $3 admission estimate limit" on npm 0.114.0.
  • Telemetry with CI=false, read from the HUMANISH_TELEMETRY_DEBUG=1 line with a scratch
    telemetry.json and nothing sent: ci false on main, true on npm 0.114.0.
  • After every run, reclaim --check exited 0 with clean. Raw sandbox ids appear only in
    sandbox-receipts.ndjson, E2B listed 0 tagged sandboxes after the runs, and no key value from
    the store appears in any run or kit file, as written, base64 or hex.

On the release branch:

  • pnpm release:check on the release head 062e97d, which adds the benchmark result: exit 0 in
    3.4 min, 483 test files passed and 10 skipped, 7,668 tests passed and 12 skipped; TUI 158
    tests; lint 439 warnings at its cap of 439; api:proof "13 values and 31 types match";
    public-surface scan of 2,973 text files and 523 binary assets; npm pack --dry-run 952 files
    with shasum 433d9384fde6d9ff88b61ece7cefedbbbc422b78. On the release commit 8b7620e before it:
    exit 0 in 3.5 min with the same test counts, a scan of 2,971 text files and the same shasum.
    pnpm docs:generate there wrote 0.115.0 into cli.mdx.
  • pnpm release:dogfood on 8b7620e (the script run with Node, keys read from the user store by
    Node's --env-file into its environment, and a temporary HOME): "release:dogfood ok", verdict
    blocked (blocked_approval), no-spend satisfied, $0.26. The participant ran init --yes and
    the free four-participant dry run on the installed 0.115.0 and reported "All 16 evidence checks
    passed; status was share_ready." It stopped at the credentials the gate withholds. verify on
    the run: share_ready, 16 of 16; reclaim --check clean.
  • The post-publish check on the release tarball from 8b7620e (TMPDIR=/tmp, HUMANISH_SPEC the
    tarball): 98 of 98 passed. Its 6 new rows cover admittedCostUsd admitting at its value and
    refusing 0.000001 below it, with the refusal naming npx humanish@0.115.0 analyze (#1681);
    keys set constructor and __proto__ refused with the provider-key list and nothing written,
    telemetry's ci false with CI=false from the debug line, the TUI's try-live screen at a
    100x40 pseudo-terminal showing the expected cost line, and the no-background warning naming npx humanish@0.115.0 study show (#1678); and export --local-only of a blocked dry run refused
    and still blocked (#1682). The same script on npm 0.114.0 passed 93 and failed exactly those
    rows other than the export row, which reads the same on both. A first pass on the release
    tarball passed 97 of 98: 0.114.0's admission row grepped for a bare humanish analyze, which
    0.115.0 now prints with the install's prefix (row 3); the grep was widened and both passes ran
    again.
  • The consumer check: the 0.114.0 check's library and CLI consumer, plus nine CLI lines for this
    release. Typecheck 0 errors on both versions. Every output line from the 0.114.0 check is the
    same on npm 0.114.0 and the release tarball, except the persona-background warning, which names
    npx humanish study show on 0.115.0 (row 8). The new lines: admittedCostUsd present and
    admitting at its value and refusing 0.000001 below it, the refusal's npx humanish analyze, the
    keys set refusals, export --local-only of a blocked dry run refused the same on both, and
    telemetry ci false with CI=false.
  • pnpm bench on 8b7620e with its defaults (3 runs per arm, openai-computer-use, neutral
    mission), committed in 062e97d: report recall 10/15, analysis recall 8/15, 0 invented on either
    arm, $6.20 estimated. 0.114.0 scored 11/15 and 9/15, and 11/15 for analysis read by hand. Two
    planted analysis lines the rubric left unresolved are D5 ("Save left editing open; Enter finished
    the edit" and "New wording did not survive the rename attempts"), which puts analysis recall at
    10/15 read by hand. Each analysis's cap was the dry run's admittedCostUsd ($1.14 to $1.33
    against bills of $0.59 to $1.01), the gate counted each analysis at its worst case, and no
    analysis was refused or skipped, so #1681 changed no number here. Every participant ended on its
    own and gave 5 or 6 impressions within #1680's limits, so no reply met a limit either. Against
    0.114.0's table, the first planted report missed D1 (its narration names Clear completed; its
    report states no D1 claim) and the second planted analysis missed D4; every other defect scored
    the same. No planted participant typed more than 30 characters in one action, so none met D2.
    Every participant ran threaded; E2B listed 0 tagged sandboxes after it.

Not verified

  • macOS and Windows. Every check ran on Linux x64 with Node 24.12.
  • A live run on the published package: the live smoke ran on a main tarball, and the post-publish
    rows are offline dry runs, refusals and local scripted runs.
  • #1682's export rule through the CLI: a planted value fails verify first, so no fixture here
    reached the path where verify passes its checks, grades the run blocked, and the export's
    analysis check fails. tests/feedback/export.test.ts covers it in release:check.
  • #1679 on 0.114.0: the concurrent-notes smoke passed on both versions.
  • A reply over the closing limits: every participant stayed within them, so no live reply tested
    what the provider does at a limit.
  • A live Codex participant; its final schema is generated from the same zod schema and its test
    compares it with the old literal keyword for keyword.
  • The impressions and closing report records in explicit-context mode: the store's OpenAI
    organization accepts server-side state, so every live participant ran threaded.
  • The known-value scrub on live evidence beyond the smoke's runs: its property tests and example
    tests run in release:check.
  • The local VM, email capture and receiving, local-agent participants, terminal participants and
    the shared-world route: not run live on 0.115.0.