Skip to content

0.143.0: Analysis costs about one seventh as much: gpt-6.1-sol at medium effort is the default

Choose a tag to compare

@danielgwilson danielgwilson released this 22 Oct 03:48
· 78 commits to main since this release
936f29b

humanish 0.143.0 analyzes studies with gpt-6.1-sol at medium reasoning effort by default, in
place of gpt-6-astra at high. On the same recordings it found what gpt-6-astra found, invented
nothing, and cost about one seventh as much: $0.06 against $0.41 per analysis on the Taskly
benchmark. A study can set the analysis effort with review.analysis.reasoningEffort, and one
that names its model keeps it at high effort with an unchanged configuration hash.

Added

  • Study analysis takes a reasoning effort: review.analysis.reasoningEffort or humanish analyze --reasoning-effort <level>, sent as OpenAI's reasoning.effort or Gemini's thinking level. Each
    model takes only the levels its provider documents, and others are refused before any request.
    An analysis that names a model and declares no effort runs at high and keeps its
    configuration hash. pnpm bench --analysis-models entries take an @effort suffix, such as
    gpt-6-luna@medium.

Changed

  • The default OpenAI analyst is gpt-6.1-sol at medium reasoning effort, in place of
    gpt-6-astra at high, for automatic analysis without review.analysis.model and for humanish analyze without --model. On the same recordings it matched gpt-6-astra on Taskly recall
    (28/60 each, 0 invented) and on the movie-game findings at about one seventh of the cost ($0.06
    against $0.41 per Taskly analysis)
    (analysis models on fixed recordings).
    A study that names a model but no effort runs it at high, so its configuration hash is
    unchanged. To keep the old analyst, set review.analysis: {model: gpt-6-astra, maxCostUsd: 10}.

  • --remote copies a run's screenshots and recordings back while the run goes on, every 30
    seconds, so the copy when it ends is only what is left; the final copy skips any file already
    here with the same digest, which also covers observe --run reconnecting after an earlier copy.

Upgrading

Studies that name review.analysis.model are unchanged. A study that relied on the default now runs gpt-6.1-sol at medium; to keep the previous analyst, set review.analysis: {model: gpt-6-astra, maxCostUsd: 10}. The efficacy benchmark for this release is the analysis-model comparison linked above, which holds the recordings fixed and varies the analyst.