Skip to content

v2.2.2

Choose a tag to compare

@kinncj kinncj released this 20 Oct 12:42
· 37 commits to main since this release

Maple Harness v2.2.2

Highlights

  • The setup page's numbers agree everywhere. For the same time and filter, the Overview's tiles, the Savings
    card, the charts, the filtered cards, the CLI, the CSV export and the DuckDB views give the same requests, AI units,
    tokens and estimates, and today's savings on GitHub Copilot's GPT models are now valued. See
    Reading the numbers.
  • One clean final answer after Maple Harness checks an answer. When a check sends an answer back, the terminal,
    the remote app, --rpc, serve and --prompt all end on the model's corrected answer, not on an argument with
    the check.
  • Days start at your local midnight. "Today" and the whole-day history ranges (--last 7d, last=7d, the
    page's range buttons) now count from midnight in the proxy's time zone, so "1 day" and "today" agree (see
    Upgrading).
  • The Overview prints, or saves as a PDF, with its charts as drawn.
  • No request is lost or counted twice after a crash, and a request for an unknown model is now recorded.

Added

  • A "Today" choice on the setup page's range buttons (since midnight, local time).
  • harness.max_auto_continues, the most automatic continues one message gets (50 unless set; see Fixed).
  • New JSON fields: without in maple-proxy savings --json and /api/savings/summary (what you spent plus the
    net avoided), session_* and resumed in --json-summary (a resumed session's totals), and interrupted there
    for a run stopped by a timeout or a signal.

Changed

  • The harness's own lines are short and start with », so they never read as the model's: » check: … when an
    answer is sent back, » checked: … when the corrected answer is in, » continuing: … when the model is asked to
    carry on. The answer a check sent back is folded to one line (click it, or press ctrl+o, to read it again).
  • yolo mode is shown in the warning colour and says "tools run without asking".

Fixed

The setup page and the proxy's numbers

  • The setup page's numbers agree with each other and with the CLI. For the same time and filter, the Overview's
    tiles, the Savings card, the charts, the filtered cards, maple-proxy usage query, maple-proxy savings, the CSV
    export and the DuckDB views now give the same requests, AI units, tokens and estimates:
    • "Today" is since local midnight everywhere; a filtered tile used the UTC day and counted last evening too.
    • Requests today now counts answers from the response cache, as the usage log does.
    • AI units today and "You spent ... today" are the same sum: a failed request that reported AI units is counted
      in both, and the Savings card now refreshes as requests finish instead of every 30 seconds.
    • The history charts are drawn again when new requests arrive, and every chart and filtered card says when it was
      read, so two cards for one filter no longer show different counts without saying why.
    • After a restart, today's Savings figures are the ones the usage log recorded, even when the saved figures were
      older than the cache view's or missing.
    • Every AI-unit figure shows the same three decimals on the page, in the CLI and in the API: totals and
      estimates are added up exactly, and a half rounds away from zero everywhere (the page showed 21.262 where
      maple-proxy usage query said 21.263).
    • The Savings headline shows the estimate, not a cap. It was capped at what was spent, so on a day whose prompt
      cache avoided more than was spent the headline showed a smaller figure than the line below it. It now
      says what the prompt cache and trimming avoided (estimated), less Maple's own costs (now including its
      cache-warming pings), beside what you spent and what you would have spent without them (spent plus the net).
      The filtered Savings card and maple-proxy savings follow the same rule, and the filtered card shows per model
      how many requests had an estimate.
    • With a filter, every Overview tile follows it. "Failed, last hour" ignored a model filter while "Requests
      today" followed it; the line above the tiles now says which tiles are live.
    • "Failed, last hour" survives a restart. It read 0 after the proxy restarted (the live view starts empty);
      it now reads the usage log.
      See Reading the numbers.
  • History ranges start at local midnight. last=7d (and --last 7d, and the page's range buttons) counted UTC
    days, so "1 day" and "today" disagreed by your time zone's offset, and right after local midnight "today" was 0
    while "1 day" still held yesterday. Whole days now start at midnight in the proxy's time zone. Custom ranges
    given as days stay UTC days, as before, and say so.
  • Cache reads on GitHub Copilot's GPT models (which report no cache writes) are now valued, as an estimate, on
    every view. They showed as "not valued yet" forever, so today showed nothing saved while earlier days (another
    model) did. A model still without an estimate is named under Not valued yet, with the reason.
  • The Cache card's "≈ AI units" for cache reads and writes divided each request's whole cost by its cached tokens,
    overstating what reads cost; it now uses the prices the Savings card uses.
  • Cache warming now values the cache the same way the Savings card does, instead of with the Cache card's
    overstated prices.
  • The Overview prints. Printing it, or saving it as a PDF, showed blank charts; charts now print as drawn, in the
    light theme, without the buttons.
  • The list of requests as they finish shows the date of a request from before midnight.
  • No request is lost or counted twice after a crash. A request that started before the proxy saved its figures
    and finished after was missing from the tiles and the Savings card after a restart.
  • A request for an unknown model is recorded. It answered 404 but left no usage line, so it was on no card.
  • Fallbacks are counted whether or not they answered. A fallback that also failed counted as 0 fallbacks; the
    outcome chart's series now says "fallback, answered".
  • A model GitHub Copilot will not serve to Maple (it answers "not available for integrator" though its model
    list offers it) now fails with the reason "model not offered to this client" and is marked in the model list,
    instead of a bare "status 400" on every attempt.

Maple Harness

  • Corrections from Maple Harness's own checks no longer turn into an argument with the model. When a model's
    answer reported a command's result though the command never ran, the harness's note could be answered back
    ("Confirmed. I ran it now…", "It's genuinely finished."), and after the model had run the command the harness
    then said it had "described a step without taking it". The reply after a check that ran is now taken as the
    answer, the notes ask the model for its corrected final answer instead of a reply, and the screen ends on that
    answer. See Troubleshooting.
  • Every client ends on the same answer after a check. The remote app on your phone or browser now folds an
    answer a check sent back and ends on the corrected one, and no longer shows an empty bubble for a reply that was
    only tool calls; --rpc, serve and a terminal session shared with remote devices get that same final answer,
    not the model's reply to the check, and the output of maple-harness --prompt ends on it (printed again after
    » final answer: on the error stream when it was not the last answer printed).
  • The final summary is shown once. The last answer of a turn could appear twice in the terminal.
  • A turn that ends on a stop message shows it, in the terminal and the remote app, also when the model had
    already answered earlier in the turn. The message a turn stops on after the same problem twice ("Stopped: the
    same problem was reported twice…") now says only what is wrong, without the instructions meant for the model.
  • A command run in an earlier message counts. A change report asked for in a later message ("now give me the
    report") was told its build or lint "was not run in this session" when an earlier message had run it.
  • The "continuing" nudge no longer follows an answer that was complete: not after a question put to you ("Which
    database should we use? …"), not after "lint passes" or "the tests pass", not because of a build's output quoted
    in the answer, and not after a step the model already took.
  • A finished task no longer runs on by itself. A model that ended each step with a complete change report
    offering more work ("If needed, next step is…") was asked to carry on after every report and did the offer, step
    after step, on work nobody asked for, until a timeout stopped it. A change report with its verification section
    now ends the task; asking the model to carry on now says to continue only the task you asked for; and one message
    gets at most 50 automatic continues (harness.max_auto_continues), and none after the model, asked to carry on,
    describes the next step again instead of taking it. Then the turn stops with "Stopped: the model kept describing
    steps without taking them", with the model's last report as the answer; type "continue" to carry on.
    A verification line such as "Build: N/A → no build step" is no longer sent back as a command that did not run,
    and python3 -m unittest, ruff, mypy, eslint and similar now count as checks.
  • A run stopped by a timeout or Ctrl-C writes its --json-summary, marked "interrupted": "terminated" (or
    "interrupt"), with what it had spent until then; the process still ends with that signal, as before.
  • --json-summary reports the run. On a resumed session it reported the session's running total as if it were
    the run's; the run's own figures are now in its fields, as documented, and the session's totals in new session_*
    fields. See Subagents.

Upgrading

  • Whole-day ranges start at local midnight: --last 7d (and the other --last Nd), last=Nd in the setup
    page's address and its range buttons now count whole days from midnight in the proxy's time zone, not from a UTC
    midnight, so --last 1d is "today" as the page counts it. For the old window, give UTC days:
    --since 2026-10-14, or the page's Custom range.
  • maple-proxy savings --json and /api/savings/summary: saved and net are no longer capped at spent,
    and costs now includes the cache-warming pings' AI units; the new without is spent + net.
  • --json-summary on a resumed session, fixed: its fields now describe the run, as documented; a script that
    wants the session's running total reads the new session_* fields.

Known limitations

  • Printing the Overview was tested in Chromium only, not in Safari or on macOS.
  • The checks against real GitHub Copilot were run on Linux. macOS and Windows were not run for this release.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.