Skip to content

Releases

v2.6.2

Choose a tag to compare

@kinncj kinncj released this 10 Nov 15:22

Maple Harness v2.6.2

Fixed

  • Copilot Fusion Max (and Auto) no longer cut a large file off halfway. Fusion and Copilot Auto listed the
    smallest output limit among the models Copilot may pick, so a model that can write far more (Claude Opus, through
    Copilot Fusion Max) stopped at 8,192 tokens in the middle of writing a file. They now list the largest, and each
    request is capped at the limit of the model Copilot chose.
  • A tool call cut off in the middle no longer breaks the session. Maple Harness kept the half-written call, and
    every later request failed (the proxy could not encode it for Copilot, and called that "unreachable"), as did
    saving the session. Such a call is now refused as cut off, the model is told to write in smaller pieces, and the
    conversation stays whole; the proxy also sends such arguments as {} instead of failing.
  • "The proxy is not answering" is only said when it is not. A proxy that answered with an error (for example
    that a local model server behind it was down) was reported as not answering, with advice to start it.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.

v2.6.1

Choose a tag to compare

@kinncj kinncj released this 10 Nov 14:55

Maple Harness v2.6.1

Fixed

  • Maple Harness no longer sends finished work back as "a step described but not taken". A reply that reports
    its subject is finished ("The simulation is finished, in a single file …") was carried on as if it had only
    described the next step, and each extra round cost a full Copilot Fusion round with its review, before the run
    ended marked "not finished". Such a report now ends the turn, unless it also says what it will do next.
  • Copilot Fusion finds its tier even when Copilot renames it. High and Max are found by the id Maple knows, or
    by their level in Copilot's catalog, and the plan is asked for under the id the catalog uses.
  • A Copilot Fusion request that cannot start says why. Copilot's tiers not being readable, or a tier the
    account does not have, are now reported as such ("not offered on this account …"), not as "unreachable", and the
    proxy's "failed before any output" line includes the error's text, so the cause is in the log.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.

v2.6.0

Choose a tag to compare

@kinncj kinncj released this 10 Nov 05:40

Maple Harness v2.6.0

Highlights

  • A receipt for every Copilot Fusion turn. When a turn ends, Maple Harness tells what the work, the reviews and
    any answers not used were billed (as Copilot stated), about how many premium requests the turn took (estimated
    from how Copilot counts them) and which files the review read. The setup page sums up each tier: how its turns
    ended, the reviews' share of what it was billed, and its premium requests. See
    Copilot Fusion.
  • Steps that produced nothing are counted. The end-of-session summary says how many model turns produced nothing
    (every call failed, or it only re-read what it had already read) and what they were billed. Measured, never a
    claim of what could have been saved.
  • Fewer wasted calls. A Copilot Fusion review that asks for a file it was already given gets it straight back,
    with no round trip to your tool; the agent no longer searches the project graph of a new, empty project; and a
    project that imports only Node's own modules is no longer told it needs a package.json.

Added

  • harness.fusion_turn_budget: AI units a Copilot Fusion High or Max turn may be billed before the rest of your turn
    runs on Copilot Fusion Standard, said when it happens (off by default).
  • Copilot Fusion turns say when one of Maple Harness's own checks asked for another round.
  • X-Maple-Fusion lists the request's other calls with their AI units and who Copilot was told started each, and
    who it was told started the request's own call.
  • maple-proxy usage query groups by initiator.

Fixed

  • Copilot Fusion's story shows the review's score when the review read files. Maple Harness said "the review
    accepted the answer" without the score (for example "review 5/5") when the review had read files through your
    tool; the score is now on the response (X-Maple-Fusion) and in the story.
  • Maple Harness's build check no longer takes Node's own modules for packages. A project whose files import only
    node:test, node:fs, fs, path and the like (or a URL) was told it could not build without a package.json,
    and the agent went on to write one and install nothing. Only real packages count now.
  • Copilot Fusion's review reads an ordinary source file in one call. Each read it is given is now up to 40 KiB
    (160 KiB in all per review), where a 16 KiB limit had it read a 16 KB page twice.
  • The usage log records who Copilot was told started a review that read files (initiator: user, as Copilot's
    own apps mark reviews), where it said agent.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.

v2.5.0

Choose a tag to compare

@kinncj kinncj released this 10 Nov 04:18

Maple Harness v2.5.0

Highlights

  • Copilot Fusion's review reads your files. The review in Copilot Fusion High and Max may now use your tool's
    read-only tools (reading, searching, listing), at most 12 calls a review, to check the answer against the files
    themselves, as Copilot CLI's review does. On by default; turn it off per sign-in with fusion_review_reads: off
    or the Let the review read files box on the setup page.
    The calls come to your tool like any other step, under its usual permission rules, and the models doing the task
    never see them. Maple Harness marks them as the review's and lets them only read, only inside the project. See
    Copilot Fusion.
  • Copilot Fusion streams. Every step of a Single turn, and every step that calls a tool, now streams as it comes.
    Only a final answer that a review will score is held until it has been scored.

Added

  • providers[].fusion_review_reads (on by default; off keeps reviews as in 2.4).
  • Copilot Fusion works from Claude Code, OpenCode, pi, GitHub Copilot CLI and Crush through the proxy (checked with
    each); see Connect a tool.

Fixed

  • Copilot Fusion High and Max no longer fail a task when Copilot cannot plan it. If Copilot refuses to plan a task,
    or answers in a way Maple cannot carry out, the task goes through Copilot Auto instead and is billed as Copilot
    Auto. Maple Harness tells you, and the usage log (fusion_phase and fusion_outcome: fallback), the setup page
    and X-Maple-Router say so. Maple asks Copilot for a plan again after ten minutes.

Known limitations

  • Use Maple Harness 2.5.0 with a 2.5.0 proxy for Copilot Fusion. An older harness does not tell the story of a turn
    whose last answer was streamed, and may stop a turn when a review reads the same file twice.
  • The review reads through your tool only with tools named as Maple Harness, Claude Code and opencode name their
    read-only tools; any other tool gets the review as before.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.

v2.4.0

Choose a tag to compare

@kinncj kinncj released this 10 Nov 01:10

Maple Harness v2.4.0

Highlights

  • Copilot Fusion works through Maple. Each Copilot account lists github-copilot/fusion (Standard, the same as
    Copilot Auto), fusion-high and fusion-max when the account has them. Maple carries out the plan Copilot gives
    for each task: one model, or a model whose final answer is reviewed, with a stronger model taking over when the
    review is not satisfied, or a draft that is critiqued and revised. It works in Maple Harness and in every tool that
    uses the proxy. A tier your account does not have is listed with GitHub's reason. See
    Copilot Fusion.
  • Every model that worked on a task, and what it cost. Each call of a Fusion turn (the reviews and the answers
    not used included) is a usage line of its own with its part in the turn, the review's score and how the turn
    ended. Maple Harness shows the turn's story when it ends and the step on its status bar, the setup page lists the
    calls on the Savings card, and X-Maple-Fusion says it on each response.
  • Copilot requests are valued at what Copilot charged. The proxy now reads the price per token Copilot states on
    every answer, and the prices and promotions Copilot publishes for your account, instead of assuming Copilot Auto
    costs 0.9 times the model by name. Each request records its list cost and the discount taken.
  • Discounts you set. providers[].discount (or the Discount fields of a Copilot provider on the setup page) sets
    the rate your organisation pays, for every request, per model or per Copilot Auto and Fusion tier. What each
    request costs at your rate is shown beside what Copilot charged, never instead of it, and budgets keep counting
    what was charged. See Discounts.

Added

  • Usage schema 8: list_ai_units, discount_rate, discount_source, configured_ai_units,
    configured_discount_rate, and fusion_id, fusion_tier, fusion_pattern, fusion_rule, fusion_phase,
    fusion_verdict, fusion_outcome, and initiator (who the provider was told started the request). Older lines
    read as before. price_source gains copilot_stated and
    copilot_published.
  • maple-proxy usage query groups by fusion_tier, fusion_pattern, fusion_phase and fusion_outcome, and has
    listed_ai_units, listed_charged_ai_units, configured_ai_units and configured_charged_ai_units.
  • The Savings card shows Copilot's own discount on a line of its own (list price less what was charged), never as
    one of Maple's savings, and the cost at your configured discount beside what was charged.
  • When a Copilot promotion has three days or less to run, the setup page and Maple Harness's status line say so
    (X-Maple-Promotion), since each request will cost more after it ends.

Fixed

  • Copilot Auto's discount during a promotion. From 28 October 2026 Copilot bills Copilot Auto at 0.7 times the
    model by name (its Fusion launch promotion), not 0.9. Savings, the cost preview and the Cache card's estimates now
    use the rate Copilot states, and go back to the usual rate when the promotion ends by itself.
  • Requests the agent loop sends are marked as such. Requests to Copilot now say whether the person started them
    or the agent loop sent them after a tool's result, as Copilot's own apps do. Before, none said, which Copilot may
    count against premium requests as new turns.
  • Maple Harness's own requests are not counted as yours. Its checks after an answer and its summaries are sent
    marked as the agent's (X-Maple-Initiator: agent), so Copilot does not take them for new turns you typed, and
    Copilot Auto and Copilot Fusion keep them in the task they belong to. The mark is honoured only from a tool on the
    proxy's own machine or a chained Maple proxy: behind a shared proxy, a member's mark is ignored.
  • The bash sandbox on Linux works when a hidden location is a symbolic link. If one of the locations it hides
    (such as docker.sock or ~/.netrc) was a link, every contained command failed to start. A link is now hidden
    where it leads, and one that leads nowhere is left alone. A hidden link that leads through or into a folder
    commands may write (the project, ~/.cache) is refused with a message naming it, since a command could change
    what it holds. A hide entry inside the project now stays hidden.
  • Security updates to two compression libraries used by the Parquet export, the hub roll-up and the response
    cache (GO-2026-6888, GO-2026-6865), and to Go's network library (GO-2026-6866, GO-2026-6860; not reached by
    Maple's code, updated anyway).
  • Copilot Auto keeps a task on one model. It now asks Copilot once for each task you type, the way Copilot's own
    apps do, instead of once for every step of it, and keeps the task's later steps on that model. Accounts and hosts
    that do not offer this keep the previous behaviour.

Known limitations

  • A Copilot Fusion review sees the task's commands and their results, not your files. A model that takes over
    continues from the files as they are: nothing a draft changed is undone.
  • In Copilot Fusion High and Max, each step's answer is sent whole rather than streamed, so the review can run before
    a final answer is passed on.
  • Copilot Fusion relies on a part of Copilot that GitHub does not document, as Copilot Auto does; it may change
    without notice.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.

v2.3.1

Choose a tag to compare

@kinncj kinncj released this 23 Oct 13:29

Maple Harness v2.3.1

Fixed

  • A Redis that answers as a read-only replica no longer looks healthy. After a Redis failover, an address still
    pointing at the old primary (now a read-only replica) left every proxy ready while its writes failed: budgets were
    refused and the setup page could not save. Proxies now say not ready there, in storage modes central and redis,
    and in storage mode redis the setup page says why and a proxy refuses to start on it. Usage was never lost: it
    waited and was written once the address was right.
  • Tested on Redis OSS and through a failover. Storage modes central and redis now also run against Redis 8.10.2,
    and a planned failover (standby promoted, address moved) loses no usage and keeps budgets exact.

Known limitations

  • Redis copies writes to a standby a moment late, so a crash failover can lose the last moment of counts. Managed
    Redis services and the Docker and Kubernetes samples on a real cluster were not run for this release. Redis Sentinel
    is not supported: storage.redis_url names one address, which must follow the primary.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.

v2.3.0

Choose a tag to compare

@kinncj kinncj released this 21 Oct 07:23

Maple Harness v2.3.0

Highlights

  • Prices for each model. Set a model's price in config.yaml, or in the model's ⋯ row on the setup page,
    and the Savings card values it from its first request, labelled as your price. Prices learned from your own
    requests always win, and the card tells you when they disagree with the one you set. See
    Prices for each model.
  • Copilot Auto and the same model by name value each other at once. Copilot bills a request through Copilot Auto
    at 0.9 times the model asked for by name; once either one's prices are known, the other is valued straight away,
    and the card says so.
  • One price for a token everywhere. The Cache card's estimates, cache warming, the model list, the cache-misses
    report, the harness's cost preview and its /savings now price a token as the Savings card does.
  • Several replicas without PostgreSQL. storage.mode: redis keeps usage, budgets, settings and tokens in one
    Redis (or Valkey) that keeps its data. PostgreSQL with Redis stays the recommendation for several replicas. See
    Kubernetes: without PostgreSQL.

Added

  • price for a model (providers[].models[].price: unit, input, cached_input, cache_write, output, per
    million tokens), in config.yaml and on the setup page. In AI units it values the model's savings; in dollars it
    gives the cost preview (before the provider's published list price).
  • storage.mode: redis (and MAPLE_STORAGE_MODE=redis): replicas share one Redis or Valkey, with no
    PostgreSQL. The proxy starts only on a Redis that keeps its data (append-only file on, maxmemory-policy noeviction), and a replica stops taking requests (not ready, and says why on the setup page) when Redis stops
    keeping it or loses it; budgets then follow storage.budgets_when_down. Docker (compose.ha-redis.yaml) and
    Kubernetes (config/ha-redis) samples with a restricted Redis user. In this mode the usage lake and SQL over the
    shared usage (--source postgres) are not available; maple-proxy usage commands and exports work as before.
  • Where each figure's prices came from. The usage log records it on each valued request (price_source,
    schema version 7: learned, copilot_auto_discount or configured); maple-proxy usage query --metrics priced_stated counts the requests valued without learned prices of their own, and the filtered Savings card shows
    it.

Fixed

  • A model is no longer valued at prices its requests cannot tell apart. When every request of a model had the
    same shape (the same prompt, the same share of it cached), its prices could come out wrong while still matching
    every bill; the Cache card's estimate of what cache reads cost was then off. Such a model now waits for requests
    that differ, and the card says why. Found on real GitHub Copilot requests.

  • Cache warming for a Copilot Auto conversation decides with Copilot Auto's prices, not the model's price by name.

  • The cost preview of a model reached only through Copilot Auto so far is no longer 10% low when you ask for it
    by name: it uses the Auto prices without the discount, and says so.

  • The model list's price (input_ai_units_per_1k, which the harness uses to pick a model for side jobs) comes
    from the same prices as the Savings card, not from a separate rough estimate.

  • The cache-misses report (maple-proxy usage cache-misses and the setup page) prices a missed token as the
    Savings card does, a Copilot Auto request at Copilot Auto's prices, and says which basis it used.

  • storage.mode in config.yaml with its key in MAPLE_STORAGE_KEY (or its addresses in MAPLE_POSTGRES_DSN
    and MAPLE_REDIS_URL) no longer stops the proxy at start with "needs storage.encryption_key": the environment
    is taken into account when the configuration is checked, as it already was when the proxy runs.

  • A session started for remote control could lose its socket when the remote gateway looked at it in the
    instant between the socket being made and it listening: the gateway took it for a crashed session and removed it,
    and the start failed ("announcing this session"). A session whose process is alive and whose announcement is new
    is now treated as still starting.

  • The harness's /savings learns each model Copilot Auto answers with on its own (their prices differ), and
    values models whose provider reports no cache writes (GPT models on GitHub Copilot) from their reads, as the proxy
    does.

  • Built with Go 1.26.10, which fixes a certificate-checking flaw in Go's standard library (GO-2026-6809).

Upgrading

  • The usage log moves to schema version 7 (one column added at the end) and the query index rebuilds itself once.
    Earlier lines read as before.
  • Today's savings may read higher than before the upgrade: Copilot Auto requests (and requests by name) are valued
    from the first request once the other kind's prices are known, where they used to wait for their own.

Known limitations

  • Redis mode was run against Valkey 9.1 built from source, not yet against managed Redis services (which may
    hide INFO or offer no append-only file: the proxy then refuses to start), Redis failover, or a real cluster.
    Anyone who can write to that Redis can change budgets and usage figures (settings and tokens stay signed and
    encrypted): give the proxy its own restricted Redis user, as the samples do.
  • GitHub Copilot sometimes answers a request with no usage at all (every token count and its price stated as 0),
    seen when a reply is cut off by a small output limit while the model is still reasoning. Maple records exactly
    that, as "the provider answered with no usage"; whether GitHub bills such a request cannot be seen from it.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.

v2.2.3

Choose a tag to compare

@kinncj kinncj released this 20 Oct 17:35

Maple Harness v2.2.3

Highlights

  • GitHub Copilot Auto's cache reads are valued, and stay valued. Copilot bills a request through Copilot Auto
    below the same model asked for by name; the two are now priced separately, so a model used both ways (for
    example gpt-5.3-codex) no longer shows its cache reads as "not valued yet" after its first request by name. See
    Copilot Auto.
  • The Cache card and the Savings card agree, and the cache hit rate has one meaning everywhere: the share of
    input tokens read from the cache, a model's first, cold request included (see Upgrading).
  • The Savings card's rows line up: each value sits on the line of its own label, at every window width.
  • The harness's cost preview works on Copilot Auto, at Copilot Auto's prices, and says so.
  • An unknown slash command never reaches the model, in any mode (see Upgrading for --prompt).

Added

  • /model <name> in the line mode (--repl), as in the full-screen view; a bare /model lists the models.

Fixed

The setup page and the proxy's numbers

  • Copilot Auto requests are valued at their own price. Copilot Auto requests, which Copilot bills below the
    same model by name, are now valued at their own price, shown as "via Copilot Auto" in the Savings card's prices.
    A model used both ways (for example gpt-5.3-codex) showed its cache reads as "not valued yet" for the rest of the
    day after its first request by name; it is valued again. Cache reads left unvalued earlier the same day are
    shown under the price they were answered at. See
    Copilot Auto.
  • The Cache card values Copilot Auto as the Savings card does. What a Copilot Auto request's cache reads cost
    is estimated at Copilot Auto's prices, not at the model's price by name.
  • A reused answer for a Copilot Auto request is valued. An answer from Maple's response cache for a Copilot Auto
    request was recorded under "auto", which has no prices, and was never valued. It is now recorded under the model
    Copilot chose for the original, as the original was, and valued at Copilot Auto's prices on the Savings card.
  • A model that costs nothing on your account (such as gpt-4.1 on GitHub Copilot) is listed as "free on this
    account: nothing to value", on the setup page and in maple-proxy savings, instead of keeping its cache reads
    under "not valued yet" forever.
  • One cache hit rate. A model's first, cold request is now counted as a miss when its provider states its cache
    figures (GitHub Copilot and OpenAI-compatible providers do, as 0 tokens read). The Overview and the Cache card
    left those requests out, so the hit rate read higher than the share of input tokens actually read from the cache.
    See What the numbers mean.
  • An answer with no usage is not a lost cache. GitHub Copilot sometimes answers with no usage at all (when a
    very small output limit is reached first); the usage log and the Cache card now say "the provider answered with
    no usage" instead of counting a lost cache.
  • The Cache card's last hour agrees with today. Its estimate of what cache reads and writes cost uses the prices
    each request was estimated with, as today's does; it could read more than the whole day's. Its request count,
    and that of maple-proxy cache, now says it is the requests sent to a provider, with the answers from Maple's
    response cache given apart (the Overview's Requests today counts both).
  • The Savings card says why a figure is not valued. The prompt-cache note says which reads were answered before
    their model's prices were known (and stay unvalued) and which models are still waiting for prices, naming only
    those. The response-cache line says why a reused answer is not valued ("1 answer reused", in the right number).
    A model's prices read "read … AI units per token; cache writes not reported by the provider", with the unit
    right after the prices.
  • The Savings card's rows line up. On the Savings card and the filtered Savings card, each value now sits on the
    line of its own label, at every window width and in both themes, however long the explanation under the label.
    On a wide window, a value could look as if it belonged to the row above.
    See Reading the numbers.

Maple Harness

  • The cost preview prices Copilot Auto. On Copilot Auto it gave no cost estimate, even when the proxy had
    already priced many of Auto's requests. The preview now uses what that model costs through Auto (the
    model Auto answered the last request with), and says so in its basis; a model reached only through Copilot Auto
    so far says its prices are Copilot Auto's. It needs the proxy from this release; until Auto has answered once in
    a session, it says why there is no estimate yet. See the cost
    preview
    .
  • An unknown slash command is never sent to the model, in the terminal, the line mode, --prompt, remote
    devices or --rpc: /model gpt-6-sol in --repl was sent as a message and answered at the cost of a request.
    The session says there is no such command and sends nothing; begin a message with // to send it starting with
    /. See
    Slash commands.
  • The context meter of a resumed session read "context: 0%" until the next request; it now shows the
    conversation's size at once (the harness's own estimate until the next request's real figure).

Upgrading

  • --prompt with an unknown slash command (/word) now exits with status 1 and sends nothing, instead of
    sending it to the model; start the prompt with // to send a literal /.
  • Copilot Auto's prices are worked out again at the first start after the upgrade, from the usage log, or,
    without enough of it, over the next few requests. Until then, the cost preview and the Savings card use the prices
    they had before the upgrade.
  • Hit rates may read lower than before, because a model's first, cold request now counts as a miss.
  • Going back to 2.2.2 keeps working, but drops the separate Copilot Auto prices and the counts of free models
    from the saved account; on returning to this release the Copilot Auto prices are worked out again, and free
    models are counted again.

Known limitations

  • The checks against real GitHub Copilot were run on Linux. macOS and Windows were not run for this release.
  • GitHub Copilot reports no cache writes for some GPT models (for example gpt-5.3-codex), so "tokens written to the
    cache" is 0 for them, as Copilot states it.
  • Printing the setup page was tested in Chromium only.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.

v2.2.2

Choose a tag to compare

@kinncj kinncj released this 20 Oct 12:42

Maple Harness v2.2.2

Highlights

  • The setup page's numbers agree everywhere. For the same time and filter, the Overview's tiles, the Savings
    card, the charts, the filtered cards, the CLI, the CSV export and the DuckDB views give the same requests, AI units,
    tokens and estimates, and today's savings on GitHub Copilot's GPT models are now valued. See
    Reading the numbers.
  • One clean final answer after Maple Harness checks an answer. When a check sends an answer back, the terminal,
    the remote app, --rpc, serve and --prompt all end on the model's corrected answer, not on an argument with
    the check.
  • Days start at your local midnight. "Today" and the whole-day history ranges (--last 7d, last=7d, the
    page's range buttons) now count from midnight in the proxy's time zone, so "1 day" and "today" agree (see
    Upgrading).
  • The Overview prints, or saves as a PDF, with its charts as drawn.
  • No request is lost or counted twice after a crash, and a request for an unknown model is now recorded.

Added

  • A "Today" choice on the setup page's range buttons (since midnight, local time).
  • harness.max_auto_continues, the most automatic continues one message gets (50 unless set; see Fixed).
  • New JSON fields: without in maple-proxy savings --json and /api/savings/summary (what you spent plus the
    net avoided), session_* and resumed in --json-summary (a resumed session's totals), and interrupted there
    for a run stopped by a timeout or a signal.

Changed

  • The harness's own lines are short and start with », so they never read as the model's: » check: … when an
    answer is sent back, » checked: … when the corrected answer is in, » continuing: … when the model is asked to
    carry on. The answer a check sent back is folded to one line (click it, or press ctrl+o, to read it again).
  • yolo mode is shown in the warning colour and says "tools run without asking".

Fixed

The setup page and the proxy's numbers

  • The setup page's numbers agree with each other and with the CLI. For the same time and filter, the Overview's
    tiles, the Savings card, the charts, the filtered cards, maple-proxy usage query, maple-proxy savings, the CSV
    export and the DuckDB views now give the same requests, AI units, tokens and estimates:
    • "Today" is since local midnight everywhere; a filtered tile used the UTC day and counted last evening too.
    • Requests today now counts answers from the response cache, as the usage log does.
    • AI units today and "You spent ... today" are the same sum: a failed request that reported AI units is counted
      in both, and the Savings card now refreshes as requests finish instead of every 30 seconds.
    • The history charts are drawn again when new requests arrive, and every chart and filtered card says when it was
      read, so two cards for one filter no longer show different counts without saying why.
    • After a restart, today's Savings figures are the ones the usage log recorded, even when the saved figures were
      older than the cache view's or missing.
    • Every AI-unit figure shows the same three decimals on the page, in the CLI and in the API: totals and
      estimates are added up exactly, and a half rounds away from zero everywhere (the page showed 21.262 where
      maple-proxy usage query said 21.263).
    • The Savings headline shows the estimate, not a cap. It was capped at what was spent, so on a day whose prompt
      cache avoided more than was spent the headline showed a smaller figure than the line below it. It now
      says what the prompt cache and trimming avoided (estimated), less Maple's own costs (now including its
      cache-warming pings), beside what you spent and what you would have spent without them (spent plus the net).
      The filtered Savings card and maple-proxy savings follow the same rule, and the filtered card shows per model
      how many requests had an estimate.
    • With a filter, every Overview tile follows it. "Failed, last hour" ignored a model filter while "Requests
      today" followed it; the line above the tiles now says which tiles are live.
    • "Failed, last hour" survives a restart. It read 0 after the proxy restarted (the live view starts empty);
      it now reads the usage log.
      See Reading the numbers.
  • History ranges start at local midnight. last=7d (and --last 7d, and the page's range buttons) counted UTC
    days, so "1 day" and "today" disagreed by your time zone's offset, and right after local midnight "today" was 0
    while "1 day" still held yesterday. Whole days now start at midnight in the proxy's time zone. Custom ranges
    given as days stay UTC days, as before, and say so.
  • Cache reads on GitHub Copilot's GPT models (which report no cache writes) are now valued, as an estimate, on
    every view. They showed as "not valued yet" forever, so today showed nothing saved while earlier days (another
    model) did. A model still without an estimate is named under Not valued yet, with the reason.
  • The Cache card's "≈ AI units" for cache reads and writes divided each request's whole cost by its cached tokens,
    overstating what reads cost; it now uses the prices the Savings card uses.
  • Cache warming now values the cache the same way the Savings card does, instead of with the Cache card's
    overstated prices.
  • The Overview prints. Printing it, or saving it as a PDF, showed blank charts; charts now print as drawn, in the
    light theme, without the buttons.
  • The list of requests as they finish shows the date of a request from before midnight.
  • No request is lost or counted twice after a crash. A request that started before the proxy saved its figures
    and finished after was missing from the tiles and the Savings card after a restart.
  • A request for an unknown model is recorded. It answered 404 but left no usage line, so it was on no card.
  • Fallbacks are counted whether or not they answered. A fallback that also failed counted as 0 fallbacks; the
    outcome chart's series now says "fallback, answered".
  • A model GitHub Copilot will not serve to Maple (it answers "not available for integrator" though its model
    list offers it) now fails with the reason "model not offered to this client" and is marked in the model list,
    instead of a bare "status 400" on every attempt.

Maple Harness

  • Corrections from Maple Harness's own checks no longer turn into an argument with the model. When a model's
    answer reported a command's result though the command never ran, the harness's note could be answered back
    ("Confirmed. I ran it now…", "It's genuinely finished."), and after the model had run the command the harness
    then said it had "described a step without taking it". The reply after a check that ran is now taken as the
    answer, the notes ask the model for its corrected final answer instead of a reply, and the screen ends on that
    answer. See Troubleshooting.
  • Every client ends on the same answer after a check. The remote app on your phone or browser now folds an
    answer a check sent back and ends on the corrected one, and no longer shows an empty bubble for a reply that was
    only tool calls; --rpc, serve and a terminal session shared with remote devices get that same final answer,
    not the model's reply to the check, and the output of maple-harness --prompt ends on it (printed again after
    » final answer: on the error stream when it was not the last answer printed).
  • The final summary is shown once. The last answer of a turn could appear twice in the terminal.
  • A turn that ends on a stop message shows it, in the terminal and the remote app, also when the model had
    already answered earlier in the turn. The message a turn stops on after the same problem twice ("Stopped: the
    same problem was reported twice…") now says only what is wrong, without the instructions meant for the model.
  • A command run in an earlier message counts. A change report asked for in a later message ("now give me the
    report") was told its build or lint "was not run in this session" when an earlier message had run it.
  • The "continuing" nudge no longer follows an answer that was complete: not after a question put to you ("Which
    database should we use? …"), not after "lint passes" or "the tests pass", not because of a build's output quoted
    in the answer, and not after a step the model already took.
  • A finished task no longer runs on by itself. A model that ended each step with a complete change report
    offering more work ("If needed, next step is…") was asked to carry on after every report and did the offer, step
    after step, on work nobody asked for, until a timeout stopped it. A change report with its verification section
    now ends the task; asking the model to carry on now says to continue only the task you asked for; and one message
    gets at most 50 automatic continues (harness.max_auto_continues), and none after the model, asked to carry on,
    describes the next step again instead of taking it. Then the turn stops with "Stopped: the model kept describing
    steps without taking them", with the model's last report as the answer; type "continue" to carry on.
    A verification line such as "Build: N/A → no build step" is no longer sent back as a command that did not run,
    and python3 -m unittest, ruff, mypy, eslint and similar now count as checks.
  • A run stopped by a timeout or Ctrl-C writes its --json-summary, marked "interrupted": "terminated" (or
    "interrupt"), with what it had spent until then; the process still ends with that signal, as before.
  • **--json-summary reports t...
Read more

v2.2.1

Choose a tag to compare

@kinncj kinncj released this 13 Oct 03:31

Maple Harness v2.2.1

Highlights

  • Developer sign-in, step by step with pictures, and clearer words along the way; a sign-in revoked at the hub
    now shows at once. See
    How developers sign in.
  • CI recipes for running Maple Harness in a pipeline, and clearer advice when a run cannot verify its proxy.
  • Settings from a ConfigMap: an id given to someone else no longer keeps the previous person's sign-in.

Added

  • CI recipes for Maple Harness. CI jobs and containers
    now has copy-paste GitHub Actions and shell examples: start the proxy in the job (recommended: nothing to set, the
    harness verifies it), or connect to a proxy in a service container with maple-harness proxy connect --token-env,
    with what each guarantees, and how a script tells a run that did not start (exit status 1 and the line This session cannot ask you, so it did not start.).

Fixed

  • maple-harness serve on an unverified proxy now says why it did not start plainly, like --prompt and
    --rpc (it printed the message behind a log time stamp), and exits with status 1 as they do.
  • An unverified proxy's fixes now fit its cause. A restart is suggested only for your own proxy on an older
    release; a proxy of another identity (another user, settings folder or a container) gets sharing the settings
    folder, trust_unchecked or proxy connect; another program on the port gets "stop it, or give your proxy another
    port"; and when the harness cannot tell, each fix says which case it is for. The same in proxy status,
    maple-harness doctor and the start-up note.
  • A proxy on this machine that asks for an access token, used with trust_unchecked: a run nobody can answer
    now stops after the first refusal (exit status 1) instead of sending requests until the proxy locks it out
    ("429: too many failed attempts"), and every such refusal says to connect to it with maple-harness proxy connect --token-env, since no token is sent to a proxy that is not verified as yours.
  • Upgrading a CI job from 2.1, corrected: a job whose config.yaml names harness.proxy.token without
    harness.proxy.url (a proxy in a container that asks for a token) is not fixed by MAPLE_TRUST_UNCHECKED_PROXY=1,
    because the token is never sent to a proxy that cannot prove it is yours: run maple-harness proxy connect --token-env MAPLE_PROXY_TOKEN <address> instead. The 2.2.0 notes said either way would do.
  • Settings from a ConfigMap: an id given to someone else no longer keeps the old person's single sign-on. When
    the proxy cannot write its settings, a person without a sign_in_marker line is now told apart by the addresses
    setup_page.sso.people names them by. Remove a person and add someone else with the same id in one edit, with other
    addresses (or none): within seconds of a replica reading the file, the old account stops signing in as that id,
    what was made for the old person is cleared and the tokens they made for themselves are revoked (sso.prune,
    reason=identity_changed). Team, budget, model and account edits never end a sign-in, and neither does adding an
    address or removing one of several; a sign_in_marker line still wins. Removing all of a person's addresses now
    ends their sign-in (put the line back and they sign in again). At the first start after the upgrade, sign-ins made
    with 2.2.0 are kept when the ConfigMap still names their address for that person; one whose address it no longer
    names (an id given to someone else while 2.2.0 ran, or the line removed after the first sign-in) is cleared with its
    terminals and own tokens, and a developer whose GitHub account lists another address first may have to sign in once
    more. See
    Kubernetes.
  • Developer sign-in, clearer words and layout. With developer sign-in on, the sign-in page names the proxy and
    says admins manage it there and developers get their own access there (it spoke of the setup page to everyone). An
    account that is not allowed is told "is not allowed to sign in here. Ask an admin of this proxy to add your company
    account or domain."; one refused only when the sign-in finishes gets the same page instead of "start again". The
    Request sent page is a proper sentence. A tool whose request goes to a hub that revoked this proxy's sign-in
    is told what to do ("The hub … refused this proxy's access (the sign-in was revoked or expired). Sign in again on
    the proxy's Hubs & chaining tab, or run maple-proxy chain connect --sso …"), still as a 502, without the hub's raw
    answer, which is in the proxy's log. maple-proxy chain status lists each hub as a short block that fits an
    80-column terminal and says REVOKED: sign in again for a sign-in the hub ended (its JSON adds revoked);
    the sign-ins' heading is now Sign-ins with a company account:. The Sign in again button keeps that one label
    everywhere, the wizard's Developer sign-in choices are short with their help below, and a request on People
    & teams
    keeps Accept and Refuse together on phones and wide screens. The single sign-on wizard's
    Groups to teams box is three lines tall, as meant, instead of a tall empty box.
  • A sign-in revoked at the hub shows at once. maple-proxy chain status (every run), and the setup page, the
    hub provider's health and maple-harness doctor (at most once a minute) now ask the hub whether this proxy's
    sign-in token still works, so a revocation on My access or by an admin shows as REVOKED: sign in again right
    away, before any request. A hub that does not answer is said to have not answered, never taken for a revocation;
    a hub before 2.2.0 cannot be asked and shows it at the next request or renewal, as before.

Upgrading

  • Settings from a ConfigMap (approved security fix): removing every one of a person's addresses from
    setup_page.sso.people now ends their developer sign-in, and at the first 2.2.1 start a 2.2.0 sign-in whose address
    the map no longer names for that person is ended. Put the address line back and the person signs in again.

Known limitations

  • The new sign-in walkthrough was captured with the demo's stand-in identity provider; real providers' screens look
    different. macOS and Windows were not run for this release.

Install

curl -fsSL https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.sh | bash
irm https://raw.githubusercontent.com/kinncj/maple-harness-dist/main/install.ps1 | iex      # Windows

Update an existing install with maple-harness update. The installers and update verify every download against SHA256SUMS
(and its signature, when the project signs releases). Third-party licences: THIRD_PARTY_NOTICES.md, or maple-harness licenses.

Free for everyone to use, commercial use included. Closed source, provided as is, without warranty — see the attached LICENSE.

macOS: these binaries are not notarised, so macOS will warn if you download them by hand. The installer and maple-harness update are unaffected.
Windows: these binaries are not Authenticode-signed, so SmartScreen may warn.