Skip to content

1.8.1

Choose a tag to compare

@github-actions github-actions released this 18 Aug 12:37
· 30 commits to main since this release

Sigma Oasis v1.8.1 — the app remembers

A small model's fourth weakness, after thin knowledge, weak arithmetic and shallow procedure, is
that it loses the thread of a long conversation: it re-remembers a figure from ten turns ago
approximately and states it confidently. This release gives the app its own memory of what a
conversation has established — mechanical, exact, never paraphrased — and measures it in the one
regime where memory actually matters. It also closes out the v1.8 sessions story with a measured
improvement to how the model uses them. Pinned by 1,345 node checks (28 new since v1.8.0).

The conversation ledger

  • What it is. Every turn, the app rebuilds a running record of what this conversation has
    established — figures a tool computed (total revenue: 139306.12 — run_python, turn 1), files
    attached, Python session variables still defined, and constraints you stated in your own words
    ("My budget is $2,000", "I have a nut allergy"). No model call; the exact strings from tool
    results and your messages, never reformatted.
  • What it refuses to record. Anything an assistant said. A reply's figures already pass
    through the grounding ladder; repeating one into the ledger would launder an unverified number
    into a "fact". The ledger is the app's memory of the conversation, not the model's.
  • How it reaches the model. From the fourth turn on, only when it has something to say, as
    turn notes: use these exact values when a question refers back; do not restate them from
    memory; if a figure is not here, compute it.
    Disclosed under the reply — 📒 Ledger: 2 computed
    facts, 1 file, 1 constraint from 7 turns.
    Settings → Models → Conversation ledger, on by default.
  • Why not the summary that already existed? That one is written by a model, only when history
    is dropped, and it paraphrases — which is exactly the failure. The ledger keeps 139306.12.

Measured, in the regime that matters

A new eval suite (EVAL_SUITES=ledger) establishes a fact on turn 1, buries it, then asks for it
back without restating it — ledger arm vs. bare, identical otherwise. The first suite was a null
result
— five turns is short enough that the bare model just reads the turn-1 tool result out
of history — and it is recorded as such. So a long-regime variant compacts the establishing turn
out of the wire history using the app's own history planner and asserts it is gone, exactly as
a real long conversation loses it. Against a 9B, three passes:

establishing turn compacted out recall stability
ledger 15/15 · 100% 0 flaky
bare 3/15 · 20% 2 flaky

Bare's three successes are all recomputation against a still-attached CSV, not memory. Where
nothing is recomputable — a stated budget and deadline — bare said "I don't have any record of a
project with a budget or deadline in this conversation"
three passes out of three; the ledger arm
said "You told me at the start that your budget is $2,000 and the deadline is Friday."

The measurement also caught a real gap on the way: a 9B answered a total straight off
analyze_file's profile without ever running Python, and the ledger — which read only
run_python — recorded nothing. The model then told the truth about its own empty ledger rather
than inventing, which is the design working; the extractor was too narrow, and profile statistics
are now facts. docs/evals.md
has the full sequence, including what did not work first.

Sessions: the habit, measured and moved

v1.8.0 measured that a 9B told variables persist still re-read the data file on 6 of 10
follow-ups. One step in the data-analysis playbook now says: run_python keeps its variables —
build on the dataframe you already loaded; check the Session variables list before reading a file
again.
Two 3-pass runs, differing only in that step: session follow-up re-reads fell from
100% to 67% while the stateless control stayed at 100% in both. Narrowed, not closed — and
now a number future work is judged against.

Also

  • The eval harness refuses to start unless the model answers a probe — a 90-minute baseline
    once ran against a stopped LM Studio server and produced 0/0 across the board; it now fails in
    about a second with the fix named. Multi-pass headlines report the aggregate, not pass 1.

Upgrade notes

Auto-update from v1.8.0 or earlier. One new setting, on by default (Settings → Models →
Conversation ledger). Nothing else changes.

Full changelog: v1.8.0...v1.8.1