Release v0.8.0 · gburd/improv

Improv runs inside Postgres, and a formula can be a SQL function. The
pg_improv extension models your own tables and keeps results current
incrementally (at 100k cells, a one-row change is visible about 50 ms after
commit); LANGUAGE improv writes SQL functions in the formula language;
the desktop and terminal apps open a Postgres-hosted model by URL and edit
its rows. The GUI now follows NeXTSTEP Lotus Improv closely, scrolls wide
grids sideways and can revert after a save conflict. A security review's
findings are fixed, and every crate is held at 90% function and branch
coverage.

Added

Documentation. A Ways to Use Improv chapter (GUI, TUI, CLI,
streaming, Postgres, LANGUAGE improv, server: what each is for and how
to start), a refreshed introduction, getting-started and running pages,
and man pages for improv-tui, improv-gui and improv-server
(improv(1) gains stream, save-view, the CSV and refresh commands).
The guide and API docs publish to Codeberg Pages and GitHub Pages.
Demo movies: the README and guide show the GUI, the TUI and Improv in
Postgres in motion, each recorded by a script under docs/demo/.
Release binaries for Linux, macOS (arm64 and x86_64) and Windows: the
CLI, TUI, GUI and server, with the man pages.
GUI: grids scroll sideways. A worksheet wider than its window has a
NeXT horizontal scroller in the bottom strip, right of the row tiles, with
the arrows grouped at its left end (spec §4.4). Before, the far columns
could not be reached.
Revert to Saved after a save conflict: GUI File ▸ Revert to Saved
and TUI R reload the model from the store (with a fresh save token),
picking up the other writer's changes, so the conflicting edit can be
redone. Undo history is dropped; a failed reload changes nothing.
LANGUAGE improv: SQL functions written in Improv's formula language
(crates/pg_improv). CREATE FUNCTION margin(price float8, cost float8) RETURNS float8 LANGUAGE improv AS $$ price - cost $$. A trusted language:
arguments are named inputs, Name = expr lines define intermediates, the
last line is the result, and the DSL and controlled English both work.
Numbers, booleans, text and dates/timestamps; NULL in gives NULL out where
the result depends on it; division by zero is NaN; aggregations are
rejected. Validated at CREATE FUNCTION; compiled once per backend
(0.14 µs per call over inline SQL). Evaluation is the new
improv_engine::scalar, which shares the dataflow engine's per-cell
operators.
pg_improv: Improv models over Postgres data (crates/pg_improv,
crates/pg_model, crates/storage_pg, crates/store). Bind fact-table
columns to categories and measures, define formulas, read results from
improv_<model>.<measure> tables. Three update paths: logical decoding
(default; a background worker tails a slot and applies each batch with the
slot advance in one transaction), triggers (in the writer's
transaction), and improv.recalc() (explicit, in your transaction);
plus improv.what_if and improv.status. The GUI, TUI, CLI and server
open postgres://…?model=<name>. Measured on a c7i.4xlarge at 100k cells:
logical keeps a writer's UPDATE at 8 ms (results ~50 ms later); a trigger
makes them visible at commit for ~1.1 s per write. See
docs/src/postgres.md.
pg_improv: aggregated bindings. improv.bind_measure(…, agg => 'sum')
(or avg, min, max, count) folds a fact table's rows per coordinate
into one input cell, in recalc, trigger and logical mode and in the GUI's
load; without agg duplicate rows remain an error. The GUI's store refuses
to save an edit to an aggregated measure (Integrity).
pg_improv: boolean, text and date measures. bind_measure takes the
measure's type from its column (bool; text/varchar/char;
date/timestamp/timestamptz; the number types as before) and refuses
others; define accepts boolean, text and datetime results, and result
tables get a matching value column (boolean, text, timestamptz).
Logical mode applies their row changes cell by cell like numbers' (the
worker reads the slot's text with TimeZone = UTC, DateStyle = ISO).
pg_improv Postgres extension, layer L0 (in progress). A pgrx 0.17
extension (crates/pg_improv, pg17) with an improv.* catalog that is a
superset of improv_storage_pg's, plus improv.create_model,
bind_category, bind_measure, define, recalc and drop_model.
recalc recomputes every derived measure in the caller's transaction and
writes only the changed rows to improv_<model>.<measure>. The new GUC
improv.max_cells (default 5M) caps the input size. The Postgres-free
logic is the new workspace crate improv_pg_model. Install and quick
start: crates/pg_improv/README.md.
pg_improv layer L1: trigger-mode sync.
improv.set_sync_mode(model, mode) installs (for trigger) or removes
statement-level AFTER INSERT/UPDATE/DELETE/TRUNCATE triggers with
transition tables on every bound relation. improv.on_change recomputes,
inside the writer's transaction, only the derived measures that depend on
the changed inputs. It restricts the recompute to the touched coordinates
when the chain is element-wise. Also new: improv.recalc(model, measures text[]) recomputes a measure subset and its dependents, and
improv.what_if(model, edits jsonb) returns the derived cells an in-memory
edit would change, without writing anything. README has the
trigger/logical/recalc trade-off table.
Postgres-hosted models (pg_improv layer L3). The CLI, TUI, GUI and
server open postgres://…?model=<name> as well as a Mentat file, through
the new improv_store::Store seam. The new improv_storage_pg::PgModelStore
keeps the model in the improv.* catalog: bound input measures read and
write their fact table, unbound cells live in improv.cell, and
improv.model_version drives the existing reload-and-retry conflict check.
scripts/gui-shot.sh takes GUI_SHOT_DB to capture such a model.
Coverage gate. scripts/coverage.sh (and a coverage CI job on
GitHub and Forgejo) fails when any crate drops below 90% of functions or
branches; the workspace stands at about 96% / 95%. New guide
docs/src/running.md: build, run each front end, security defaults, tests,
coverage.
TUI: Shift+arrows select a range, drawn highlighted, and the status line
shows its Sum (numeric cells only; Sum — when there are none), as the
GUI does. A plain arrow, a click, or any re-layout collapses the range.
GUI screenshot regression in CI (review plan D5). A gui-shot job
(GitHub and Forgejo, rust:1.97-trixie) runs scripts/gui-shot-check.sh,
which compares the headless 1400×900 capture against the golden
docs/reviews/golden/gui-1400x900.png and fails above 200 differing pixels
(magick compare -metric AE -fuzz 2%), uploading the diff. --update
refreshes the golden (see CONTRIBUTING.md).
CLI save-view. improv save-view <db> <view-id> <name> <measure> [--rows Cat,Cat] [--cols Cat,...] writes a named view the same shape the
GUI saves (stacked rows/cols, remaining categories as pages pinned to their
first item). improv list now prints a views: section.
GUI screenshot harness (scripts/gui-shot.sh). Builds the binaries,
loads a fixed Time×Product×Scenario demo model, runs improv-gui on a
headless Xvfb at 1400×900 and writes target/gui-shot/gui.png plus
compare.png, the capture beside the NeXT Improv reference. Output is
pixel-identical run to run.
GUI range selection. Shift+arrow (or Shift+hjkl), Shift+click and
dragging across cells select a rectangle; a plain arrow or click collapses
it. The status-bar Sum adds the range's numbers.
GUI window title. The native title reads Improv — <file>, with a
trailing • while edits are unsaved (an edited in-memory model, or a
failed autosave).
GUI transient status. Status-bar messages (save, recalc, undo/redo,
added formula, errors) clear themselves after 5 seconds.

Changed

improv_engine::session::Edit::value is a CellValue (was f64), so
the live engine takes boolean, text and date edits and seeds every typed
input cell. Engine::set still takes an f64.
improv.what_if returns jsonb values (was float8), so it reports
boolean, text and date measures too (dates as RFC 3339 strings), and its
edits may set inputs of those types.
Mentat 0.14 → 1.11.1 (pin 6fc2bc7f, Mentat master at v1.11.1).
No Improv API changes were needed. Measured on 100k/200k-cell models
(docs/reviews/2026-10-10-mentat-1.11-upgrade.md): loads, partial loads,
evaluation and edits 9–26% faster, re-import 13–15% faster, first import
~10% slower, store files ~29% larger (new history and value indexes).
Reported RSS is higher because Mentat now memory-maps the store (1 GiB cap);
set MENTAT_MMAP_SIZE=0 to turn that off. The old pin 1af15e15 lived on
an unpushed branch and was not fetchable from a clean clone; the new pin is
on Mentat's pushed master.
TUI: the status box grows to fit its content, so the edit/command prompt
and messages stay visible when pages:/filters: lines are shown. The
key help wraps to the terminal width, and the prompt sits right under the
title so a short terminal clips the help first.
TUI: a derived measure's formula shows on a status line in controlled
English (Revenue = Price times Quantity).
GUI: the model browser now leads with a NeXT-style white list of saved
views (Name - Worksheet rows with a worksheet icon). A single click
selects a view and shows its computed description (measure, categories,
pages); a double-click opens it. Categories and Measures move below as
collapsible sections.
GUI: NeXT scrollers. The grid, model browser and formula list show an
always-visible 19 px #808080 trough with a square #AAAAAA knob,
a 1 px bevel on the knob and NeXT's two arrow buttons grouped at the
bottom of the vertical trough (one line per click).
GUI filtering moved to the tile menu. The worksheet's Filters
shelf is gone; right-click a category tile for a checkbox per item (uncheck
to hide it from that worksheet) and Show all. A filtered tile reads
Name*.
GUI grid field fills the table area. The white field reaches the
worksheet's edges instead of stopping at the last cell.
GUI model browser labels measures. Each measure shows a small gray
input or formula label instead of the · / = prefix.
GUI first worksheet window. A fresh window opens clear of the
application menu and is sized to its grid (360–900 × 300–700 px).
GUI grid follows NeXT Improv. Cells are plain white with thin black
rules (no stripes, no per-cell buttons); the selection and its row and
column header items fill #AAAAAA. Input cells carry a faint inset
underline; derived cells show formula: … on hover. Double-click (or
Enter / F2) edits a cell; a single click only selects it.
GUI category tiles follow NeXT Improv. Tiles are 17 px dark-gray chips
with white names (no grip, no ⏷ button; right-click for
Move to Rows / Columns / Pages). Page tiles sit top-left, each above a
white item field that picks the page (replacing the < item [i/n] >
stepper); column tiles sit top-right and row tiles bottom-left. Gutters size
to their tiles, with no placeholder text, and highlight as drop targets
only during a drag.
GUI worksheet windows open at 560×420 (was 480×320) so the grid and
formula list fit.
GUI formulas are edited in place in the formula list. The top formula
bar is gone: the selected row is the editor (Enter commits, errors show
under the row, Esc reverts), and typing Name = formula on the list's
last line adds a derived measure.
GUI formula list lives in each worksheet window, under the grid, with a
Calc button. The operator palette and the "New derived measure" form are
gone.
GUI model browser and Info panel are NeXT windows. The docked
explorer and inspector side panels are gone, so the dark workspace spans
the full width above the bottom panels. The model
browser (<file> - Model, bottom left; Categories, Measures, Views) has a
close box and View ▸ Model Browser toggles it. The Info panel (top
right, closed at start) opens from Info ▸ Info Panel… and closes from
its close box. A fresh matrix starts right of the application menu.
GUI matrices are NeXT worksheet windows on a dark workspace. The
scrollable canvas is gone: the area between the panels is a #555555
workspace, and each matrix floats on it as its own window with a black
(key) or dark gray title bar reading <matrix> - Worksheet - <file>
(Untitled when in memory), a close box while more than one matrix is
open, and a bottom resize bar. Drag the title bar to move a window; clicking
a window gives its matrix the keyboard. Window positions and sizes are saved
in views as before, so existing views load unchanged.
GUI application menu replaces the tool palette and tab strip. A
floating NeXTSTEP vertical menu (Info, File, Edit, Format, Worksheet, View,
Graph, Print…, Services, Hide, Quit) with cascading submenus now carries
pivot, undo/redo, chart, CSV, save view, save model, add/remove matrix, and
the Info panel (inspector) toggle. The View submenu lists the saved views
that the top tab strip used to show. New key equivalents: Ctrl/Cmd+S saves
the model, Ctrl/Cmd+H hides and Ctrl/Cmd+Q quits, next to the existing
undo/redo chords. None of them fire while a text field has focus. A
selected row in the explorer and formula lists now shows a white face, so
selection no longer looks the same as hover.
GUI theme is NeXTSTEP grayscale. The GUI uses NeXT's four grays
(#000000, #555555, #AAAAAA, #FFFFFF, plus a #808080 scroller
trough) and nothing else: the selection is filled #AAAAAA instead of blue,
corners are square, strokes are 1 px, the grid sits on white, and category
tiles are dark gray with white text. Formula errors are marked by a 2 px
underline rather than red, and charts use gray shades. Body text passes WCAG
AA: black on #AAAAAA is 9.0:1, black on white 21:1, white on #555555
7.5:1.
GUI text is Arimo 12. The GUI bundles Arimo 1.33 Regular and Bold
(OFL-1.1, crates/gui/assets/fonts/), metric-compatible with NeXTSTEP's
Helvetica, as its primary proportional font; egui's fonts stay as glyph
fallback. Body and button text are 12 pt, and headings are 12 pt bold.
GUI grid right-aligns numbers so their digits line up, as Improv does;
text, booleans and dates stay left-aligned. The cell editor for a numeric
measure is right-aligned to match.
GUI item headers are plain text in a ruled band. Row and column headers
(Q1, Widget) were beveled button chips. They now draw as plain dark text
with rule lines, as in Improv, so only the category tiles look draggable.

Performance

pg_improv logical mode applies changes incrementally (G1). The sync
worker keeps each model loaded with a live engine, turns decoded row
changes into input-cell edits, and writes only the result rows that
changed (a one-row UPDATE on a 100k-cell model rewrites two result rows).
It rebuilds on a structural change, TRUNCATE, a new item, or an error. It
also logs a standby snapshot after each batch so polls decode only new
WAL. Commit → result visible at 100k cells (p50): 1161 → 52 ms for one
row, 1552 → 559 ms for a bulk UPDATE.
Store load is no longer one query per cell (P1). load_model /
load_partial fetch (measure, coord, value) in one query per value
attribute (three in all) instead of a query per cell. On a 100k-cell
model (release CLI, 100×1000 CSV) improv eval went from 279.9 s to
0.58 s and improv list from 283.2 s to 0.67 s. Date cells now load back
as dates instead of text.
Engine edits cost O(changed measures), not O(model) (P3).
session::Engine now returns a copy-on-write session::Snapshot
(Arc<HashMap<MeasureId, Arc<MeasureValues>>>): an edit shares every
unchanged measure with the previous snapshot instead of cloning all
tracked cells. On the incremental_many_edits stress test (release,
5,000 single-cell edits) the time per edit drops from 950 us to 62 us at
10k cells (1,052 to 16,097 edits/s), and from 5,639 us to 71 us at 100k
cells (177 to 14,039 edits/s). It is now almost flat in model size.
Engine::new/set/clear/apply return Snapshot, and callers clone
it cheaply.

Fixed

Numbers display without binary noise. Every front end showed a value
at full f64 precision, so 1173 * 0.05 read 58.650000000000006. They
now show 15 significant digits (58.65); stored values and exports keep
every bit.
pg_improv: item ids are minted in one place. improv.mint_items
(created by both the extension and improv_storage_pg::CATALOG_SQL) mints
ids for new dimension values, serialized per model, with a unique
(model_id, category_id, name) index. improv.recalc, the trigger and
worker paths and the GUI's store all call it, so concurrent loads agree on
ids and nothing is minted twice. Minting no longer bumps model_version.
Postgres saves no longer clobber SQL edits to bound fact tables
(storage_pg, design §9.2 G2). save_model now writes only what changed
since the handle loaded (or last saved). Each bound cell it changes or
removes is guarded by its loaded value (… WHERE value IS NOT DISTINCT FROM <loaded>), so a cell SQL changed in the meantime fails the whole save
with Conflict ("reload and retry"). A removed bound cell sets the value
column to NULL and keeps the row. A save deletes only the catalog rows
(categories, items, measures, views, improv.cell cells) it had loaded, so
rows added by another client survive. A save with no changes writes no
cells. PgModelStore::snapshot / set_snapshot carry the base across
handles, like loaded_tx; without a snapshot a save writes everything
unchecked, as before.
pg_improv: recalc, define and bind_* no longer fail with "SELECT
FOR UPDATE is not allowed in a non-volatile function" when they are the
first statement of a transaction.
Broadcast binary ops with the larger operand on the left. A formula
such as Net = Price[Time,Product] - Discount[Product] (or *, +, /)
evaluated to no cells. Only the mirrored Discount - Price worked. The
dataflow MapBinary joined its operands on the full coordinate, so a row
matched only when the right operand already spanned the result's
dimensions. Join, MapBinary and two-argument scalar calls now share one
broadcast-join helper that keys both sides on their shared categories.
Incremental recalculation no longer drops cells whose value went down.
The engine's update stream orders a changed cell's +new before -old
when the new value sorts lower, and the session snapshot applied the
retraction by deleting the key, which wiped the value just written. So
Engine::set lost, for example, Revenue when a Price decreased. A
retraction now removes a key only if it still holds the retracted value.
Sandboxed external functions on non-FHS hosts (NixOS). The interpreter
is now run by its canonical host path, and /nix/store is bound read-only
when present. Before, every sandboxed call failed on NixOS with
execvp python3: No such file or directory.
A save no longer clobbers another process's writes. save_model retracts
stored cells the model lacks, so when two processes shared a store (improv set beside a running GUI or server) the second saver deleted the first's
new cells. A save now checks the store's transaction high-water against the
one it loaded at and, if another writer committed since, fails with
StoreError::Conflict and writes nothing. The GUI and TUI report "reload and
retry"; the server answers 409 Conflict and drops its cached model so a
retry reloads. New ModelStore::{tx_high_water, loaded_tx, set_loaded_tx, reload_model}.
Storage: a measure's category order survives save/load. :measure/categories
is a set, so a reload returned the categories sorted by id, and anything
mapped by position (SQL/CSV columns, coordinates) landed on swapped
categories: refresh-sql on a 2-dimension measure wrote
{Year=A,Region=2025}. The order is now stored in a new
:measure/category-order attribute. Stores written before it load as
before (id order) and gain it on their next save; no migration needed.
TUI: the grid scrolls to keep the cursor visible on models larger than
the terminal (previously the cursor could move off-screen).
TUI: mouse clicks select the cell under the pointer (they were offset by
the row-label column and selected the column to the right).
TUI: text and boolean input cells can be edited (parsed per the measure's
type, as improv set does); previously every edit had to be a number.
TUI: editing is refused when an axis is empty or fully filtered. It used
to store a cell with a missing dimension.
TUI: [ / ] page correctly when an empty category precedes another
page dimension.
CLI --at accepts every natural spelling of a coordinate. set and
scenario now take repeated flags (--at Time=Q1 --at Product=Widget) and
space-separated pairs (--at Time=Q1 Product=Widget) as well as the
comma-joined form, merging them. Naming a category twice is a clear error,
and stray non-pair arguments are still rejected.
GUI tool-palette icons no longer render as tofu boxes. Undo, redo,
import/export and save-view used glyphs (↶ ↷ ⇄ ▤) missing from egui's
bundled fonts; they now use glyphs those fonts contain, and a unit test
checks every palette glyph against the loaded fonts.
The matrix no longer draws over the Inspector. Gutter tiles and the page
well escaped the canvas clip (egui's nested panels reset the clip rect) and
painted on top of the side panels in small windows. They are now clipped to
the canvas. The window also opens at 1280×800 (was eframe's 800×600) with a
960×600 minimum.
The GUI no longer loses in-memory work silently. File ▸ Save (Ctrl+S)
is disabled for an in-memory model instead of reporting "saved model", and
quitting an edited in-memory model warns first; a second Quit discards it.
GUI menus stay on screen. A submenu that would run off the right edge
cascades to the menu's left instead and is lifted to keep its bottom on
screen, and the menu now opens on the workspace rather than over the formula
bar's label.
Re-saving a model no longer piles up cell datoms (P2). save_model
diffs cells against the store: an unchanged cell is not re-asserted, a
value left under another type's attribute is retracted, and a cell removed
from the model (cleared in the GUI, deleted through the server) is
retracted instead of lingering on disk and coming back on the next load.
An unchanged re-save of a 100k-cell model now adds no datoms (it added 200
and grew the file ~30 KB) and takes 1.3 s instead of 321 s.
improv init no longer wipes an existing store. It installs the
schema only; it used to save an empty model, which under the cell diff
above would have retracted every stored cell.

Security

External functions: the host picks the sandbox. Every body runs
Restricted unless the operator sets IMPROV_EXTFN_TRUSTED=1; a model's
pure: false no longer turns the sandbox off (S2).
External functions fail closed. Without bwrap (bubblewrap) a
Restricted body is refused with "bwrap not found; install bubblewrap or
set IMPROV_EXTFN_ALLOW_UNSANDBOXED=1"; that variable falls back to rlimits
only. A failed setrlimit aborts the spawn (S4, S18).
External functions: tighter bwrap. Only /usr, /bin, /lib,
/lib64 and /etc are bound read-only; /tmp, /home and /root are
empty tmpfs; all namespaces are unshared (no network). The child
environment is cleared to PATH=/usr/bin:/bin, HOME=/tmp and LANG, so
API tokens and cloud credentials no longer leak (S5, S6).
External functions: bounded output. stdout and stderr are drained
concurrently and capped at 1 MiB each; output over 64 KiB no longer
deadlocks into a timeout (S7).
External functions: the timeout kills the process group, so forked
grandchildren die with the body (S8).
Wasm functions: a fuel budget stops runaway loops (the worker thread
ends), linear memory is capped at 64 MiB, and bodies must start with
wat: or base64: (a bare path is no longer read from disk) (S9).
Python functions: the body is passed verbatim as JSON data instead of
inside an escaped raw string, so backslashes keep their meaning (S20).
Server: the refresh scheduler is off by default (IMPROV_SCHEDULER=1
enables it) and runs only external functions listed by digest in the
IMPROV_EXTFN_ALLOW file, always in the Restricted sandbox. Serving a
shared model no longer runs its code (S1).
Server: request limits. 10 s header-read timeout (slow-loris), 30 s
request timeout, 64 requests in flight, 1 MiB bodies, 10,000 edits per
/eval (S10, S13).
Server: write validation. Writes need an existing input measure, a
coordinate covering exactly its categories, and finite values. Writes are
serialized, so concurrent ones no longer lose updates. New
IMPROV_API_TOKENS_RO read-only tokens (S12).
Server: internal errors return {"error":"internal error","request_id":N}
and log the detail, instead of echoing store paths (S14).
Server: auth hardening. Constant-time token comparison; open mode on a
non-loopback address needs IMPROV_ALLOW_OPEN=1 (S15).
Server: model cache. Requests share one cached model, reloaded only
after a write, instead of loading the store up to three times per request
(P4).
SQL refresh is read-only. A stored SqlSource.query must be a single
SELECT/WITH statement (check_read_only_sql; stacked statements are
refused), and every backend executes it read-only: Postgres in a
READ ONLY transaction, DuckDB in a READ ONLY transaction with
enable_external_access = false (no read_csv('/etc/passwd'), URLs or
ATTACH), SQLite only if sqlite3_stmt_readonly. A data-modifying CTE
(WITH x AS (DELETE ...) SELECT ...) is refused by the server.
Postgres fails closed on cleartext: this build has no TLS, so
connect_postgres refuses a non-loopback host unless the connection
string says sslmode=disable explicitly, and refuses sslmode=require
(it cannot be honoured). Use an SSH tunnel to a loopback port.
Postgres NUMERIC columns now import as numbers (they failed to decode).
CSV export neutralizes formula injection. Text cells, item names and
headers starting with =, +, -, @, tab or CR are written with a
leading ', so Excel/LibreOffice show them as text instead of running
them (S16). Numbers are untouched. ExportSpec { neutralize_formulas: false, .. } with export_measure_csv_with turns it off for an exact
round trip.
CSV import is size-capped. A field over 1 MiB, a record over 64 MiB
(checked while streaming, so an unterminated quote can't buffer the
file) or more than ImportSpec::max_records records (default
10,000,000) is an error instead of an out-of-memory (S17).
CNL parser depth bound (S3). The nl_formula recursive-descent parser
had no nesting limit: one ~200 KB POST /nl/parse of nested parens
overflowed the stack and aborted improv-server. Paren nesting plus
operator count is now capped at 200 (as in the formula parser), returning
the new NlError::TooDeep; tested on a 2 MiB stack with 100k parens.

Navigation Menu

Repository navigation

v0.8.0

Choose a tag to compare

Sorry, something went wrong.