Speculative Lazy Grid

Design: a lazy grid

Speculative design. Open, not settled: thinking and proposals, not shipped behavior. See Roadmap for what's in flight.

Status: accepted (the maintainer, 2026-10-08); queued as box-primitives slice 2b, after slice 2. Answers in §7. One box primitive for showing a massive, unknown-sized or infinite collection as rows, in every environment. Builds on Speculative-Box-Primitives (decisions 9 and 10, slice 2's GridBox rows interface and PaneBox), Speculative-Sessions-as-Data (calls by message) and the current <notatio-collection-table>. Companion: Speculative-Lazy-Engine.

The short version. CollectionTable (and any TableView) lowers to a TableViewBox whose rows come from a row source: rows(range) to boxes, a count that is exact, a growing lower bound or infinite, and random access when the collection has At. The web draws it as one custom element that windows rows dgrid-style; the TUI as a pager; a static environment through reduce, which pins a first page and a Skeleton for the rest. Sort and filter are expressions over the source (SortBy, Filter), so each says honestly what it costs, and declines where it can't answer.

1. dgrid: keep and drop

dgrid's OnDemandList is the model; most of the rest of dgrid is not.

Keep

• Windowing over placeholders. Rendered rows sit between a top and a bottom preload node whose heights are estimated rows × estimated height, so the scrollbar is roughly right before anything loads.
• Queries sized by the viewport, with bounds. minRowsPerPage/maxRowsPerPage cap a request, bufferRows overscan, a paging delay debounces a fling.
• farOffRemoval. Rows a viewport or more away are evicted, so memory follows the view, not the history of the scroll.
• Measured correction. The estimate is replaced by the mean of measured rows, and the placeholder resized without moving what's on screen.
• Unknown totals. A store may answer a range without a total, and the list still scrolls.
• Row identity from the store, not the DOM (we have it free: the source index).
• Keyboard and selection as small layers: one tab stop, arrows within, one selection.

Drop

• Editor, drag and drop, column reorder/resize/hide, compound columns, cell selection, the Tree mixin (a TreeDiagram or an OpenerView row can come later).
• Store observation (live inserts and removals): our collections are values, and a new view is a new expression.
• The dojo/declare mixin lattice: our layers are options on one box, drawn by one renderer.
• ColumnSet's independently scrolled column groups; one frozen index column is enough.

2. The box

Wolfram precedent. TableView lowers to TableViewBox (its options TableViewBoxHeaders, …ItemSize, …Dividers, …Alignment, …Background, …ItemStyle are in our vocabulary), a scrolling grid distinct from GridBox. Pane with Scrollbars and ScrollPosition is the scroll; Dataset pages with MaxItems; Short and Skeleton («n») elide. We take TableViewBox read-only (its spreadsheet editing is out of scope), the Scrollbars/ScrollPosition/MaxItems options, and Skeleton for what isn't shown.

Why a box of its own and not a GridBox in a PaneBox, as decision 10 reads: a GridBox is eager layout its parent draws as CSS grid; this box meets all four of §2's element criteria (arrow-key input, its own lifecycle and size, a grid role, a focus stop), and its rows are not operands. It reuses GridBox's cell layout and slice 2's rows interface for each window, and behaves as a PaneBox for its own size.

CollectionTable(SymmetricGroup(6), Columns -> [Descents, CycleCount])
  → TableViewBox(
      RowSource(SymmetricGroup(6), [Descents(_), CycleCount(_)]),
      TableViewBoxHeaders -> [ItemBox("#"), ItemBox("Descents"), ItemBox("CycleCount")],
      Scrollbars -> (False, True), ScrollPosition -> DynamicBox(_pos), Selection -> DynamicBox(_s),
      MaxItems -> 20)

RowSource is held, not evaluated, as decision 9's producer is: the view is the expression.

The row source contract

type RowCount =
  | { kind: "exact"; n: bigint }
  | { kind: "atLeast"; n: bigint; upper?: bigint; growing: boolean } // a scan in progress
  | { kind: "infinite" };

interface RowSource {
  count(): RowCount;                       // cheap; never walks the collection
  rows(range: Range, columns: ColRange, signal: AbortSignal): Promise<RowBatch>;
  readonly random: boolean;                // At exists: any index is O(cost of At)
  readonly columns: readonly ColumnSpec[]; // label, width estimate (em), numeric?
  readonly rowHeight?: number;             // estimate in em; the drawer corrects it
}

interface RowBatch {
  readonly start: bigint;                  // source index of the first row
  readonly rows: readonly (readonly Box[])[]; // one Box per visible column
  readonly count: RowCount;                // may have moved: a scan found its end
  readonly end?: boolean;                  // the collection ended inside this range
}

• Indices are bigint. SymmetricGroup(25) has more rows than MAX_INDEX; today's pager stops at 2^53. A row's key is its source index, the one At(expr, #) reproduces.
• Random vs sequential. CE's at handler is optional (IndexedCollectionHandlers). With it, any range is direct. Without it, the source walks the iterator and keeps a prefix cache (CE's elementMemo already memoizes walks of Map/Filter/Tabulate), so a range past the cached prefix costs the walk to it, and the source reports progress.
• Order. CE's iterator order is undefined, and a table needs a stable one. A source is built only over an indexed collection or one with a first-class order (Speculative-Options-Restrictions-and-Orders); an unordered carrier with no order declines ("no row order").
• Rows are boxes. The kernel lowers each cell by makeBoxes, so a permutation cell is its value picture (a GraphicsBox) or its InputForm, the same as anywhere else.
• Column virtualization. columns is a range too: wide data (a Tabulate of 500 columns, a matrix) asks only for the columns in view, with the index column frozen.
• Height. rowHeight is a hint (one text line, or a glyph's height); the drawer measures rendered rows and keeps a running mean, falling back to per-row heights only where a column draws graphics of varying size.

3. Drawing it

Web: <notatio-table-view>

One custom element, named for the head per §2's "tags follow heads". It observes its size, takes focus, and windows:

• Window = viewport rows + BufferRows above and below (default 10), never fewer than MinRowsPerRequest (25) or more than MaxRowsPerRequest (250) per call. Placeholders above and below hold the estimated height.
• Page cache keyed by aligned blocks of rows (64) so overlapping requests share; blocks further than FarOffRemoval (two viewports) from the view are evicted, except the block holding focus or the selection.
• To the worker kernel. The element registers its RowSource once (the session holds it by handle, so later calls carry {handle, start, end, cols}, not the expression), batches the frame's missing blocks into one message, and keeps at most one request in flight plus one queued. A scroll that moves the window aborts the in-flight call (the kernel's abort, by request id) and drops late answers by generation, as notatio-collection-table does now.
• Debounce while the scroll velocity is high (a fling): placeholders show row indices only, and rows fill when it settles.
• Scale. Browsers cap an element's height (roughly 17M–33M px). Past a cap the scroll height is clamped and scrollTop maps to an index linearly with a scale factor; wheel and arrows move by rows, not pixels, so fine motion stays exact (§4).

TUI: a pager

notatio show at a TTY draws a TableViewBox with layout's cell metrics, one window the terminal's height less chrome. j/k a row, space/b a page, h/l a column window, g/G the ends (G asks for an index when the count isn't exact), : jump to an index, / a filter, q out. The engine is in-process, so rows is a direct call; the same AbortSignal cancels a walk when the user moves on. A pipe is static (below).

Static: reduce pins a page

A print, an SSR page or a pipe can't scroll. reduce pins ScrollPosition (default 1) and MaxItems, and the box becomes a plain GridBox: those rows, then one Skeleton row:

count	last row
exact	… 719,980 more (Skeleton(719980))
at least n	… at least 4,096 more
infinite	…

toText and toMarkdown draw it as they draw any GridBox.

Native (later)

UITableView, RecyclerView and LazyColumn are this box: a data source by index with estimated heights. The row source maps onto them through the native.Hint -> "list" path (Speculative-Native-Rendering); an unknown count maps to their paging adapters.

4. Unknown and infinite counts

• The thumb isn't proportional, and says so. For exact it is (within the scale cap). For atLeast and infinite the scroll height is the explored extent plus a runway of a few viewports; reaching the runway asks for more and grows it, as an infinite feed does. The thumb then means "where in what's been seen", and the status reads rows 1,201–1,220 of ≥ 4,096 or of ∞.
• Coarse position is an index, not a pixel. A jump field (and : in the TUI) takes an index; with random it goes there directly. Without it, the jump walks forward with progress and a cancel, and refuses past a walk budget rather than freezing.
• Growing totals. A filter scan (§5) or a sequential walk moves atLeast.n up; a batch that reaches the end turns the count exact and the runway collapses to the real end, keeping the view's top row where it was.
• Past 2^53. Exact counts print as now (≈ 2.43 × 10¹⁸), and an index jump takes an exact integer, not a float.

5. Sort and filter

Both are expressions over the source, so the view is SortBy(Filter(coll, p), f) and the table just draws it; the cost and honesty follow from what each head can promise.

• Filter makes the count atLeast until the scan has covered the source: a finite source gives upper (k matches plus the unscanned remainder), an infinite one none. A scan runs in budgeted slices in the kernel; a range that finds no match within its budget returns a marker row (scanned 20,000 more, no match: continue?), never a hang. At on a filtered view is a walk, so a filtered view is sequential even over a random source, unless the filter has an index of its own (an At handler on the filtered head, such as a ranked restriction).
• Sort needs every key. A finite source within SortLimit materializes the key column (in the kernel) and sorts; a larger one either sorts a prefix (today's note, "over the first 2,000") or, better, keeps a top-k heap over a full budgeted scan, so the first page is the true first page once the scan ends and is labeled until then. An infinite source declines to sort: the header says why. The exception is a key the collection's own order already ranks (index order, or a declared monotone statistic), which is not a sort at all.
• Sort then filter, or filter then sort, is the user's expression; the table doesn't reorder it.

6. Accessibility

• role="grid" on the element (it takes arrow keys), role="row"/"gridcell", a header row of columnheaders with aria-sort on the sorted one.
• aria-rowcount: total + 1 (the header) when exact, -1 when atLeast or infinite; each rendered row carries aria-rowindex (source index + 1). With column virtualization, aria-colcount and aria-colindex likewise.
• Focus survives recycling. One tab stop (roving tabindex); the active cell is state, (index, column), not a node. The block holding it is pinned against eviction, so a focused row scrolled away stays in the DOM; when focus jumps (: / jump field) the new row is fetched and focused when it lands.
• Placeholders are aria-busy; a polite live region announces a settled window (rows 41 to 60 of at least 4,096), not every scroll frame.
• Selection is a DynamicBox binding, as GraphicsBox's is: a selected row is announced and its expression available to the rest of the page.

7. Where it sits, and open questions

Slice 2b, after slice 2 (GridBox rows interface, PaneBox) and independent of slice 3: TableViewBox and RowSource in boxes; the row-source call in the kernel protocol (register, range, abort); <notatio-table-view> on the web; the pager in cli; reduce to a GridBox plus Skeleton. CollectionTable lowers to it, and <notatio-collection-table> keeps only its editors (columns, filter, expression) as chrome. Once slice 3's InputFieldBox lands those editors are boxes too and the element retires; frontend/src/collection-table.ts's column and carrier logic moves to the kernel side of RowSource. Pass: CollectionTable(NonNegativeIntegers) scrolls without end on the web and in the TUI, and SymmetricGroup(25) jumps to its last row.

Answers (2026-10-08)

• TableViewBox, not a lazy GridBox (question 1); Speculative-Box-Primitives decision 10 is amended.
• Scroll by default (question 3).
• Infinite and unknown counts (question 4): the view is a fixed-height viewport by default (ImageSize), scrolling inside itself, so the page around it never reflows as rows load or the runway grows; the element sets contain: strict. The runway and the jump field live inside that viewport.
• The row source is a held head in the box, a handle only on the wire (question 2).
• Questions 5–8 take their leans.

Open questions (as posed)

1. TableViewBox, or a lazy GridBox in a PaneBox (decision 10's wording)? Lean: TableViewBox. It is a different kind of thing (a focus stop with its own lifecycle), Wolfram draws the same line, and a GridBox stays plain layout.
2. Is RowSource a head (data, held) or a kernel-side object (a handle)? Lean: a held head in the box, so reduce, the TUI and SSR see the expression, and a handle only on the wire.
3. Paging or scrolling by default? Dataset pages; dgrid scrolls. Lean: scroll on the web, with the pager's readout kept as the status line; Pagination -> True restores pages.
4. Infinite scrollbar: runway or a log-scale scrubber? Lean: runway plus the jump field; a log scrubber is cute but nobody can read its position.
5. Sort over a large finite source: prefix or top-k? Lean: top-k over a budgeted full scan, labeled until complete; the prefix sort goes.
6. Variable row heights (graphics cells). Lean: uniform per-column estimate, corrected by a running mean; a per-row height table only if a column declares graphics.
7. Where the row cache lives: page or kernel? Lean: rendered boxes cached in the page (evicted far off), elements memoized in the kernel by CE's elementMemo; nothing in IndexedDB.
8. Do we want Wolfram TableView's editing ever? Lean: no; a cell editor would be a different head over a List, not a lazy view.