Materialize libghostty scrollback into the Emacs buffer#73
Merged
Conversation
dakra
force-pushed
the
scrollback-in-buffer
branch
from
19d22d6 to
8196244
Compare
There was a problem hiding this comment.
Pull request overview
This PR changes ghostel’s rendering model so libghostty scrollback is continuously materialized into the Emacs buffer, enabling buffer-based search/navigation over full terminal history without entering copy mode, while simplifying copy mode and bounding memory usage via a lower default scrollback cap.
Changes:
- Track and synchronize libghostty scrollback rows into the Emacs buffer during each redraw (including trimming when libghostty evicts rows).
- Simplify copy mode to freeze live redraw and rely on normal Emacs buffer navigation (removing “load all scrollback” flows).
- Reduce default
ghostel-max-scrollback(and align benchmark configuration) to limit the new Emacs-side heap cost.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/render.zig |
Implements growing-buffer redraw, scrollback promotion/trim, viewport-anchored rendering, and bounds URL detection to viewport. |
src/terminal.zig |
Adds scrollback_in_buffer state and resets it on resize due to reflow. |
src/module.zig |
Lowers default scrollback size, removes full-scrollback redraw API, and clears buffer on resize to force rebuild. |
src/emacs.zig |
Adds char-before to the pre-interned symbol cache used by render logic. |
ghostel.el |
Removes full-buffer copy-mode machinery, switches scroll wheel behavior to Emacs scrolling, bounds URL detection, and preserves point while reading scrollback. |
test/ghostel-test.el |
Adds tests for in-buffer scrollback growth and property preservation; updates copy-mode/clear-screen tests to match new model. |
README.md |
Updates user docs for always-materialized scrollback, new default size, and revised benchmark numbers. |
bench/ghostel-bench.el |
Fixes scrollback unit mismatch by converting benchmark “lines” into bytes for ghostel--new. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
dakra
force-pushed
the
scrollback-in-buffer
branch
from
50d3098 to
257fdeb
Compare
6 tasks
The Emacs buffer now mirrors libghostty's full scrollback above the viewport, so isearch, consult-line, swiper, occur, and any other buffer-based command work over the entire history without entering copy mode. How it works (src/render.zig redraw): - Track scrollback_in_buffer in the Terminal struct (src/terminal.zig). - Each redraw polls libghostty's total_rows. When new rows have scrolled off, promote the existing top viewport rows to scrollback by bumping the counter -- the buffer text is never touched, so any text properties applied while the row was the viewport (URL detection, ghostel-prompt) survive automatically. - Bootstrap fallback: when the buffer doesn't have enough viewport rows yet (cold start, post-resize, large bursts), fetch the remaining rows from libghostty via insertScrollbackRange, which pages the libghostty viewport across the requested range. Each row is built into text_buf with its trailing newline appended in-buffer so the row + newline goes through a single env.insert call. - forward-line counts the position past the last char as a moveable "line" when the buffer doesn't end in \\n -- that position is the terminal cursor row, not a real scrollback row, so detect it via char-before and decrement the promotion count. - Trim from the top when libghostty's scrollback cap evicts old rows. - Anchor the viewport render at viewport_start_int instead of point-min: deleteRegion(viewport_start, point-max) on full redraw, forwardLine offsets on partial redraw, applyHyperlinks gets a viewport_start parameter, cursor positioning uses it as base. - ghostel--detect-urls now takes optional (begin end) bounds and the redraw passes the viewport region only -- avoids O(N^2) scans of the growing buffer. Copy mode is now just "freeze the redraw timer + swap keymap": - Removed ghostel--copy-mode-full-buffer state and the ghostel-copy-mode-load-all command (and its C-c C-a binding). - Removed ghostel-copy-mode-auto-load-scrollback custom and the ghostel--redraw-full-scrollback module binding. - Copy-mode scroll/navigation commands use plain Emacs primitives (scroll-up/down, forward-line, recenter) instead of toggling libghostty viewport scrolling. - ghostel-copy-mode-exit moves point to point-max before invalidating so the redraw is allowed to position point at the terminal cursor (otherwise the new point-preservation logic would keep point in the scrollback region where the user was navigating). Live-output point preservation: - Track the terminal row count as ghostel--term-rows (set when the terminal is created/resized). - ghostel--delayed-redraw saves point as a marker when point is in the scrollback region (above the last term-rows lines) and restores it after the redraw, so scrolling up to read history during live output no longer yanks the user back to the cursor. Resize: - fnSetSize now erases the buffer after term.resize, and Terminal.resize resets scrollback_in_buffer = 0. Reflow invalidates the row layout, so the next redraw rebuilds via the bootstrap path. Default scrollback lowered from 20 MB to 5 MB: - The new growing-buffer model materializes scrollback into the Emacs buffer with text properties on every cell, so the cost is no longer just libghostty's compact byte storage. - Streaming throughput is bounded by scrollback size: at 20 MB it runs at ~25 MB/s, at 10 MB ~38 MB/s, at 5 MB ~48 MB/s, at 1 MB ~59 MB/s. 5 MB still holds ~5,000 rows on a typical 80-column terminal -- plenty for build outputs, grep results, log tailing -- while keeping the per-redraw cost close to the no-scrollback baseline. - Updated the docstring, README, and the C default fallback in fnNew to reflect the new memory model. Bench fix (bench/ghostel-bench.el): - ghostel-bench-scrollback is documented as "scrollback lines" and passed to vterm/term as lines, but ghostel--make-ghostel was passing it directly to ghostel--new which interprets it as bytes. At the default value of 1000 this meant vterm got 1000 lines of scrollback while ghostel got 1000 bytes (effectively zero rows), giving ghostel an unfair head start in the comparison. Convert to bytes (* 1024) so all backends test against ~1000 lines. README perf numbers refreshed against the fair comparison and the new growing-buffer model: ghostel 64 MB/s (was 72), vterm 28 MB/s (was 33). Ghostel is still ~2x faster than vterm on PTY plain ASCII. The drop versus the old 72 MB/s number reflects both (a) the new model paying a small per-redraw cost for scrollback bookkeeping even at zero scrollback and (b) the bench fix making the comparison honest. Tests (test/ghostel-test.el): - New: ghostel-test-scrollback-in-buffer (12 rows into a 5-row term, assert scrolled-off rows live in the buffer). - New: ghostel-test-scrollback-grows-incrementally (two-batch scroll, assert all earlier rows survive subsequent redraws -- exercises the partial-promotion path). - New: ghostel-test-scrollback-preserves-url-properties (write a row with a URL, redraw, scroll the row off, redraw, assert the URL's help-echo property is still attached -- the regression test for "URLs in scrollback are clickable"). - Removed ghostel-test-copy-mode-load-all (function gone). - Replaced ghostel-test-copy-mode-full-buffer-scroll with ghostel-test-copy-mode-buffer-navigation. - Rewrote ghostel-test-copy-mode-recenter from 60 lines of mocks to 6 lines verifying it delegates to recenter. - Updated the scroll-event tests to mock scroll-up/scroll-down. - Updated ghostel-test-clear-screen to check the materialized scrollback directly instead of relying on the legacy viewport-scroll behavior.
Follow-up correctness fix on top of the scrollback-in-buffer commit, in a separate commit so it can be reverted independently. When libghostty's scrollback hits its byte cap and starts evicting the oldest rows in lockstep with new ones being pushed, the row at scrollback index 0 changes underneath us. The normal delta-detection in redraw() tracks `total_rows` deltas, but those don't capture content rotation — and worse, the existing trim path (delta < 0) removes our top rows under the assumption they match the rows libghostty just evicted, which isn't true after rotation has shifted the content. User-visible symptom: after sustained streaming past the cap, isearch / consult-line over the buffer's scrollback returns rows that no longer exist in libghostty. Fix: - Add `wrote_since_redraw: bool` and `first_scrollback_row_hash: u64` to the Terminal struct. - vtWrite sets `wrote_since_redraw = true`. The end of redraw clears it. resize() also clears `first_scrollback_row_hash` because reflow invalidates the row content. - Add `computeFirstScrollbackRowHash` helper: scrolls libghostty's viewport to the top, reads the first row's first ~16 cells, mixes them into an FNV-1a 64-bit hash, restores the viewport. Six libghostty calls — cheap, gated to only run when rotation is suspected (writes happened + we have scrollback + we have a stored hash). - At the start of redraw, before the existing delta sync, run the rotation check. If the stored hash differs from the freshly sampled hash, libghostty's scrollback has rotated underneath us: eraseBuffer, set scrollback_in_buffer = 0, force a full redraw. The delta-sync below will then see libghostty_sb - 0 = libghostty_sb and refetch everything fresh via insertScrollbackRange. - After the delta-sync, update the stored hash whenever we have scrollback so the next redraw has a fresh baseline. Why hash-the-row instead of comparing counts: libghostty's total_rows is allowed to plateau OR shrink when the cap is hit (it depends on page allocation and eviction semantics). Counter comparison alone misses the case where total_rows is steady at the cap with content rotating, and is wrong in the case where total_rows shrinks while content has actually rotated. Sampling the first row's content is the only signal that always tracks rotation correctly. Test (test/ghostel-test.el): - ghostel-test-scrollback-rotation-rebuild — write 5000 EARLY rows into a tiny-cap terminal (libghostty saturates at ~920 rows) + redraw, then write 5000 LATE rows without an intervening redraw, then redraw. The second redraw must detect rotation and rebuild so the buffer no longer contains any "early-" markers and shows the most recent late- rows. A previous version of this commit also included a "batched multi-row insert" optimization in insertScrollbackRange that collapsed N per-row env.insert calls into roughly N/page_rows calls. Empirically it only bought ~2-8% on the streaming bench because libghostty's vt_write parsing dominates redraw cost in that workload, not Elisp FFI. The added complexity (~140 lines: new RowMeta struct, new flushScrollbackChunk helper, cumulative char_offset arithmetic, chunk overflow handling, oversized-row fallback) wasn't worth the small gain. The batched-insert patch is preserved at .claude/batched-insert.patch and can be re-applied with `git apply .claude/batched-insert.patch` if the streaming hot path becomes more important later.
When a row's encoded bytes exactly fill text_buf, the in-buffer newline
append was skipped, leaving the row without a trailing newline. The
prompt/wrap property math downstream assumes `after_insert - 1` points
at the row's newline, so a missed newline would misapply those
properties to the last character of the row instead.
For a standard 80-column terminal the max encoded row is ~321 bytes
(far below the 16 KB buffer), so this was only reachable on pathological
column counts — but the invariant shouldn't depend on that. Fall back
to a separate env.insert("\n") when the newline couldn't fit in the
text buffer, so the "one row per line" contract always holds.
`buildRowContent' now walks the row cells and tracks the position right after the last non-blank cell, then truncates `byte_len' and `char_len' back to that position before returning. A cell is considered blank when it carries no grapheme (libghostty's unwritten-cell padding) and has default style; cells the terminal explicitly wrote — even spaces — anchor the trim point and are preserved. Cells with non-default style (colored background, underline, …) are also preserved so visible styling is not lost. This removes libghostty's full-terminal-width padding from the Emacs buffer. Short prompt rows no longer run out to column 80 with trailing spaces, and the right edge of the buffer reflects the actual last character written by the terminal. Style runs that extend past the new trim point are clipped by `insertAndStyle''s existing `content.char_len' cap — no change required there. `prompt_char_len' is capped at the new `char_len' so the leading-prompt region never points past the end of the trimmed text (fixes an out-of-range text-property call that would fire when the prompt ran to the viewport edge without further input). Three tests updated to match the new semantics and a new test added: - `ghostel-test-scrollback-in-buffer' now expects 12 lines instead of 13 (the trailing empty cursor row trims to nothing). - `ghostel-test-incremental-redraw' expects 4 lines instead of 5 for the same reason. - `ghostel-test-wide-char-no-overflow' asserts the visual width is 2 (the emoji) instead of 40 (the full terminal width). - `ghostel-test-resize-width-change-full-repaint' asserts each row is no longer than the terminal width, instead of exactly equal. - New `ghostel-test-render-trims-trailing-whitespace' covers both sides of the rule: unwritten padding is stripped, shell-written trailing spaces like `$ ' are preserved.
Re-ran `bench/run-bench.sh` at the new 5 MB default size (up from 1 MB — the larger run amortizes measurement overhead and gives more stable numbers) on Apple M4 Max, Emacs 31.0.50, post-trim. Plain-ASCII PTY throughput ticked up slightly for ghostel (64 → 65 MB/s) on top of the wider engine improvements from scrollback-in-buffer; the standout jump is URL-heavy input, which doubled from 22 to 42 MB/s since the previous README snapshot thanks to the scrollback promotion path preserving URL text properties instead of re-detecting on every scroll-off. With link detection disabled ghostel holds 65 MB/s regardless of the input mix. vterm / eat / term numbers are essentially unchanged from the previous run, re-measured for consistency.
dakra
force-pushed
the
scrollback-in-buffer
branch
from
95cacee to
1a31d37
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
isearch,consult-line,swiper,occur, and any other buffer-based command work over the entire history without entering copy mode (vterm parity)ghostel-prompttext properties survive automatically: scrollback URLs stay clickableghostel-max-scrollbackfrom 20 MB to 5 MB (~5,000 rows on an 80-col terminal) to bound the new Emacs heap costghostel-copy-mode-load-all,C-c C-a,ghostel-copy-mode-auto-load-scrollback,ghostel--copy-mode-full-buffer, andghostel--redraw-full-scrollbackBackground
Before this PR, ghostel's Emacs buffer only ever held the visible viewport (~24 rows). Full scrollback lived inside libghostty and was materialized on demand via
C-c C-a→ghostel-copy-mode-load-all. Consequence: any buffer-based search command only saw the current screen.vterm gets full-history search for free because libvterm fires a
term_sb_pushcallback for each scrolled-off row. libghostty exposes no equivalent callback, so this PR pollsgetTotalRows()against a tracker on every redraw and synthesizes the same effect.How it works
src/render.zigredraw()flow:delta = libghostty_total_rows - rows - scrollback_in_buffer.delta > 0: walk forwarddeltanewlines fromviewport_start_int. Whatever rows we walk past become scrollback simply by bumpingscrollback_in_buffer— no fetch, no re-render. Any text properties on those rows survive because the text isn't touched. (The trailing cursor row, which has no terminating\n, is detected via(char-before)and excluded so it isn't promoted as a stale row.)insertScrollbackRangefor the rest, which pages libghostty's viewport across the requested range.delta < 0(libghostty's scrollback cap evicted rows), trim from the top.viewport_start_intinstead ofpoint-min(deleteRegion(viewport_start, point-max)on full redraw,forwardLineoffsets on partial redraw).ghostel--detect-urlsnow takes optional(begin end)bounds; the redraw passes the viewport region only, avoiding O(N²) scans of the growing buffer.Performance trade
Real-world PTY benchmark, 1 MB streamed through
cat, all backends with 1000-line scrollback (the bench had a unit-mismatch bug where ghostel was getting 1000 bytes while vterm was getting 1000 lines — fixed in this PR for fair comparison):ghostel is ~7 % slower than its own pre-PR baseline (per-redraw bookkeeping) but still ~2.3× faster than vterm.
The synthetic streaming bench (in-process writes + periodic redraw) is bounded by scrollback size in the new model: at the new 5 MB default it runs at ~48 MB/s vs ~59 MB/s baseline. TUI apps (vim, htop) are unaffected — alt-screen content doesn't go through the scrollback path.
Test plan
make build— native module builds clean (zig 0.15.2)make test— 54/54 pure Elisp tests passmake test-all— 94/94 ghostel tests pass (54 elisp + 40 native module), including 3 new tests:ghostel-test-scrollback-in-buffer— scrolled-off rows live in the bufferghostel-test-scrollback-grows-incrementally— earlier rows survive subsequent redraws (exercises the partial-promotion path)ghostel-test-scrollback-preserves-url-properties— URLhelp-echoproperties survive being scrolled into the materialized scrollbackmake test-evil— 29/29 evil-mode tests passmake lint—package-lintandcheckdoccleanfind /usr -type f | head -500,M-x isearch-forwardfor an early line — finds it without entering copy modeKnown follow-ups
delta == 0and doesn't update the buffer. The buffer's scrollback can show old rows that have since been evicted. Doesn't affect normal interactive use; would need a "force-rebuild on cap-bound steady state" check to fix.insertScrollbackRange(rare — only on cold start, post-resize, or huge bursts) don't get URL detection applied. The common case (interactive use) is fine because rows pass through the viewport first.insertScrollbackRangedoes ~5 Elisp calls per row. Batching into a single multi-rowenv.insertwould meaningfully improve the worst-case streaming bench. Not pursued in this PR.