Skip to content

Logging

Jump to bottom
github-actions[bot] edited this page Oct 7, 2026 · 54 revisions

Logging

简体中文

Start with the newest log below. Use debug for frame timing, info for memory snapshots, and the bug-report checklist at the end. Crash and hang evidence has its own section; a missing crash dump does not mean a clean exit.

Paths

  • Log files: ~/.sonicterm/logs/sonicterm.log.*
  • Fatal-signal fallback path: ~/.sonicterm/logs/sonicterm.log
  • Panic artifacts: ~/.sonicterm/logs/crashes/
  • Session markers: ~/.sonicterm/logs/sessions/
  • Bounded breadcrumbs: ~/.sonicterm/logs/breadcrumbs/

tracing-appender uses daily names such as sonicterm.log.YYYY-MM-DD; the file with the newest modification time is active. Size rotation may add a Unix-time suffix. On Windows, ~ means the current user's profile directory. Native runtime smokes on macOS, Windows, and Linux use the explicit logs/ child of SONICTERM_RUNTIME_SMOKE_DIR instead of the user log tree; their separate config/ child is used for config/reload state, and HOME is preserved. The outer runner removes inherited NO_COLOR and retains failed stdout/stderr plus SonicTerm logs for CI artifacts.

Performance scenario runs, the perf_scenarios example that scripts/perf-compare.py drives, also stay out of the user log tree: each run logs into a new scratch directory under the OS temporary directory, and HOME is unchanged. The harness refuses an inherited RUST_LOG, which would replace the configured level, so both sides of a comparison log through the same filter, and it logs its scratch path at startup. The comparison parses the memory snapshot line (Aggregate snapshot at info) and, in --laps runs, the [render_timing] line (Render and performance diagnostics). Development and Release describes how each run shows that ~/.sonicterm did not change.

Configuration and retention

[logging]
level = "warn"                    # error | warn | info | debug
max_file_size_mb = 10
max_rotated_files = 3
max_age_days = 2
max_crash_dumps = 10
max_crash_age_days = 2
max_crash_bytes = 10485760        # 10 MiB
max_breadcrumb_files = 10
max_breadcrumb_age_days = 2
max_breadcrumb_bytes = 1048576    # 1 MiB

warn is the default. SonicTerm reads [logging] before installing the tracing subscriber, so the configured level applies to normal startup. RUST_LOG overrides the configured filter for one run. Stderr follows that same filter with an added global warn fallback; more specific target directives can still emit debug lines there, so warn is not a hard stderr ceiling.

Before the appender opens, SonicTerm rotates an active log over max_file_size_mb unless the value is 0, then applies age and count limits to older log files. The active file is never deleted. Crash and breadcrumb artifacts are each bounded independently by count, age, and aggregate bytes; oldest files are removed until all enabled limits are satisfied. Setting an age or aggregate-byte limit to 0 disables that axis. Cleanup is fail-soft and artifact cleanup runs on a background thread.

Levels and diagnostic targets

Level What it admits
error errors only
warn warnings, errors, sonic_exit, sonic::gpu device records, and user-visible reclamation/exhaustion warnings
info normal SonicTerm information plus the aggregate memory snapshot
debug detailed SonicTerm diagnostics, pane/renderer memory lines, state-machine events, render_timing, tear_out_timing, and frame_counters

wgpu, naga, sonicterm-vt, and sonicterm-grid remain warning-oriented in the configured filters. Very hot font-shaper dumps are trace; no configured level admits them. Use a targeted RUST_LOG directive only when investigating that path.

At debug, the sonicterm_app::sync_output target records each synchronized update (DEC 2026) a pane's worker released at the 150 ms bound, with its pane_id and epoch: synchronized output held past 150 ms; released.

Font diagnostics

A missing configured font emits an error on the config target, naming its family, weight, stretch and style. SonicTerm uses fallback fonts, so a running window does not prove that the requested face loaded. Check [font].family in sonicterm.toml and whether that font is available to SonicTerm; the diagnostic links to the English Configuration page, which has a language switch. Synthesized bold/italic requests and fallback-only entries do not add missing-font errors.

With the configured warn, info or debug filters, config errors reach stderr but not the log file or crash history. RUST_LOG replaces the configured filter rather than extending it. This capture recipe retains the default warning filters and adds font-configuration errors to all three outputs:

RUST_LOG=config=error,sonic_exit=warn,sonic=warn,sonicterm=warn,sonicterm_vt=warn,sonicterm_grid=warn,memory::reclaimed=warn,wgpu=warn,naga=warn

To preserve an existing custom filter, append config=error to its complete value instead. The configured error filter also admits these errors. Repeated errors can describe separate font resolutions; these errors have no deduplication policy.

A missing-glyph warning instead reports the number of unresolved codepoints and placeholder rendering without including the requested text. It recommends installing a covering font or changing [font].family, using the same SonicTerm configuration page. Its existing per-generation/hour warning throttle is independent of missing-font errors. Neither message establishes the cause of GPU software fallback; adapter diagnostics are described below.

Local-path click diagnostics

Enable [logging] level = "debug" before reproducing an explicit local-path click failure, or use RUST_LOG=sonicterm_app::app::path_target=debug for one run. The local path activation unverified event records a click whose detected explicit path has no current authorized filesystem selection. It uses the immutable click snapshot, without re-reading the filesystem or the parser. A confirmed-missing auto-detected path also logs this event; the click then stays an ordinary terminal click.

Field Meaning
window_id, pane_id, pointed, view_top Clicked window/pane, absolute cell, and viewport origin
screen_epoch, scrollback_evicted Screen and retained-history identity
cwd, cwd_revision That pane's OSC 7 authority/path and revision, not the process CWD
clicked_path Explicit path text associated with the click
candidates Bounded candidate set with typed provenance, resolved paths, and cell spans, in probe order
reason Current probe failure key, or path-error-pending when no matching failure is available

A pending result is not evidence that the file is missing. The event contains paths, which can be sensitive, but no whole terminal rows or environment dump; review it before sharing. Paths use escaped debug formatting, and one event may be large even though candidate enumeration is bounded. Default warn logging does not emit it. Hover and unverified bare-name clicks do not emit it either.

An absent event proves nothing about success or failure: logging may be disabled, no target may have been detected, or a rejected target/native-open failure may have followed another branch. Native-open failures retain their separate path open failed warning. To investigate a relative-path failure, compare the same file's relative path, absolute path, and local file URI in the same pane; retain the matching click identity and do not infer its CWD from another shell.

PTY input rejection diagnostics

The default warn level reports input that was refused, including terminal parser replies. The producer assigns source: Keyboard, Paste, FileDrop, Ime, PointerButton, PointerMotion, Wheel, FocusReport, TerminalReply, ScriptDraft, or StateMachine. Sources are never guessed from payload bytes.

Field Meaning
pane_id Stable pane identity supplied at the producer
window_id The pane's current window when the event loop handles the rejection; absent after pane closure or when the event loop is unavailable
source, rejected_bytes, reason Input category, refused byte length, and payload-free reason
observation="concurrent" Queue and writer fields are independent observations, not one rejection-time transaction
queued_messages, queued_bytes, queue_capacity Waiting message count, payload byte count, and four-slot limit; excludes the active native write
writer_phase Idle, Writing, Flushing, or Stopped; a boundary observation, not a child-health verdict
in_flight_bytes, in_flight_millis Active message size and time spent in the observed write or flush; time is absent when idle/stopped
completed_messages Native writes whose write_all and flush both succeeded

The event carries no rejected payload. Its debug representation, warnings, and notification never include typed text, commands, paths, or clipboard content. Notification follows the pane's current window after tab transfers; a closed pane produces a warning but no notification on an unrelated window. When the proxy is absent or event delivery fails, the producer logs the same metadata without a current-window identity.

Production terminal replies enter a separate FIFO with 64 KiB RAM and private temporary-file spill, outside parser and side-effect locks. UI queue saturation does not discard replies, trigger rejection warnings, or stop output processing. Storage/native failure is latched and reported once as terminal reply delivery failed with pane identity and byte count; output, redraw, and exit observation continue. Native writer/spool read errors are independently logged. Intentional teardown releases spill storage without a rejection notice. See Terminal IO and VT for storage and delivery limits.

Four small messages can fill the channel before a healthy writer is scheduled. Controlled tests use the production admission and writer loop to demonstrate that burst draining preserves order. Separate blocked-write and blocked-flush fixtures demonstrate zero queued bytes with one in-flight message, followed by four additional queued messages and explicit refusal of the next message. These fixtures distinguish mechanisms; they do not retrospectively identify which producer or native condition caused an older un-attributed warning.

Queue capacity and per-message limits are unchanged. Each pane coalesces native pointer motion into one fixed 64-byte slot before queue admission. A full queue keeps the latest position pending and retries after 10 ms without requesting a frame; a later position replaces it. Discrete input bundles preceding pending motion into the same queue message when it fits, preserving byte order without using an extra slot. A cap-sized or refused discrete message supersedes pending motion rather than replaying it afterward. Disconnected motion is reported once and cleared. A changed mouse-tracking/encoding/screen profile invalidates deferred motion. A busy parser defers motion-only retries until its profile can be checked; a following discrete input supersedes unvalidated motion rather than delaying the key or replaying stale bytes. Pane teardown releases its slot. Other UI input still refuses rather than blocking; worker-owned replies use the spill FIFO instead. Interpret repeated observations of the same pane and progress counter rather than a single QueueFull warning.

PTY resize failure diagnostics

The default warn level reports a PTY resize the native layer refused, as one pty resize failed event.

Field Meaning
pane_id Stable pane identity of the pane whose PTY refused the resize
cols, rows Requested geometry, not the geometry the pty currently holds
error The failure rendered by Display. A native refusal shows the platform text; a zero axis shows refusing pty resize to <cols>x<rows>

The event carries no terminal content: only the pane id, the requested columns and rows, and the error.

Each pane logs only the first resize failure until a success resets its warning latch; this avoids input-rate logs during tab activation or window drags. Attempts still run: only invalid sizes and successful duplicates are skipped by the IO boundary. The grid keeps the requested geometry. There is no rollback, retry timer, or failure heartbeat.

Render and performance diagnostics

frame_collection warns once per invalid-topology episode with id (window) and reason when duplicate/missing leaves or active/zoom disagreement prevent a complete frame. The latch resets only after a complete held frame passes viewport reconciliation, not after valid source capture alone. Repeated post-lock validation failures therefore stay in the same warning episode. Closing-tab NoLayout is silent; ordinary lock contention does not emit this structural warning.

Set level = "debug", restart, and reproduce the problem. The render_timing target records frame phases including grid walking, overlay assembly, glyph upload, surface acquisition, submission, and presentation. It identifies main or child renderers, mode=full, and damaged_rows. No-op frames return before completed-frame timing is emitted. Partial GPU damage limits the draw scissor, not frame assembly. There is no separate render-timing option.

Each redraw that runs to completion writes one line for its window: [render_timing] window=<label> total=<ms>ms <lap>=<ms>ms ... tail=<ms>ms. <label> is main or child, each <lap> names a frame phase and the last is tail, and every value is milliseconds with two decimals. In the log file the line is the value of the event's line field, after line=. scripts/perf-compare.py parses it only in --laps runs, which log at debug and so write it: formatting the line costs time on every frame, so laps runs form their own set and are never pooled with timed runs, which log at info and write no render_timing line.

The same DEBUG target records renderer initialization operation boundaries. Synchronous renderer_init spans carry window_id, role, and shared; finishing a prepared startup carries window_id, role, and prepared=true instead. startup_prepare spans identify owner-thread instance and surface creation by window_id; recovery_init spans identify the requesting window_id during ContextRequest::run, including startup requests. The renderer_finish operation covers owner-thread assembly after negotiation. Operation records retain their entry span as parent. phase="enter" precedes the call; phase="return" records elapsed_ms and outcome. ok and error describe a returned Result; returned means only that the call returned, not that initialization or presentation succeeded. Surface configuration has a separate phase="gate" record with the existing device gate's accepted reading.

Operations cover instance creation/reuse, surface creation and capabilities, adapter/device requests, surface configuration, pipelines, frame storage, atlas storage/uploads, font stacks, and cell metrics. An unmatched entry can mean an unfinished call, unwind, or lost log output; it is not a diagnosis by itself. Elapsed time includes scheduling and diagnostic overhead, not just native execution. With DEBUG disabled, the timing helper reads no clock and retains no span. These records add no terminal payload, font names, paths, or environment values.

Font operations also use render_timing at DEBUG. Entry and explicit return records separate shape_impl from fallback_receive, rasterizer_new from rasterize_glyph, and uncached font resolution from metrics. font_shape spans carry loaded_font_id and retry iteration; font_raster spans carry only loaded_font_id and fallback_idx. Neither includes a glyph index or character. Renderer font_style spans identify bold, italic, and row. The Windows native font test adds a font_phase parent with window_id, scale, and the test phase; this parent is specific to that fixture.

Each queued fallback request captures its own dispatcher and parent, rather than inheriting the first request's context on the reused worker. Its font_request span carries request_id. The return-only queue_wait record measures from request-context capture to worker entry, including request preparation and worker startup when applicable. Lookup records distinguish fallback_locator, fallback_font_dirs, fallback_built_in, and fallback_selection. completion_called=true marks the point immediately before invoking the existing completion callback; false means no handles were selected and the callback is not invoked. An error outcome from fallback_receive can mean the sender disconnected because no fallback was found; it is not by itself a rendering failure.

Disabled font timing reads no diagnostic clock, allocates no request ID, and retains no span or dispatcher. A disabled request suppresses only these timing records, not ordinary worker logs, and restores the previous timing state on return or unwind. Enabled timing adds clocks and output, including a record under the existing pending-fallback lock before the callback, so it can change scheduling. These durations do not distinguish active native work from waiting or scheduling delay, and a run without a stall does not explain a previous one.

Startup logs the selected wgpu adapter, device type, and software-adapter classification. On RDP, VM, or VDI hosts, look for software-render degrade engaged and compare it with [appearance].software_render_mode on Configuration. scripts/perf-compare.py reads each scenario run's first wgpu adapter selected line, or else its first wgpu adapter reused line, for backend, name, driver, device_type, and software_rendering; a comparison names that adapter in its presenter row, and a run on another adapter than its set's first valid run makes the pair invalid. At level = "debug", each renderer also writes renderer LCD subpixel policy at startup and whenever mode, opacity, theme, or presenter state changes. Its requested, effective, windows_host, opaque_target, software_presenter, and dual_source_supported fields explain every LCD-to-grayscale fallback without relying on a screenshot.

At info, DPI transition synchronized records window_id, old_scale, new_scale, native_scale, and size_scale alongside old_inner, suggested, minimum, available, and target. renderer_before/renderer_after are physical surface pixels; cell_before/cell_after are raster-pixel cell extents. On macOS, size_scale follows the native backing scale because old_inner is already reported in that domain. The other platforms use the stored old scale. These paired inputs/outputs distinguish double scaling from a surface or cell mismatch without recording terminal content.

When the foreground-probe worker cannot start or stops, the app logs one warn on sonicterm_app::app, "foreground-process probes unavailable", with a reason field. From then on tab process names and per-tab privilege warnings resolve to no process instead of sampling; there is no retry.

Frame and lock counters

The frame_counters target holds debug-only counters for what render_timing cannot see: redraws that were deferred or found a lock busy, frame outcomes other than a presented frame, present intervals, parser lock waits and holds, flush-to-redraw delay, dispatch stalls, wake causes, foreground-worker probes, buffer uploads, row-cache hits, and shaping requests. They change no behavior.

Turning the counters on

Set [logging].level = "debug"; the Debug filter admits frame_counters. A RUST_LOG that admits frame_counters=debug works too. Each App reads the filter once, when it starts, and keeps that decision for its lifetime, so a level change takes effect only after a restart. A process that embeds the App, such as a test harness, can force an App's counters on whatever the filter, but only before that App creates its first window or pane; the App refuses afterwards. Forced counters count, and their lines appear only where the filter admits frame_counters.

With the counters off, each instrumented path makes one check and stops there: no clock read, no atomic or thread-local write, and no allocation. With them on, the App allocates once its VT statistics, its dispatch totals, one line state per window and for the app line, and one closed-windows record; each window allocates its counters and each pane one 8-byte pending-flush slot. Nothing is allocated per frame or per batch, apart from building a line at most once a second.

Line format

[frame_counters] window=<main|child-N|app> [final=1] span_ms=<ms> <field>=<value> ...

Each window writes window=main or window=child-N, where N is the window's registration order in the App, and the App writes one window=app line. Each source writes at most one line a second. Values are deltas since that source's previous line, span_ms is the time since that line, and zero counts and empty histograms are left out.

A window line follows a frame attempt, a flush the window took, or any other window event for it; a RedrawRequested that neither attempted a frame nor took a flush prints no window line. The app line follows any window or user event. Maintenance wakes (new_events, about_to_wait, a resume-time wake, and the 30-second retention wake) are counted but never print a line, and no timer is armed for one, so an App that only wakes for maintenance logs nothing.

At window close and at exit, a source with pending counts writes one last line marked final=1, and nothing from that source follows it. A closed window's totals, including its renderer's counts and the handler time of the event that closed it, move into an App-wide closed-windows aggregate, so the App's totals never drop. Every count and sum is cumulative and never reset; a reader keeps its previous snapshot and takes deltas.

Window fields

Field Unit Meaning
attempts count redraws that passed the deferral rules and went on to collect a frame
presented count attempts that presented a frame
cached count attempts that re-presented the cached frame
settled count attempts that settled without presenting
retry count attempts the renderer asked to retry
surface_retry count attempts that hit a surface retry
stopped count attempts that found the GPU device stopped
failed count attempts that failed
contention_parser count frame collections that found a visible pane's parser lock busy
contention_images count frame collections that found a visible image store busy
defer_timeout count redraws deferred because a surface timeout is pending within the frame period
defer_contention count redraws deferred by the lock-contention retry floor
defer_sync count redraws held because a visible pane's synchronized update (DEC 2026) was open, at admission or by the recheck under the parser guards
defer_streaming count redraws deferred by streaming-output pacing
stream_clock_exempt count hardware attempts for new input that settled without presenting and kept the streaming clock, so the echo they waited for is not paced from them
display_link_ticks count display-link ticks the window accepted: the tick's generation was the running link's and a Link admission was pending; stale ticks are not counted
display_link_admissions count display-link-paced streaming frames admitted by a tick
display_link_fallbacks count display-link-paced streaming frames admitted by the two-period fallback ceiling because no tick came
contention_retry_armed count lock-contention retries armed
dirt_ack_dropped count presented-frame receipts dropped at the next collection because the pane was not held, its parser changed, or its grid was resized or switched screens after assembly; output written after assembly does not drop a receipt, it only keeps the rows it dirtied. Each drop costs a later re-assembly, never pixels
native_request_redraw count native redraw requests for the window, on every request path; a dispatch's requests reach the totals when it ends, so a window line shows them one dispatch late (final=1 lines are complete)
user_request_redraw count output events serviced for the window: PaneOutput from a VT worker's flush (at most one outstanding per pane) and RequestRedraw from a harness or test, counted before the visible-output filter
redraw_requested count RedrawRequested events for the window
present_interval ms histogram time between consecutive presented frames
handler ms histogram each window_event dispatch for the window
flush_to_redraw ms histogram oldest pending flush to the first redraw of a window that shows the pane

The four defer_* counts record the rule that won. The rules are checked in that order, and a later one is never evaluated once an earlier one holds, so each deferred redraw counts once (Rendering Modes describes the retry floor).

The three display_link_* counts are always present and read 0 where no link runs (Windows, Linux, macOS before 14, the software path). Each display-link-paced streaming admission is counted once, as a tick admission or a fallback, so attempts is at least their sum. A phase is unexercised when both are 0, link-paced when only admissions are non-zero, fallback-only when only fallbacks are (display-link pacing was unavailable in that run), and mixed when both are. display_link_ticks minus display_link_admissions counts unused ticks: ticks refused by a timeout or contention rule, replaced before use, or spent by an admission that was not streaming. The counts show which path authorized each frame; they do not prove where frames land relative to vsync (Rendering Modes).

At each RedrawRequested, flush_to_redraw takes the pending flush of every pane the window shows: the active tab's panes, or the zoomed pane. A hidden pane's flush waits until its tab is shown. One observation covers a group of coalesced flushes.

App fields

Field Unit Meaning
wake_init, wake_poll, wake_wait_cancelled, wake_resume_time count new_events wakes by cause: Init, Poll, WaitCancelled, ResumeTimeReached
wake_user count user_event dispatches
native_request_redraw_unregistered count native redraw requests for a window id with no registered counters, such as a window already closed; one app-wide total
about_to_wait ms histogram each about_to_wait dispatch
user_event ms histogram each user_event dispatch
new_events ms histogram each new_events dispatch
ui_parser_locks count event-loop-thread locks of a pane's parser
ui_guard_custody_ns count (ns) summed time a frame held any parser guard, from the first guard acquired to the last released, over every collection that took a guard, early failures included
ui_guard_custodies count collections that took at least one parser guard
frame_dispatch_ns count (ns) summed time from a frame's first parser guard to the return of the renderer call, over frames that reached the renderer
frame_dispatches count frames that reached the renderer
frame_dispatch_failed_ns count (ns) summed time from the first parser guard to the release, over collections that took a guard but never reached the renderer (contention, a synchronized-output hold, a reconcile failure, no renderer); their own population
frame_dispatches_failed count those failed collections
ui_parser_wait µs histogram the wait for each of those locks
fg_probe_calls, fg_probe_panes count retired event-loop probes; always 0, kept so a comparison against an older base shows that work falling to 0
fg_probe µs histogram retired event-loop probe durations; always empty, kept for the same comparison
fg_worker_probes count foreground-probe worker batches
fg_worker_panes count panes those batches covered
fg_results_stale count worker results the event loop dropped: the pane closed, its process identity changed, or its child exited
fg_worker_probe µs histogram each batch's duration

These come from the sonicterm-fg-probe worker thread; the event-loop thread never probes. On macOS a batch looks up each pane's process, re-reading its start token before and after; on Windows it takes one process-table snapshot. With the gate off the worker reads the clock once per probed pane and records nothing. Linux and other platforms capture no process identity, start no worker, and report zeros.

VT fields

The VT fields print on the window=app line. They are one App-wide aggregate that every pane's VT worker records into, with no per-pane split. A pane that closed, or whose worker finishes after it, still adds to it.

Field Unit Meaning
parser_lock_wait µs histogram the VT worker's wait for a pane's parser lock
parser_lock_wait_ns count (ns) the same waits summed in nanoseconds, over every section
parser_sections count parser sections recorded, the count parser_lock_wait_ns sums over
parser_lock_hold µs histogram how long the worker held that lock
parse µs histogram parsing under the lock
parse_bytes bytes bytes parsed
batches count nonempty output batches; a batch that takes the lock several times counts once, and each acquisition is recorded in the histograms
flushes count output flushes a worker made, with or without a target, sent or suppressed
flushes_untargeted count flushes while the pane had no redraw target; no timestamp is stored
flushes_coalesced count flushes that found an earlier flush still pending, which keeps its time
flushes_suppressed count targeted flushes that sent no event because the pane's output event was still outstanding; a send the event loop refused is not counted
sync_timeouts count synchronized updates (DEC 2026) a worker released at the 150 ms bound rather than at their reset

No identity holds between flushes, flushes_untargeted, flushes_coalesced, and the flush_to_redraw count. A pane that closes drops its pending timestamp, and separate counters are not read as one snapshot, so read each on its own. Only quiescent totals, read after every worker has finished, satisfy flushes_suppressed ≤ flushes − flushes_untargeted; a live snapshot or a phase delta can break it, and nothing checks it.

Renderer fields

The renderer fields print on their window's line. Each count belongs to the renderer that collected it.

Field Unit Meaning
vertex_bytes bytes bytes written to the vertex buffer
index_bytes bytes bytes written to the index buffer: the whole index pattern on the first frame and the first frame after the buffer grows, 0 on every other frame
damage_permille_sum permille sum of each frame's damaged share of the surface; divide by damaged_frames for the mean
damaged_frames count frames whose damage was recorded
damage_waste_permille_sum permille sum of each frame's union-rectangle share of the surface minus the share its parts exactly cover; divide by damaged_frames for the mean waste
software_frames count frames the software presenter drew, on Windows with software-render degradation
gpu_frames count frames drawn through wgpu, including degraded frames on macOS and Linux
row_cache_hits count row glyph cache lookups that hit
row_cache_misses count row glyph cache lookups that missed
shape_requests count FontStack shaping and measuring requests the renderer made
full_frames count frames whose render plan was Full; a frame whose plan was Noop is not counted
partial_frames count presented frames whose render plan was Partial (only the rows that meet the damage were assembled); counted after presentation, so divide by presented frames, not by full_frames
partial_fallbacks count Partial plans reassembled Full in the same frame because the final damage reached a row they did not assemble
row_cells_hashed count cells hashed into row glyph cache keys, one row per assembled terminal row
row_cache_invalidate_visits count kept for counter-contract compatibility and always 0: the content-keyed row glyph cache drops no row for dirt
row_cache_invalidate_us µs kept for counter-contract compatibility and always 0
recolor_glyphs_visited count glyphs examined when recoloring glyphs under the cursor, the copy-mode cursor or a search match on the frame's main glyph list: the rows whose ink meets the target plus every glyph outside the terminal rows, such as tab titles; overlay text is not counted
font_fallback_applies count frames whose font preparation applied a newer fallback notice or generation, clearing shaped rows, missing-glyph atlas entries and the tab-title width epoch once; supporting evidence that a resolved fallback face reached the screen, which tests prove by pixels
glyph_atlas_growths count glyph atlas doublings, counted at every end-of-frame check and when the renderer settles its statistics; a frame that grows the atlas starts a growth episode
atlas_growth_abandoned count growth episodes no frame presented: settled when the device stops, before a rebind replaces the device, and when the App settles a retiring or exiting window's statistics; a reset in place abandons nothing
shape_ns ns time inside every shaping and measuring request, face merging included; a request made inside a timed rasterizer call counts as rasterizing
raster_ns ns time inside every glyph-atlas rasterizer call, glyph-zero resolution, rasterizer creation and tile conversion included; shaping inside it is not counted again
raster_calls count glyph-atlas rasterizer calls, terminal and chrome text alike; an atlas hit calls nothing
raster_tiles count rasterizer calls that returned a tile with pixels; no tile and an empty tile are calls, not tiles
font_generation_applies count preparations that applied a newer generation of the fallback notice already applied; unlike font_fallback_applies it leaves out the first preparation and a replaced font stack
font_prepare_ns, font_generation_prepare_ns ns time inside frame font preparation, invalidation included, for every preparation and for those that applied a newer generation; outside every render attempt
tab_title_reuses count tab titles drawn from the title cache with no shaping, one per title per assembly pass, presented or not
tab_title_prepares count tab titles fitted and shaped on a title-cache miss, one per title per assembly pass; their shaping requests also count in shape_requests
chrome_run_reuses count chrome-run lookups (the search overlay's icon and label measures and draws) served from the chrome-run cache with no shaping, one per lookup inside the assembly pass
chrome_run_prepares count chrome-run lookups that shaped, one per lookup inside the assembly pass; their shaping requests also count in shape_requests
render_attempts, render_attempts_presented count render_releasing calls, and those that presented
render_attempt_ns, render_attempt_shape_ns, render_attempt_raster_ns ns time inside those calls, and the shaping and rasterizing time spent inside them
render_attempt_shape_requests, render_attempt_raster_calls, render_attempt_raster_tiles count shaping requests, rasterizer calls and tiles inside those calls
apply_attempts, apply_attempts_presented, apply_attempt_* as render_* the same for the render attempts that carried a fallback generation apply
assembly µs histogram CPU frame assembly in the renderer: from the frame-key check to the end of overlay assembly, before the atlas-retry check, upload, surface acquire, submit and present; one sample per assembled frame, including frames that later retry or fail to present; a Noop or skipped frame adds none. It is not the app's render lap
atlas_growth_to_present ms histogram from the start of the first frame that grew the glyph atlas to the next successful present; one sample per presented growth episode

On Windows a frame the GDI presenter draws counts as software_frames; the hosted Windows CI runner has no GPU, so its runs report software_frames and no gpu_frames. A frame presented through wgpu, including on its software adapter, counts as gpu_frames.

shape_requests counts each call to FontStack::shape_text_for_frame or measure_text_width_for_frame that the renderer makes through frame_stats::shape_request, failures included; a call skipped for empty text is not a request. It counts requests, not HarfBuzz attempts or fallback retries. A title or chrome run served from its cache makes no request.

Each fallback generation apply is carried by exactly one render attempt: a preparation that applies a newer generation owes it, and the next render_releasing call takes it, however many preparations or copies of the font token come between. A retry is a later call and carries none. A helper the same renderer opens during an attempt, such as notification text layout, joins that attempt; another renderer's work does not. Every _ns field is summed nanoseconds; perf-compare subtracts and pools them exactly and shows them in microseconds. Per phase it adds a pooled split of every attempt and of the apply attempts: the matched totals of the runs that carry them all are summed, then divided into shaping, rasterizing and remaining shares, with means per attempt.

Histograms

Every duration is a cumulative bucket histogram with an exact sum.

Unit Bucket upper bounds
ms 4, 7, 9, 12, 17, 25, 34, 50, 100, then above 100
µs 10, 50, 100, 500, 1000, 5000, then above 5000

A value equal to a bound falls in that bound's bucket. On a line, a histogram prints its bucket counts in order, its sum, its p95, and its maximum:

<name>_ms=[<count>,...] <name>_sum_us=<µs> <name>_p95_le_ms=<bound> <name>_max_le_ms=<bound>

A µs histogram uses _us in place of _ms. The sum is exact: the sum of the recorded microsecond values, at the clock's resolution and with the instrumentation's own cost included, so the sum divided by the bucket total is the mean. The p95 and the maximum are never exact. _le_<unit>=N means at or below the bound N, and _gt_<unit>=N means in the overflow bucket above the largest bound N.

Measurement boundaries

Around each lock of a pane's parser, the VT worker reads the clock four times: before lock(), as it returns, after parsing, and after the keyboard-snapshot store, before the guard drops. The wait is the first interval, the parse the second, and the hold runs from the second read to the fourth. The last three reads happen under the lock and lengthen the hold slightly; that is the enabled cost. Every subtraction and counter update waits until the guard has dropped.

Every event-loop-thread lock of a pane's parser goes through one lock_parser helper. Inside a dispatch of an App whose counters are on, it reads the clock before and after lock(), then, with the guard held, adds one bucket increment and one sum to thread-local counters that the App takes at the end of the dispatch. Otherwise it is exactly lock().

flowchart TD
    batch["VT worker finishes an output batch"] --> target{"pane has a redraw target?"}
    target -- no --> untargeted["count flushes_untargeted, store no time"]
    target -- yes --> pending{"a flush still pending?"}
    pending -- yes --> coalesced["keep the older time, count flushes_coalesced"]
    pending -- no --> store["store the flush time in the pane's slot"]
    coalesced --> watch["an armed echo watch records the publication"]
    store --> watch
    watch --> token{"an output event outstanding?"}
    token -- yes --> suppressed["send nothing, count flushes_suppressed"]
    token -- no --> send["send PaneOutput"]
    send -- refused --> refused["clear the token"]
    suppressed --> decision["an armed echo watch records the decision"]
    send -- sent --> decision
    refused --> decision
    send -- sent --> redraw["first RedrawRequested of a window that shows the pane"]
    redraw --> take["take the time and record its age in flush_to_redraw"]
Loading

The worker stores the flush time before it sends the output event, so the event loop never wakes before the time is published. A flush published while a redraw is taking the slot is taken by that redraw or left for the next one, never lost or counted twice.

Readings are observational. Fields are read one after another, not as one atomic snapshot, and a count belongs to the line or snapshot that read it after it was published; several batches may publish between two field reads.

The S2 echo watch

Each pane also has an echo watch while the gate is on. The perf harness arms it for S2/default and S2/flood samples, and only when it is built with perf-echo-trace. Arming issues a nonzero token that an App never reuses; the harness takes the record once, when a frame is credited, and every other close of the sample takes and discards it. A writer changes the record only for the armed token, while it is not yet taken, and only with an instant no earlier than the arm, so a writer paused across a take or a re-arm changes nothing. The lock order is parser, then watch slot; arming and taking take only the slot.

Under the parser guard, the worker reads the target cell before parsing (after the locked_at read) and after it (after the parsed_at read, before the keyboard snapshot is stored). The section in which the cell goes from absent to present, with the row identity (scrollback_evicted, screen_epoch, size_generation) unchanged at both reads, is the appearance: it records the generation its batch publishes, the parse instant and whether a synchronized update is set. Any identity change, an echo already present before an appearance, and an echo absent again after it are recorded as sticky facts.

The split's boundary is the publication: after publish_flush returns and before the token decision, an armed watch with an appearance and no publication reads the clock under its slot lock. The decision that follows, sent, suppressed or refused, is recorded after it. The watch never changes the decision or the send.

Part From To
input_to_parse_ms injection the appearance section's parse instant
parse_to_publication_ms that parse instant the publication
publication_to_present_ms the publication the end of the credited dispatch
delivery_lag_us the publication the worker recording its decision, including the send call for sent

The three parts add up exactly to the sample's latency. delivery_lag_us is not event-loop delivery latency. A suppressed publication is split: its publication_to_present_ms includes the remaining queue wait of the earlier event, which is real latency. A refused one reads send-refused.

Work Lock wait parse_us Hold
The unarmed Acquire load and the pre-read no yes yes
The post-read and any recorded-fact slot lock no no yes

Unarmed, a section and a flush each cost one load. Armed, the slot is locked once to fetch the target, at most once per new fact, once per eligible publication and once for its decision. The flush path reads the clock 0 times unarmed or with no eligible publication, once for an eligible untargeted flush and twice for an eligible targeted one.

Synchronized output: a worker holds its update and publishes at the reset, or at the published deadline, so its hold is included in parse_to_publication_ms and excluded from flush_to_redraw, which starts at the publication. A window cannot hold past that deadline for the same epoch, so publication_to_present_ms includes a hold only from another visible pane or a later epoch, each stretch capped at 150 ms. sync_open is the parser's set bit after the appearance section, not proof of a hold.

The S2 echo timeline

An armed echo watch can also own its window's echo timeline, a per-sample record of the event loop from the arm to the credited frame. The first pane armed in a window owns it while that owner lasts; another pane armed in the same window records no timeline, and re-arming the owning pane replaces it. The owner's successful take ends the owner, as do the pane leaving the window or being retired and the window closing.

The owner's record keeps at most 64 events, each stamped in nanoseconds from the arm; the 65th sets overflow and is dropped. It records:

  • the latest PTY read stamp among the chunks of the section where the echo appeared;
  • each output check that evaluated the watched pane, except a repeat of the previous check; recording stops only once the appearance is known and a kept check has reached it;
  • each accepted display-link tick, by the window's accepted-tick count (never reset at the arm) and its link generation;
  • each redraw admission decision for the watched window, and each clearing of a held permit other than its own consumption;
  • the entry and return of each watched RedrawRequested dispatch, and the enter and exit of its renderer call.

A ring of 256 entries keeps every completed output service in the owner's window, the watched pane's included; the 257th evicts the oldest and records its loop_seq. The comparison's flood count leaves the watched pane out.

take_echo_timeline_v1 transfers the record exactly once. The owner's take of the echo watch freezes it with a copy of the window's ring, and the transfer moves it out of the App. The answers, in the order they are decided: the gate off GateOff; an unknown pane NoPane; another token Mismatch; a second transfer AlreadyTaken; an arming that owned no timeline NotRecorded, before any check of the take; a watch not yet taken NotTaken; and a take that did not freeze the record, because its timeline ended before it, NotRecorded. A pane created while the gate is on has its PTY reader stamp each chunk with the instant its native read returned; that stamp is the record's read.

The harness binds each credited sample to the one watched dispatch whose entry and return lie inside the credited forward, validates the raw record, and writes the sample's echo_timeline and the run's echo_timeline_coverage to result.json; nothing is logged (Comparing performance describes the fields).

The S10 attribution watch

While the gate is on, the perf harness may arm one S10 attribution watch per pane with the run's sentinel line, the prompt and the phase's update count; arming returns a nonzero arming id and re-arming a pane replaces its watch. With the gate off, or for an unknown pane, an empty or over-long marker (more than 128 bytes) or a phase of no updates, nothing is armed.

Each main-window attempt that passes the guarded synchronized-output recheck copies, under the frame's parser guards, each watched pane's state: its resets, epoch and set, whether the sentinel and the prompt show from the cursor's row up to 8 rows above it, and why the frame was admitted. Only an attempt that presents writes them, one WARN line per watched pane on sonic::perf_present, which the default sonic=warn filter admits:

perf_present kind="present" arming=<id> seq=<n> window="main" pane=<id> resets=<n> epoch=<n> set=<bool> sentinel=<bool> prompt=<bool> admission="<admission>"

seq is the main renderer's presented count after that frame. admission is closed, forced-first, forced-visibility, forced-device, forced-resize, forced-surface, timeout or credit. A failed or retried attempt writes nothing. A watch writes at most 4 × updates + 64 present lines, then one kind="overflow" line with emitted=<n>, then nothing. Disarming drains nothing.

What the counters do not measure

The counters themselves do not split keystroke latency at the flush; the S2 echo watch above does, for S2/default and S2/flood. flush_to_redraw measures delivery and scheduling delay to the first redraw, and does not credit a presented frame. Maxima and p95s are bucket bounds, sums include the instrumentation's own cost, and shaping counts are requests, not HarfBuzz work.

GPU device error diagnostics

Each wgpu device keeps one error state, shared by every window built from the same GPU context. The sonic::gpu target writes one record per state change and one for the first isolated fault; the default filter's sonic=warn admits them. Repeated errors update counts without writing new records.

Message Level Written when
GPU device stopped accepting work error a Validation, OutOfMemory, or Internal error moves the device from Usable to Unusable
GPU device lost error the device-lost callback records Lost, including after an intentional destroy
contained isolated GPU error warn the first isolated fault from the test fault hook; later ones only count
Field Meaning
generation process-unique number of the device
state device state when the record was written
kind error class that caused the record: validation, out of memory, internal, or device loss
operation label of the renderer operation that raised the error, such as render.submit, try_resize, or glyph_upload.rebuild
description wgpu's error message
lost_reason wgpu's loss reason; empty except on loss records
destroy_requested whether SonicTerm destroyed the device on purpose; on state-change records
validation, out_of_memory, internal, isolated, lost coalesced counts per error kind

After an error record, every window stops drawing on that device. Shells, input, sessions, and window lifecycle keep working. A recorded loss starts shared-device recovery; an unusable device without loss remains stopped. Each affected renderer logs one warning, render error for main or child render error for another window, the first time it observes the stop. Architecture Internals has the containment rules.

Shared-device recovery records

The sonic::gpu::recovery target is admitted by the default sonic=warn filter. Warnings identify scheduling, request admission or refusal, negotiation and renderer preparation/commit failures, timeouts, busy-worker refusals, actual request completion (outcome and decision), and a successful shared GPU recovery committed. Error records identify a disconnected worker, an unusable device without a loss, and shared GPU recovery exhausted; terminal sessions remain running.

generation identifies the committed or newly committed device, ticket identifies an admitted request, attempt is its one-based budget position, delay_ms is the scheduled backoff in milliseconds, and rebound is the number of renderers committed together. Failure records include the native error where available. These records contain no terminal output or input.

A successful commit record proves replacement and gate acceptance, not native scanout or that a later frame presented. The stability timer starts only on an acknowledged Presented frame. A request timeout does not prove the native worker exited, and shutdown disposal is best-effort; compare request identities and later completion records rather than interpreting silence as cleanup. The retry policy and limits are on Rendering Modes.

Memory diagnostics

Aggregate snapshot at info

At level = "info", target="memory" writes one memory snapshot at most every 30 seconds. It combines OS process figures, all sampled pane seams, visible and warm renderers, and one shared-device allocator reading:

The following log examples are schematic, not captured measurements. Angle-bracket values stand for runtime fields; <metric> can be a byte count or unsupported, and <delta> can be a signed change or unavailable. Lines wrap for readability.

memory snapshot process_private_committed_bytes=<metric> process_resident_bytes=<metric>
                process_virtual_bytes=<metric> process_private_committed_delta=<delta>
                process_resident_delta=<delta> process_virtual_delta=<delta>
                session_total_bytes=<bytes> session_delta=<delta>
                grid_visible_bytes=<bytes> grid_history_bytes=<bytes> grid_alternate_bytes=<bytes>
                parser_bytes=<bytes> hyperlink_bytes=<bytes> inline_media_bytes=<bytes>
                pty_output_bytes=<bytes> pty_input_bytes=<bytes> panes_total=<count> panes_sampled=<count> panes_contended=<count>
                renderer_total_bytes=<bytes> renderer_total_items=<count>
                renderer_row_glyph_cache_bytes=<bytes> renderer_row_glyph_cache_items=<count>
                renderer_row_quad_cache_bytes=<bytes> renderer_row_quad_cache_items=<count> renderer_delta=<delta>
                live_renderers=<count> live_fg_probe_workers=<count> renderers="visible[<window-id>] glyph=<bytes>/<items> image=<bytes>/<items> row_glyph=<bytes>/<items> row_quad=<bytes>/<items> software=<bytes>/<items> vertex=<bytes>/<items> total=<bytes>/<items>; warm[<slot>] glyph=<bytes>/<items> image=<bytes>/<items> row_glyph=<bytes>/<items> row_quad=<bytes>/<items> software=<bytes>/<items> vertex=<bytes>/<items> total=<bytes>/<items>"
                allocator_state=measured allocator_source=main allocator_label=<window-id>
                allocator_allocated_bytes=<bytes> allocator_reserved_bytes=<bytes>
                allocator_allocations=<count> allocator_blocks=<count> allocator_largest_block_bytes=<bytes>
                [checkpoint_index=<index> checkpoint_label="<label>" checkpoint_attempt=<attempt> checkpoint_complete=<bool>
                 trimmed=<bool> trim_source="<source>" trim_seq=<count>]

Process figures come from the OS, so they include allocator fragmentation, retired pages, mapped files, GPU-driver mappings, and thread stacks that SonicTerm's own seams do not count.

Field Meaning
process_private_committed_bytes memory charged to this process alone; Windows reports PrivateUsage, macOS and Linux report unsupported
process_resident_bytes pages currently resident in physical memory; macOS reports resident size, Windows reports WorkingSetSize, and Linux reports unsupported
process_virtual_bytes reserved address space; macOS and Windows measure it, Linux reports unsupported; measured values can be hundreds of gigabytes without representing consumed memory
*_delta change since the preceding snapshot; +0 is measured, while unavailable means no comparable sample
panes_total all panes visited
panes_sampled panes included in session_total_bytes
panes_contended panes skipped because a parser or inline-image lock was busy; non-zero makes the session total partial
renderer_total_bytes / renderer_total_items CPU-side storage across visible and warm renderers
renderer_row_glyph_cache_bytes / renderer_row_glyph_cache_items row glyph cache payload plus tracking storage and cached row count, summed across renderers (each bounded to 512 MiB)
renderer_row_quad_cache_bytes / renderer_row_quad_cache_items per-row background/decoration quad cache storage and cached row count across renderers
live_renderers process-wide renderer count; a count above the renderers entries can expose an unreachable live renderer
live_fg_probe_workers this App's foreground-probe worker threads: 0 before the first demand or after the worker stops, otherwise 1
renderers per-renderer role and glyph/image/row-cache/software storage breakdown; each entry ends with renderer_trimmed and renderer_gpu_released_requested_bytes, the GPU request sizes a covered-window trim gave back, outside total; the renderer-local names keep them apart from the line's own trimmed tag
allocator_state measured, unsupported for a backend without a report or a stopped GPU device, or none before a renderer exists
allocator_source / allocator_label renderer class and identifier used for the one shared-device reading
allocator_allocated_bytes bytes assigned to live wgpu allocations
allocator_reserved_bytes bytes reserved in wgpu allocator blocks
allocator_allocations live allocation count
allocator_blocks allocator block count
allocator_largest_block_bytes largest allocator block in bytes
checkpoint_index / checkpoint_label / checkpoint_attempt only on a sample taken for a perf checkpoint: which checkpoint, and which attempt at it, from 1
checkpoint_complete only on a checkpoint sample: true when no pane was contended and every pane was sampled
trimmed / trim_source / trim_seq only on a checkpoint sample from a build with the trim hook: whether a visible renderer was trimmed in its current covered stretch, what requested the latest trim (scheduler, hook, or none before any), and the trims since startup

The allocator is reported once per shared device/context, not once per renderer. Sampling shares the retention cadence. An idle session wakes for a due sample, but that wake suppresses redraw and draws no frame.

At debug, a covered-window trim writes one covered window trimmed line on memory with the window, presenter, released frame-texture size, present-buffer bytes before the trim, gpu_released_requested_bytes and reason="occlusion_trim"; a released image atlas writes its own release line with the same reason.

scripts/perf-compare.py reads this line from every scenario run (Comparing performance). Each scenario's final memory checkpoint comes at least 60 s after GO, when the harness releases the workloads (5 s with --short, as in the smoke). Most scenarios end with an idle phase that lasts at least until then; S4 and S5 end on their 60 s stream phase, with the date loop still running, and S12 ends on its 10 s uncovered hold. S11 and S12 also take intermediate checkpoints. A harness built with perf-hook-checkpoint-memory takes its own tagged sample at each checkpoint, and a checkpoint's figures come only from those samples; without the hook a checkpoint has no memory figures (n/a: unsupported). The process_* byte fields read a byte count or unsupported, while session_total_bytes and renderer_total_bytes are always integers. renderer_total_bytes counts renderers' CPU-side storage only, and the macOS process sample has no footprint figure, so in a managed run perf-compare.py answers each checkpoint request with a macOS footprint reading. It bounds footprint at 40 s and writes the checkpoint's .done file only after footprint has exited and been reaped, or when it never launched; otherwise .done is withheld, and the harness's own wait ends the run. If .done does not appear within 60 s, the harness ends the run at once as invalid (exit 3), with a reason that names the checkpoint, and the next phase never starts. That is not an occlusion: the smoke fails instead of retrying it, and a comparison retries the run.

Pane and session detail at debug

At level = "debug", the same cadence writes one pane retention line for each sampled pane, then one session retention line. A pane whose parser or inline-image lock is busy is skipped, not waited on.

pane retention pane="<window-id>/<pane-id>" total_bytes=<bytes>
               grid_visible_bytes=<bytes> grid_history_bytes=<bytes>
               grid_alternate_bytes=<bytes> parser_bytes=<bytes> hyperlink_bytes=<bytes>
               inline_media_bytes=<bytes> pty_output_bytes=<bytes> pty_input_bytes=<bytes>
               largest_seam="<seam-name>" largest_seam_bytes=<bytes>
session retention panes=<count> total_bytes=<bytes> grid_visible_bytes=<bytes>
                  grid_history_bytes=<bytes> grid_alternate_bytes=<bytes> parser_bytes=<bytes>
                  hyperlink_bytes=<bytes> inline_media_bytes=<bytes>
                  pty_output_bytes=<bytes> pty_input_bytes=<bytes>

The eight seam fields are disjoint and sum to total_bytes:

Field What it owns First response
grid_visible_bytes current screen rows, prompt storage, rare attributes, and grid-container overhead no action; this includes the screen
grid_history_bytes retained scrollback lower scrollback if needed
grid_alternate_bytes saved primary-screen rows and history while the alternate screen is active leave the full-screen program
parser_bytes in-flight escape or media-capture buffers recheck the next sample; normally transient
hyperlink_bytes interned OSC 8 URI and id strings no action; bounded and reclaimed when links leave retained history
inline_media_bytes decoded inline images retained by panes display fewer images or close image-heavy panes
pty_output_bytes local PTY output queued or in flight let output drain
pty_input_bytes input queued toward the shell, usually a large paste let the shell drain it

Read largest_seam first, then compare at least five consecutive samples. A large flat working set is different from a value that rises every sample. Pane labels contain window and pane identifiers; a pane keeps its pane id after a tab moves even though the window id changes.

Renderer detail at debug

Each visible or warm renderer also writes one renderer retention line:

renderer retention window="<window-id>" role="visible" total_bytes=<bytes>
                   glyph_atlas_bytes=<bytes> glyph_atlas_items=<count>
                   image_atlas_bytes=<bytes> image_atlas_items=<count>
                   row_glyph_cache_bytes=<bytes> row_glyph_cache_items=<count>
                   row_quad_cache_bytes=<bytes> row_quad_cache_items=<count> software_frame_bytes=<bytes>
                   vertex_scratch_bytes=<bytes> vertex_scratch_items=<count>
                   row_ink_bytes=<bytes> row_ink_items=<count>
                   frame_scratch_bytes=<bytes> frame_scratch_items=<count>
                   chrome_cache_bytes=<bytes> chrome_cache_items=<count>
renderer retention window="warm[<slot>]" role="warm" total_bytes=<bytes>
                   glyph_atlas_bytes=<bytes> glyph_atlas_items=<count>
                   image_atlas_bytes=<bytes> image_atlas_items=<count>
                   row_glyph_cache_bytes=<bytes> row_glyph_cache_items=<count>
                   row_quad_cache_bytes=<bytes> row_quad_cache_items=<count> software_frame_bytes=<bytes>
                   vertex_scratch_bytes=<bytes> vertex_scratch_items=<count>
                   row_ink_bytes=<bytes> row_ink_items=<count>
                   frame_scratch_bytes=<bytes> frame_scratch_items=<count>
                   chrome_cache_bytes=<bytes> chrome_cache_items=<count>
Field What it owns First response
glyph_atlas_bytes CPU glyph-atlas capacity for this renderer bounded; a warm entry is controlled by warm_window_pool
glyph_atlas_items glyph entries in that atlas use with bytes to distinguish occupancy from capacity
image_atlas_bytes CPU inline-image atlas pixel-buffer capacity, including the nonempty 1×1 placeholder allocation reduce image use or renderer count
image_atlas_items inline-image atlas entries use with bytes to identify image occupancy
row_glyph_cache_bytes cached glyph-record, underline, tofu and missing-character vector capacities plus tables, slot vectors and pin lists at most 512 MiB per renderer; a pane is released when an assembly pass's begin_frame omits that pane
row_glyph_cache_items cached glyph rows a falling count with flat bytes can mean reusable table capacity remains
row_quad_cache_bytes hash-table backing plus cached background/decoration quad vector capacities compare with cached rows and pane/window churn
row_quad_cache_items cached quad rows a falling count confirms row eviction even when table capacity is sticky
software_frame_bytes full-window Windows software-present buffer reduce window size; zero outside that path
vertex_scratch_bytes the UploadStaging part: the presentation pipeline's reused CPU vertex-assembly buffer plus each atlas upload's dirty and coalesced rect lists and staging buffer the vertex buffer follows the largest recent frame and shrinks to twice a frame's use once over four times that use and over 1 MiB; a sync releases the rect lists; a staging buffer holds at most one atlas during a sync and follows the vertex rule, keyed on that sync's largest write, after each wgpu sync; a covered-window trim releases all of it
vertex_scratch_items 1 while the vertex buffer holds an allocation, else 0 —
row_ink_bytes the RowInk part: the allocated buckets of the per-row ink table (where each presented row drew, per pane and slot) plus one frame's staging buffer bounded by the visible rows; a pane that shrinks or closes releases its records at the next presented frame
row_ink_items committed per-row ink records, one per visible row —
frame_scratch_bytes the FrameScratch part: the per-frame draw vectors (glyphs, quads, overlays, images, row spans, underlines, staging indices, tofu, pane rects, column edges and row keys) the renderer keeps between assembled frames every restoration clears each vector and holds it within its cap (16,384,000 bytes in total); only a completed assembly also shrinks a vector to twice its use once over four times that use and over 1 MiB and drops column-edge slots above its peak, so a failed or retried frame keeps its warm capacity; zero while a frame holds it
frame_scratch_items scratch vectors that hold an allocation —
chrome_cache_bytes the ChromeCache part: the 64-slot tab-title table and the 32-slot chrome-run table, each kept run's text and glyphs, and the kept UI palette's color strings bounded by the fixed tables and their admission limits (256 text bytes, 512 glyphs) and by a 4 KiB palette allowance (a larger palette is derived again on every request, never kept); a font change, scale change or clear_shape_cache empties both run tables
chrome_cache_items kept tab titles plus kept chrome runs —

role="warm" means the renderer belongs to the standby pool, not a visible window; closing a window does not release it. Renderer figures are host memory, not GPU video memory.

Reclamation warnings

The default warn filter admits warnings on memory::reclaimed because they explain visible loss:

grep 'memory::reclaimed' ~/.sonicterm/logs/sonicterm.log*
  • cancelled a media capture that stopped receiving means no bytes arrived for two 30-second samples. The incomplete image will not appear and staging was reclaimed.
  • discarded inline images from idle panes to stay within the process ceiling means older decoded images were removed; resend any still needed.

The default filter also admits inline image atlas full; skipped older images without evicting text glyphs on sonic::glyph_atlas; renderer atlas pressure prevented older images from being uploaded. inline media evicted to hold the process-wide ceiling is a warning on the memory target and therefore appears when level is info or debug; it means a pane removed older images while retaining at least its newest image.

Crash, hang, and exit evidence

The panic hook runs on every thread and writes a session-tagged crashes/crash-<timestamp>.log containing version, panic payload, source location, forced backtrace, and up to 50 admitted tracing events. Normal shutdown writes sonic_exit warning lines. On Unix, the first of SIGSEGV, SIGBUS, SIGILL, SIGABRT, and SIGFPE to arrive appends a fixed FATAL: SIG… line through an async-signal-safe path. The handler then calls the action installed before it, at most once per process, with the original signal information, so Rust's runtime can still name a thread that overflowed its stack. When that action returns, or there was none, the handler raises the signal again under its default action. That ends the process, so the operating system can still produce its diagnostics, which describe the raised signal rather than the original fault; an ignored fatal signal still ends the process. A stack overflow is therefore logged as FATAL: SIGSEGV (or SIGBUS), although the process then ends by the SIGABRT that follows Rust's report. Windows relies on WER or LocalDumps when the system is configured to create them.

Crash history uses the selected RUST_LOG/configured filter without widening it, plus a DEBUG ceiling and an explicit persistence predicate. TRACE is never retained there, even when an output sink opts in. Font shaping text and collections use sonicterm_font::payload, which is excluded from crash history at every level; normal warning/error targets retain safe stage/count diagnostics. Routine white-text and untinted-color-glyph emission is not a warning.

Each record owns at most 4 KiB of variable payload, including at most 256 bytes of target. The ring also enforces a 64 KiB aggregate variable-capacity bound and the 50-record limit, evicting oldest records. Formatting uses bounded storage and UTF-8-safe truncation; [truncated] fits inside each cap. Fixed metadata is bounded separately by record count. Panic payload text and the rendered summary each have an independent 4 KiB bound, read from borrowed panic data.

Backtrace capture/output and a chained panic hook are separate surfaces. These limits cover recorder-controlled formatting/retention, not allocations inside arbitrary producer Debug implementations or a guarantee that arbitrary logs contain no secrets. Payload TRACE remains opt-in normal-sink evidence; structured breadcrumbs keep their separate metadata-only contract.

A hang may produce no panic artifact. On macOS, sample before force-quitting:

sample <pid> 10 -file /tmp/sonicterm-hang.sample.txt
grep -nE 'dispatch_sync_f_slow|redraw_target|__psynch_cvwait' \
  /tmp/sonicterm-hang.sample.txt

SIGKILL, Force Quit, TerminateProcess, power loss, and a hard OOM run no cleanup code. SonicTerm cannot write a final line or post-failure dump in those cases. Instead it leaves two pre-failure records:

  1. sessions/session-<id>.marker records only session id, pid, version, platform, start time, and state. A stale marker means the session was not marked clean; native PTY teardown that did not settle also leaves it behind. It does not identify the cause. Live sibling processes are skipped, damaged markers still count as evidence, and each prior marker is reported once on the next launch.
  2. breadcrumbs/breadcrumbs-<id>.log is a bounded atomic snapshot with no terminal text, commands, environment values, tokens, or credentials. It pins the latest version, platform, renderer, counts, full process resource sample, retention, allocator state, and bounded lifecycle transitions. Its event=retention record includes renderer_bytes, row_glyph_cache_bytes / row_glyph_cache_items, and row_quad_cache_bytes / row_quad_cache_items, so the last complete pre-failure snapshot preserves both row-cache size and occupancy. A separate fixed-cost event=resource_history private_committed=... resident=... sample is taken immediately and every 5 seconds, retaining at most 48 samples. Virtual address space appears only in the full event=resource record.

A breadcrumb rewrite replaces the old file atomically. The surviving file after a hard kill is the latest complete pre-failure snapshot, not a dump and not proof of cause. The default file budget is 64 KiB; configured limits must be able to hold all mandatory records, lifecycle capacity, and one maximum-width history line. The absolute minimum is 4096 bytes.

On the next launch, SonicTerm also checks OS records by conservative filename convention:

Platform Locations checked
macOS ~/Library/Logs/DiagnosticReports, /Library/Logs/DiagnosticReports (.ips)
Windows %LOCALAPPDATA%\CrashDumps, %LOCALAPPDATA%\Microsoft\Windows\WER\ReportQueue, ...\ReportArchive

A match only “may relate to” SonicTerm. The Windows check does not read WER registry configuration, so no file means only that the standard locations held no match.

Bug-report bundle

Include:

  1. SonicTerm and OS versions.
  2. The last 200 lines of the newest sonicterm.log* file.
  3. The relevant panic artifact, OS record, or process sample, if present.
  4. Exact reproduction steps and a screenshot or short recording for visual, input, VT, font, or layout defects.
  5. Hardware/software adapter information for rendering problems.
  6. For memory growth, at least five consecutive memory snapshot records; with debug, also include the identified pane's pane retention lines and all session retention and relevant renderer retention lines over the same interval. State the pane's largest_seam and what the session was doing.

Do not post secrets, tokens, full environment dumps, terminal output, or sensitive command data.

Clone this wiki locally