Repository navigation
Memory usage with lots of session #196
Replies: 8 comments 1 reply
|
did some profiling here and your diagnosis holds up. Almost all of it is GPU (IOSurface + Metal render targets), the heap stays small, and it scales with the number of live surfaces, not the one on screen. Not a leak, agreed. it's also structural, not workload. I spun up an isolated instance with 32 empty sessions at the same window size and it sits at basically the same footprint (~3.3 GB) as my real 32-terminal window (~3.4 GB), even though the real one is running a pile of agents, splits and scrollback. So it's the surfaces themselves, nothing to do with what runs inside them. and it's a ghostty trait, not something specific to agterm. Ghostty.app with the same tab count holds the same ~4 IOSurfaces per surface. The reason kitty stays low is a different renderer design (one shared GPU context, no per-tab framebuffers), so the comparison that matters for us is vs Ghostty.app, and there we are in the same ballpark mechanism-wise. where agterm makes it worse: the detail pane is an eager deck. Every session's surface across every workspace is realized AND sized to the full window up front, even when hidden, so switching is instant with no flicker. The price is you pay full render-target cost for surfaces you are not even looking at. Ghostty only fully pays for tabs you have opened and lets background ones stay at their last size. on the fix, I think it is mostly upstream. What you prototyped (free a hidden surface's renderer while keeping its shell + scrollback) needs a libghostty API that does not exist yet, plus the ghostty#13021 teardown race sorted out. Since ghostty itself carries the exact same cost, the place that helps most is ghostty unrealizing occluded surfaces, which would benefit every embedder, not just us. We build libghostty from a pinned upstream source and avoid carrying a fork on purpose, so an agterm-only renderer patch is not a direction I want to take. there is a lighter agterm-side lever: stop eagerly full-sizing hidden surfaces. But that gives back the flicker-free switching the eager deck exists for, and it still would not free the base targets, so it is a partial mitigation at best. yes, pls share the patch, happy to read it as input for the upstream ask. And if you want to open or point at a ghostty realize/unrealize issue, link it here and we can track it. |
|
Thanks for profiling it and confirming. Your 32-empty-sessions number lines up with what I saw, and I agree this is a ghostty trait rather than something agterm is doing wrong. On a related realize/unrealize issue: I looked and could not find one that asks to expose it or to free a hidden surface's renderer. The closest is ghostty#13021, a use-after-free in the exact teardown path this would use (Metal/CoreAnimation can present a freed IOSurface after the swap chain is torn down). On macOS ghostty already stops drawing an occluded surface, but it never frees the targets, so that is the real gap. If we want an upstream issue, I think it is really two: expose the API, and fix that teardown race so it is safe to call often. I attached the prototype patch against the pinned SHA ( Numbers again for reference, isolated build, one 1600x1000 window: 41 sessions went from ~2.6 GB to ~320 MB with the patch on. |
|
read the patch, and went through the renderer source at our pin rather than just the diff. Not taking it into the build, and it is for the reason you already flagged rather than a new one. three things worth having on the record:
two smaller ones. The draw path is not gated on on the prototype working for you, I believe it did. But the failure mode is a race between the frame permit coming back (metal completion fired, present requested) and CA actually consuming the IOSurface on the main queue, and that does not fail deterministically. Driving it per-hide rather than on display changes is what makes it reachable, which is what your own commit message says. A clean run at 41 sessions and 1600x1000 does not clear it. one number to add to yours, from my daily instance rather than an isolated one: 63 realized surfaces across 4 windows, 5830 MB footprint, 4165 MB of it IOSurface, whole heap 231 MB. About 5 of those surfaces are on screen, so ~3.8 GB is targets for panes nobody is looking at. Worse than the 32-session window I measured before. where I think this goes is upstream, teardown lifetime first and the API second. The ask is not really "expose realize/unrealize", it is that ghostty should free an occluded surface's targets itself, which Ghostty.app wants as much as any embedder does. The prerequisite is a one thing on our side that is not about your patch. We call |
|
I hit the same thing and ended up writing essentially the same patch, so here is what I can add: the upstream issue @paul-nameless could not find, two renderer bugs that bite once you drive this per-hide, and a few days' worth of running it. The upstream issue exists — it is just invisible
It is ghostty#12032, "GPU resources allocated per-surface including hidden/inactive tabs". Same analysis as yours, down to the same line references. You could not find it because And to be straight about it: I cannot carry it upstream either. I am not in Independently the same patchMine exposes Two renderer bugs in the realize pathBoth only show up once you unrealize per-hide rather than on display changes, and neither is in the version of the patch posted here. These took a while to find, so they are the part worth reading. 1. I suspect this is at least part of the flicker you saw — a rebuilt swap chain painting a handful of dirty cells over an empty target is going to look like a flash. 2. Fresh frame states get undefined uniform buffers that never get written. Symptom: a session restored hidden, then resized while unrealized, comes back correctly painted but clipped to the old geometry, and stays that way until you type. Fix is to zero Running itPatch plus an agterm-side gate, unrealize debounced 10s so transient covers (overlay, quick terminal, glancing at another session) do not churn the swap chain. A surface born hidden also needs handling explicitly: the visibility observer only fires on a change, so every restored session past the selected one would otherwise hold its resources forever. Memory. Around 60 sessions across 11 windows. Before: ~9.5 GB physical footprint, 6.3 GB of it IOSurface plus IOAccelerator, 410 IOSurface regions. Now: ~1.0-1.2 GB, 89 IOSurface regions, peaking around 6.7 GB while sessions restore and settling as they park. A fair chunk of what is left is Claude processes rather than the terminal. Useful sanity check if anyone reproduces this: IOSurface regions divided by 3 gives you the live surface count, since Rendering dormant tabs after a resize. Correct now, after both fixes above. This was the last thing broken. Flicker. Next to nothing. BenQ PD3200U, 3840x2160 running at 1920x1080 UI scale, 60 Hz. I would not notice it if I were not looking for it. Window resizing. Not silky, particularly with a tab holding a lot of scrollback open, but it works and it is not laggy enough to bother me. Not fixed: ghostty#13021 is still open in the sense that matters. It is closed NOT_PLANNED — filed against a fork, on iOS — but @paul-nameless's warning that per-hide teardown makes it much easier to hit is the right call, and I was wrong to initially read the closure as meaning the race was absent. The 10s debounce cuts the number of teardowns but not the race. No crashes for me so far, which is weak evidence for a timing bug. Happy to share the patch — it is 105 lines across 6 files, purely additive, against |
|
Here is the patch, against Shape of it: To be explicit about scope, since you have already said you would rather not carry a ghostty patch: I am not proposing agterm ship this. It is input for the upstream ask, as you put it. The one shape that could work on the agterm side without you carrying anything is weak-linking 0001-surface-realize-api.patchdiff --git a/include/ghostty.h b/include/ghostty.h
index b099741..07c8233 100644
--- a/include/ghostty.h
+++ b/include/ghostty.h
@@ -1112,6 +1112,7 @@ GHOSTTY_API void ghostty_surface_draw(ghostty_surface_t);
GHOSTTY_API void ghostty_surface_set_content_scale(ghostty_surface_t, double, double);
GHOSTTY_API void ghostty_surface_set_focus(ghostty_surface_t, bool);
GHOSTTY_API void ghostty_surface_set_occlusion(ghostty_surface_t, bool);
+GHOSTTY_API void ghostty_surface_set_realized(ghostty_surface_t, bool);
GHOSTTY_API void ghostty_surface_set_size(ghostty_surface_t, uint32_t, uint32_t);
GHOSTTY_API ghostty_surface_size_s ghostty_surface_size(ghostty_surface_t);
GHOSTTY_API uint64_t ghostty_surface_foreground_pid(ghostty_surface_t);
diff --git a/src/Surface.zig b/src/Surface.zig
index 5d16f33..a879fa2 100644
--- a/src/Surface.zig
+++ b/src/Surface.zig
@@ -3279,6 +3279,24 @@ pub fn occlusionCallback(self: *Surface, visible: bool) !void {
try self.queueRender();
}
+/// Callback for whether this surface should hold GPU resources at all.
+///
+/// Stronger than `occlusionCallback`: that one pauses drawing, this one
+/// releases the swap chain and shaders so a surface hidden indefinitely
+/// costs no GPU memory. The terminal state and the shell are untouched, so
+/// output keeps being processed while unrealized and the first frame after
+/// realizing shows it. See the `realized` message for the pairing rule.
+pub fn realizedCallback(self: *Surface, realized: bool) !void {
+ // Crash metadata in case we crash in here
+ crash.sentry.thread_state = self.crashThreadState();
+ defer crash.sentry.thread_state = null;
+
+ _ = self.renderer_thread.mailbox.push(.{
+ .realized = realized,
+ }, .{ .forever = {} });
+ try self.queueRender();
+}
+
pub fn focusCallback(self: *Surface, focused: bool) !void {
// Crash metadata in case we crash in here
crash.sentry.thread_state = self.crashThreadState();
diff --git a/src/apprt/embedded.zig b/src/apprt/embedded.zig
index 730913e..9a5b9d5 100644
--- a/src/apprt/embedded.zig
+++ b/src/apprt/embedded.zig
@@ -918,6 +918,13 @@ pub const Surface = struct {
};
}
+ pub fn realizedCallback(self: *Surface, realized: bool) void {
+ self.core_surface.realizedCallback(realized) catch |err| {
+ log.err("error in realized callback err={}", .{err});
+ return;
+ };
+ }
+
fn queueInspectorRender(self: *Surface) void {
_ = self.app.performAction(
.{ .surface = &self.core_surface },
@@ -1755,6 +1762,24 @@ pub const CAPI = struct {
surface.occlusionCallback(visible);
}
+ /// Set whether a surface holds GPU resources.
+ ///
+ /// Unrealizing frees the surface's swap chain, per-frame render targets
+ /// and font atlas textures; realizing rebuilds them. Intended for surfaces
+ /// an embedder keeps alive but hidden indefinitely (background tabs,
+ /// stacked panes), where the per-surface GPU cost dominates. Both calls
+ /// are asynchronous and land on the renderer thread.
+ ///
+ /// The shell and terminal state are unaffected: an unrealized surface
+ /// still reads its pty, and the first frame after realizing reflects
+ /// everything that arrived meanwhile.
+ ///
+ /// The embedder must alternate the two. A repeated call in either
+ /// direction is ignored rather than treated as an error.
+ export fn ghostty_surface_set_realized(surface: *Surface, realized: bool) void {
+ surface.realizedCallback(realized);
+ }
+
/// Filter the mods if necessary. This handles settings such as
/// `macos-option-as-alt`. The filtered mods should be used for
/// key translation but should NOT be sent back via the `_key`
diff --git a/src/renderer/Thread.zig b/src/renderer/Thread.zig
index 5087213..0c5f281 100644
--- a/src/renderer/Thread.zig
+++ b/src/renderer/Thread.zig
@@ -107,6 +107,12 @@ flags: packed struct {
/// This is true when the view is focused. This defaults to true
/// and it is up to the apprt to set the correct value.
focused: bool = true,
+
+ /// Whether the renderer currently holds its GPU resources. False only
+ /// between an unrealize and the matching realize. Drawing is skipped
+ /// while false: the swap chain is defunct and would reject the frame
+ /// anyway, this just avoids the error round trip on every draw request.
+ realized: bool = true,
} = .{},
pub const DerivedConfig = struct {
@@ -376,6 +382,35 @@ fn drainMailbox(self: *Thread) !void {
// check the visible state themselves to control their behavior.
},
+ .realized => |v| realized: {
+ // If our state didn't change we do nothing. This also makes
+ // an unpaired message harmless rather than tripping the
+ // defunct assert in displayRealized.
+ if (self.flags.realized == v) break :realized;
+
+ if (v) {
+ self.renderer.displayRealized() catch |err| {
+ // We keep the flag false so a later realize can retry.
+ // Leaving it true would let draws through to a swap
+ // chain that was never rebuilt.
+ log.warn("error realizing renderer err={}", .{err});
+ break :realized;
+ };
+ self.flags.realized = true;
+ // The rebuilt swap chain's targets hold nothing, so the
+ // next frame must repaint everything rather than the cells
+ // the terminal happens to consider dirty. A quiet surface
+ // dirties nothing at all, and one resized while unrealized
+ // dirties only part, which paints the new target at the old
+ // geometry and leaves the rest blank until a keystroke.
+ self.renderer.markDirty();
+ self.drawFrame(false);
+ } else {
+ self.flags.realized = false;
+ self.renderer.displayUnrealized();
+ }
+ },
+
.focus => |v| focus: {
// If our state didn't change we do nothing.
if (self.flags.focused == v) break :focus;
@@ -495,6 +530,9 @@ fn drawFrame(self: *Thread, now: bool) void {
// If we're invisible, we do not draw.
if (!self.flags.visible) return;
+ // If we hold no GPU resources there is nothing to draw into.
+ if (!self.flags.realized) return;
+
// If the renderer is managing a vsync on its own, we only draw
// when we're forced to via `now`.
if (!now and self.renderer.hasVsync()) return;
diff --git a/src/renderer/generic.zig b/src/renderer/generic.zig
index 0f4a294..5c627db 100644
--- a/src/renderer/generic.zig
+++ b/src/renderer/generic.zig
@@ -959,6 +959,16 @@ pub fn Renderer(comptime GraphicsAPI: type) type {
);
self.reinitialize_shaders = false;
self.target_config_modified = 1;
+
+ // Forget the screen size so the next frame reconciles it from
+ // scratch. Every frame state we just built has an UNINITIALIZED
+ // uniform buffer (see `FrameState.init`), and the only writer,
+ // `updateScreenSizeUniforms`, runs behind the draw path's
+ // `size_changed` check. A surface resized while unrealized already
+ // took `setScreenSize`, so that check would be false and the new
+ // buffers would never be written — the frame then draws at
+ // whatever projection the undefined memory happens to hold.
+ self.size.screen = .{ .width = 0, .height = 0 };
}
/// This is called by the GTK apprt when the surface is being destroyed.
diff --git a/src/renderer/message.zig b/src/renderer/message.zig
index a47b960..d7a7439 100644
--- a/src/renderer/message.zig
+++ b/src/renderer/message.zig
@@ -22,6 +22,19 @@ pub const Message = union(enum) {
/// and still have focus.
visible: bool,
+ /// Whether the surface should hold GPU resources at all.
+ ///
+ /// `visible` only pauses drawing; the swap chain, its per-frame render
+ /// targets and the font atlas textures stay allocated for the life of the
+ /// surface. Unrealizing releases them, so a surface that stays hidden for
+ /// a long time costs no GPU memory, at the price of rebuilding on the way
+ /// back. Reserve it for surfaces hidden indefinitely, not for every
+ /// occlusion change.
+ ///
+ /// The apprt must alternate these strictly: realizing an already-realized
+ /// surface, or unrealizing an unrealized one, is ignored.
+ realized: bool,
+
/// Reset the cursor blink by immediately showing the cursor then
/// restarting the timer.
reset_cursor_blink, |
|
Pushed it as a branch rather than a PR, since you have already said you would rather not carry a ghostty patch — this is for reading, not for merging: https://github.com/p4elkin/agterm-vim/tree/surface-unrealize Based on The two One thing that is not about this feature at all
Splitting it into three stages fixes it: the Incremental builds never hit this, because that file does not get re-type-checked — which is why it went unnoticed. A clean build does, and so would a new contributor. I only found it because the branch is based on your master rather than my fork, where the vim work happens to have changed that file. Verified: branch builds clean on |
|
PR that fixes memory issues |
|
Related to the agent-process side of this: I keep ~20 Claude Code sessions around, most of them long finished, because I want to come back to those conversations later — but I don't want to pay RAM for 20 live agents just to preserve a session id. The state is already on disk ( Would it make sense for |
|
@timurvafin there is no two things to know for the recipe:
|
Uh oh!
There was an error while loading. Please reload this page.
I keep a lot of sessions open, around 40-50 across a 3 windows. With that many, agterm sits at about 8 GB of RAM. I used to run kitty the same way and it stayed closer to 1-3 GB.
Probably for most of the people memory is not an issue but this is something that feels not right for terminal to take so much RAM.
I poked at where it goes and wanted to keep discussion open.
Where the memory goes
It is not a leak. Almost all of it is GPU memory (IOSurface and Metal render targets, about 6 GB). The normal heap is only ~250 MB.
Every session's surface stays fully live even when it is off screen, and each live one holds three full-window render targets. So memory grows with the total session count, not with the one you are actually looking at. Fine for a few tabs, less fine for 40.
What I measured
I ran an isolated build, spawned a bunch of workspaces and sessions, and read phys_footprint. One 1600x1000 window:
That is roughly 60 MB per hidden session at that size, and more at full-screen retina, which is where my ~8 GB comes from.
I also hacked together a fix that frees a hidden session's render targets and rebuilds them when you switch back.
The same 41 sessions dropped to ~320 MB, about 8x less.
The catch
Freeing those targets needs a small change to libghostty. Nothing in the C API lets you release a surface's renderer while keeping its shell alive: occlusion only pauses drawing, and freeing the whole surface kills the shell and scrollback. The prototype works, but it flickers on switch (the swap chain gets rebuilt) and it leans on ghostty's swap-chain teardown, which has an open use-after-free (ghostty#13021). So I would not run it daily yet.
Options
Just starting a conversation, maybe somebody will come up with a simpler fix to reduce memory usage. I would be okay to have option to have flicker but 8x less memory usage. If you are interested, I can share a libghostty patch.
All reactions