Excessive memory usage #2626
Comments
|
What does |
wez
added
the
waiting-on-op
|
I was curious to test this on my end so I wrote a quick hack of a shell one-liner to print out the RSS usage every 5 seconds for 5 minutes, across a few different scenarios. If I:
For me, it rises from about 100M to 200M usually, and the increase seems non-linear. It seems to allocate about 3x as more as it deallocated during the beginning, but around 200M is where it starts to increase much more slowly. Below is a quick hacky script to print out the timings of the RES/RSS memory every 5 seconds for 5 minutes, with the results I get under various workloads of both the default configuratopm. as well as my custom configuration: Default/empty config's memory increase rate (Idle workload, just this script running, and htop running as a base line in 1 of 2 vertical splits)mfiano@workstation ~$ nvidia-smi Fri Oct 14 22:15:13 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:09:00.0 On | N/A | | 0% 46C P0 29W / 150W | 948MiB / 6144MiB | 4% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ My full personal config's memory increase rate (Idle workload, just this script running, and htop running as a base line in 1 of 2 vertical splits)mfiano@workstation ~$ nvidia-smi Fri Oct 14 22:13:39 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:09:00.0 On | N/A | | 0% 44C P0 29W / 150W | 948MiB / 6144MiB | 4% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ Default/empty config's memory increase rate (High frequency output: this script running, and mpv with libcaca running in 1 of 2 vertical splits)mfiano@workstation ~$ nvidia-smi Fri Oct 14 23:01:38 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:09:00.0 On | N/A | | 0% 49C P0 31W / 150W | 1039MiB / 6144MiB | 13% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ My full personal config's memory increase rate (High frequency output: this script running, and mpv with libcaca running in 1 of 2 vertical splits)mfiano@workstation ~$ nvidia-smi Fri Oct 14 23:08:39 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:09:00.0 On | N/A | | 0% 50C P0 34W / 150W | 994MiB / 6144MiB | 14% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ Default/empty config's memory increase rate (Regular workflow with a few tabs and panels containing vim and some external processes I use while coding)mfiano@workstation ~$ nvidia-smi Fri Oct 14 23:38:08 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:09:00.0 On | N/A | | 0% 47C P0 26W / 150W | 913MiB / 6144MiB | 4% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ My full personal config's memory increase rate (Regular workflow with a few tabs and panels containing vim and some external processes I use while coding)mfiano@workstation ~$ nvidia-smi Fri Oct 14 23:18:19 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:09:00.0 On | N/A | | 0% 49C P0 30W / 150W | 1142MiB / 6144MiB | 4% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ Observations
Interesting 🤔 |
I'm currently working on a different PC, which has a Ryzen 5700U with integrated graphics, same issue. Currently one terminal open, started 5 hours ago, memory usage is at ~700MB. Is there a similar tool for Radeon graphics that I should run on that box?
No |
github-actions
bot
removed
the
waiting-on-op
|
Same here, Ubuntu 22.04 LTS Gnome+Wayland with integrated Intel GPU on a Thinkpad. Memory consumption was about the same as my two day old firefox with 40 tabs. I was running 6 tabs and 12 panes maximized, a single os//mux window. The terminal overall got sluggish, key presses lagging, repeating keys coming in bursts, search mode was particularly bad. In the end Gnome crashed so could've been a Gnome issue in part. I also get tons of wezterm errors from gsd-media-keys I'm on a recent nightly: don't do anything with images, just symbol/emoji font in prompt/nvim. Versions and logs if of any help.LogsI have thousands of these coming from gsd-edia-keys: over a few days: Past 14 hours: A couple days: Some system detailsUpdate: After upgrading wezterm to 20221017-204532-ec4d5eb1 RSS is staying below 600 MiB after two days but that's still quite a lot for a term. I'm still getting the same tons of errors in logs. |
|
Back on the nvidia box, I had 2 wezterm windows opened, which consume ~900MB of memory, and the whole system was lagging a lot. Xorg CPU usage was upwards of 60% whenever I tried to use firefox (which felt like a slideshow at best). nvidia-smi output: |
I spent a few hours in heap profilers. What I found was:
* Inefficient use of heap when building up runs of
`Action::Print(char)`.
-> Solve by adding `Action::PrintString(String)`
and accumulating utf8 bytes rather than u32 codepoints.
* Inefficient use of heap when building Quad buffers: the default
exponential growth of `Vec` tended to waste 40%-75% of the allocated
capacity, and since we could keep ~1024 of these in cache, there's
a lot of potential for waste.
-> Solve by bounding the growth to 64 at a time. This has similar
characteristics to exponential growth at the default 80x24 terminal
size. May need to add a config option for this step size for users
with very large terminals.
* Lazy eviction from the LFU caches. The underlying cache advisor is
somewhat probabilistic and has a minimum cache size of 256, making
it difficult to maintain low heap utilization.
-> Solve by replacing it with a very simple LFU algorithm. It doesn't
seem to hurt much at the default terminal size with the default
cache sizes. If we make the cache sizes smaller, its overhead is
reduced.
Some further experimentation is needed to adjust defaults, but this
should help reduce heap usage.
refs: #2626
|
I've made some adjustments to help keep the memory usage bounded better. If you find that the usage is still on the high side, then I would suggest that you experiment with adjusting some cache sizes in your config; the items below are shown with their default values: -- This should be set to at least the sum of the number of lines in the panes in a tab.
-- eg: if you have an 80x24 terminal split left/right then you should set this to at least 2x24 = 48
-- Setting it smaller than that will harm performance
line_quad_cache_size = 1024,
-- Should also be set >= number of lines as above.
-- Values are relatively small, may not need adjustment.
line_state_cache_size = 1024,
-- Should also be set >= number of lines as above.
-- Values are relatively small, may not need adjustment.
line_to_ele_shape_cache_size = 1024,
-- should be >= the number of different attributed runs on the screen.
-- hard to suggest a min size: try reducing until you notice performance getting bad.
shape_cache_size = 1024, |
I can't think of a good reason for this being here. I think I may have been lazy about resolving the lifetime annotations and just stuck in an extra buffer while building the original version of this logic, and then forgot about it. This commit resolves the lifetime annotations and directly references the passed in buffer. refs: #2626
|
Memory usage seems to be growing slower, but it's still growing. Currently at 700MB with a single window opened. And things got really sluggish. Cursor motion in vim when I hold down With There's nothing but With a fresh wezterm instance, the number of such calls per keypress is closer to 700. |
|
If tried to lower the settings you mentioned, and maybe that made a bit of a difference in terms of memory usage, but it also made the whole system get usably slow even quicker than before. If you want me to try something specific, I'll be happy to help, but for daily work, I'll have to switch back to alacritty for now. |
|
I'm still seeing high memory usage (near 600 MiB) and stuttering in shell and especially nvim with the new settings applied. I tested nvim in Kitty and it's as fast as expected, only Wezterm lags and stutters. Attaching a recording of editing a text file in vim: both scrolling a plain text file and editing lags. Key repeats come in chunks and sometimes with 100ms+ delay. As said, Kitty doesn't exhibit any of these issues so it's not nvim lagging. Same file as in recording is blazingly fast to scroll and edit without any stuttering. If I open a new window it's slightly better until it deteriorates again. The lag is so bad I had to switch to Kitty despite preferring Wezterm. Hope this can be fixed soon. Also wonder why it's consuming 3.8% CPU, same as the top Firefox instance. Kitty running idle at the same time ranges 0.0%-0.6% and consumes 67 MiB. |
This reduces peak heap usage by several MB by making better use of allocated memory--no wasted overhead in the Vec capacity. refs: #2626
|
I've made a couple of improvements to cache efficiency that may help here; I'd appreciate it if you could try a852e47 which should show up as a nightly build soonish. I want to note that I cannot reproduce the growing/leaking characteristics that you have described; I wonder if there is something missing from your reproduction scenario? I also wonder if there might be some graphics driver specific behavior that I just don't see: I'm using a radeon card. |
wez
added
the
waiting-on-op
I'll give it a spin, thanks!
The only thing I could think of would be something in my vim config. If you're interested, you can find it at https://github.com/dotdash/dotvim
I've seen the issue on both an nVidia 1080 TI as well as integrated graphics on a Ryzen 5600U. |
github-actions
bot
removed
the
waiting-on-op
wez
added
the
waiting-on-op
Since we're no longer tied to contiguous Vertex slices, we can record the Quad data used for HeapQuadAllocator in a smaller structure, so that's what this does. It reduces the per-Quad storage from 272 bytes down to 84 bytes, which helps with overall memory usage: the default cache size is 1024 lines worth of quads, so this can add up to multiple MB of RAM. refs: #2626
|
FWIW I use the Nix package manager and the issue is present (on both Mac and Linux) in the newest wezterm version packaged here: https://github.com/NixOS/nixpkgs/blob/497db553014c63d95f61a902bcde07761732b9ff/pkgs/applications/terminal-emulators/wezterm/default.nix#L33. This version is a little outdated (from September) but it's another data point that the excessive memory usage has been observed from other users as well. For me it's not a big issue because I use tmux to manage session state, so I just restart wezterm once a day or two (or three) whenever the memory creeps up past the 1 GB mark (and tmux has everything for me where I left off). Interestingly on Linux the wezterm instance appears to make X server reach 100% CPU for ~5-10 seconds when (1) I leave wezterm open for a while and it has > 500MB memory usage and (2) I close wezterm or hover my mouse over it to gain focus, but I think that's a separate problem related to my X configuration with Xmonad. |
Since we're no longer tied to contiguous Vertex slices, we can record the Quad data used for HeapQuadAllocator in a smaller structure, so that's what this does. It reduces the per-Quad storage from 272 bytes down to 84 bytes, which helps with overall memory usage: the default cache size is 1024 lines worth of quads, so this can add up to multiple MB of RAM. refs: wez#2626
I've seen constant CPU usage over 60% with 1-2 second 100% spikes when wezterm has been running for a while, so probably not specific to your setup. |
github-actions
bot
removed
the
waiting-on-op
|
With wezterm started about 3 hours ago and currently 6 windows opened (one at 426x111, the others at 212x111), I see 1.4G RSS, while it was at 1.6GB a few minutes ago. Xorg CPU usages hovers around 20%. wezterm 20221114-144425-a852e47d |
|
Some more data points: The same wezterm instance from before is still running, the memory usage stayed around 1.4-1.5G, but for a few minutes now, whenever I move my mouse, Xorg CPU usage skyrockets, and moving the mouse quickly makes the whole screen freeze. I've closed all gui apps except wezterm and the freezing effect persisted. Stopping wezterm immediately made the problem go away. nvidia-smi output: |
|
For me the symptom of high CPU usage/lagginess when moving the mouse in/out of a long-running wezterm instance is only present on Linux with X server (where I use an NVIDIA card as well). This lagginess is not present on my M1 Mac (but the high memory usage is still there). |
|
There is an open issue for high CPU usage at #2665 which appears to describe very similar symptoms. |
To summarize, the solution for that seems to be:
src: #2665 (comment) ie. don't do this |
Indeed this is the first thing I checked but AFAICS the Nix build already does the recommended setting: https://github.com/NixOS/nixpkgs/blob/497db553014c63d95f61a902bcde07761732b9ff/pkgs/applications/terminal-emulators/wezterm/default.nix#L80 |
|
I'm using wez's Debian11 Nightly package, which I assume is built with the recommended settings. |
|
This might not apply here in the case of wezterm(aside from the fact that they're both written in Rust and maybe using Vec`s), but here's what I'm doing to alacritty to avoid memory usage doubling due to |
|
The new release has landed in Nixpkgs here and I've been using it for over 48 hours. Normally memory would exceed 1GB by this point but it is 259MB. This is at least a 75% reduction in memory usage. In addition, all of the CPU spike behavior and lagginess has disappeared! Thank you @wez for the new release! I assume this matter can be closed if @dotdash can also confirm the new improvements on their end. |
I have spoken too soon (by a few hours...?!). The lag has returned and I had to restart WezTerm to make it go away. For now I'll just lurk at #2665. |
|
Happens here as well (even when building from source), noticed it when I was checking btop and kept seeing WezTerm climbing up non-stop. 1 - WezTerm 2023-03-02.00-10-52.mp4Edit: Config: local wezterm = require 'wezterm'
return {
colors = {
tab_bar = {
inactive_tab_edge = "none",
active_tab = {
bg_color = '#1a1a1a',
fg_color = 'silver'
},
inactive_tab = {
bg_color = '#0a0a0a',
fg_color = '#808080'
},
inactive_tab_hover = {
bg_color = '#202020',
fg_color = '#909090'
},
new_tab = {
bg_color = '#0a0a0a',
fg_color = '#808080'
},
new_tab_hover = {
bg_color = '#202020',
fg_color = '#909090'
},
},
background = 'transparent',
cursor_bg = 'silver',
cursor_border = 'white',
selection_bg = 'silver',
},
window_frame = {
active_titlebar_bg = "#0a0a0a",
inactive_titlebar_bg = "#0a0a0a",
font_size = 10.0
},
font = wezterm.font 'JetBrains Mono',
font_size = 10.0
} |
|
I am having the same issue as @vars1ty from the Nixpkgs. Memory usage keeps climbing over time (even with only one window open). I am using wezterm with the default config on Hyprland (and thus wayland). I am on NixOS and all packages are on the unstable branch of nixpkg. The following are my observation using btop: There seems to be a 10MB increase in memory usage by wezterm-gui when the second window (this only happens to the second window, all successive windows only seem to add 1MB at most) and it gives a short burst to the above mentioned rate (pushes it to about 1.5MB/s for 2 seconds), this seems to only happen for the first 4-7 windows. No memory is freed after windows are closed (with at least 1 window still open). Over the period of 2 hours the memory usage climbed to 330MB (with only a single open window). There seems to be an occasional burst of 1MB/s over 5-8 seconds but I can't figure out what causes it. Over the period of 30 hours (window was left open over night) the memory usage was 1.1GB. After closing said window and relaunching the process the above pattern repeats. Unlike @listx, I have not experienced any lag or cpu spikes during the above testing or normal usage - so that seems to be an x11 specific bug. EDIT: It should be noted the system was running for the entire duration of the 30 hour test - I don't have any idle or hibernation setup on the system. The display was turned off (as in the manual switch on the monitor) for about 7 hours if that makes any difference. |
|
After more testing I realized that running btop in wezterm might contaminate the results. I ran btop in foot and left wezterm to idle. This did not cause an increasing in memory (thus stayed static at 144MB). Running a simple command like ls or cd did not cause an increase in memory usage. Commands that produce lots of visual output (such as tree) cause memory usage to jump about 5MB and simply running journalctl and scrolling down caused an increase of 40MB (it should be noted rerunning the commands, and thus getting the same output, does not increase memory usage). TUI progams that remain mostly static (such as neovim when idle or calcurse) caused an increase of about 2MB. Running something like Donut.c increases the memory usage by about 10MB before it plateaus, whereas something like this falling matric script causes a constant increase in memory between 0.5MB/s-1.5MB/s depending on the amount of visual clutter. The memory usage eventually plateaus - I assume it is because all permutations of the falling text has been seen. Running btop in wezterm (like @vars1ty) causes a constant increase in memory usage that never stops (see my previous comment). That being said the rate of increase is directly related to update time (by default 2000ms) of btop. The above information leads me to believe (please note I have not read wezterm's source code so this is an educated guess) that wezterm either:
Either way the issue seems to be state/cache data which is kept in memory. Whether this is intentional (cache) or a bug the memory should be freed eventually. DISCLAIMER: My machine does not have a swap partition/file, but does has zswap enabled (I believe is allotted 16GB) so I have no idea if said memory increases would have been written to swap if available - which does alleviate the problem somewhat, but is still undesirable. |
|
See also: #3815 |
|
Thanks for the response, while I agree this isn't a priority issue for most workflows (and wezterm otherwise has great performance) this does make wezterm unsuitable on very low memory machines (as in 4GB or less). I understand the feeling of burnout and would like to thank you for your work thus far. It might be worth marking this issue as a known-issue or closing this one and opening an new issue with an "enhancement" tag (something like "Improve Memory Allocation") just to centralize the discussion and increase visibility - maybe someone else has the time to delve into the problem. Again, thank you. I have only been using Wezterm for a week and I am very happy with the terminal's performance on my desktop thus far. I guess I will just have to stick with Foot on my old laptop (4GB ram) for the time being :) |
I don't know how much this actually contributes to wezterm's memory usage specifically, but from browsing the code, I noticed that the For example, here is Lines 18 to 21 in cce0706 Here, (One way to improve this could be to move the representation of the "has (non-)default foreground/background color" situation into two new bits in the rxvt-unicode is very careful about the per-cell overhead, and in fact has avoided adding more attributes because it "ran out of bits" in its cell attribute bitfield. The author is considering moving to extents to represent attributes (since they're often the same for contiguous cells), which would allow significantly lower memory use, but would also significantly complicate all involved algorithms. |
|
This really should be fixed, one of the things holding me back from using wezterm, I run awesomewm and with wezterm it takes up almost the same amount of ram usage as running kde/gnome simply because of this issue. To compare, stterm takes about 16mb RAM idle and when stuff is open it's about 20-30MB. I can see it being around the 50-100mb range because of all the features but still about 400mb of ram for a terminal emulator is way too much |
What Operating System(s) are you seeing this problem on?
Linux X11
Which Wayland compositor or X11 Window manager(s) are you using?
awesome wm
WezTerm version
wezterm 20221012-092613-6415ba0e
Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?
Yes, and I updated the version box above to show the version of the nightly that I tried
Describe the bug
After a few hours, the memory usage of wezterm grows to excessive amounts and rarely seems to go down. My current wezterm instance is running for ~ 18hours and has grown its RSS to 1.4G with just a single open window (there have been up to ~10 at some point in time). Max terminal sizes for the windows were 426x111.
To Reproduce
Work with wezterm for some hours, mostly using vim.
Configuration
Full config is as follows. I don't think anything is really relevant, but unless required, I'd rather not have to work with the default settings for an extended amount of time to reproduce this problem 😅
Expected Behavior
Memory usage should not grow as much, or at least drop down again when windows get closed.
Logs
Debug Overlay
wezterm version: 20221012-092613-6415ba0e
OpenGL version: NVIDIA GeForce GTX 1080 Ti/PCIe/SSE2 4.6.0 NVIDIA 510.85.02
Enter lua statements or expressions and hit Enter.
Press ESC or CTRL-D to exit
23:20:37.273 WARN mux::localpane > unknown DeviceControlMode::Enter EnterDeviceControlMode(params: [], intermediates: [], byte: 'z' 0x7a, ignored_extra_intermediates=false)
23:20:37.273 WARN mux::localpane > unhandled DeviceControlMode::Data 7a z
23:20:37.273 WARN wezterm_term::terminalstate::performer > unknown unspecified CSI: "0%m"
23:45:21.544 WARN mux::localpane > unknown DeviceControlMode::Enter EnterDeviceControlMode(params: [], intermediates: [], byte: 'z' 0x7a, ignored_extra_intermediates=false)
23:45:21.544 WARN mux::localpane > unhandled DeviceControlMode::Data 7a z
23:45:21.544 WARN wezterm_term::terminalstate::performer > unknown unspecified CSI: "0%m"
23:58:06.376 WARN mux::localpane > unknown DeviceControlMode::Enter EnterDeviceControlMode(params: [], intermediates: [], byte: 'z' 0x7a, ignored_extra_intermediates=false)
23:58:06.377 WARN mux::localpane > unhandled DeviceControlMode::Data 7a z
23:58:06.377 WARN wezterm_term::terminalstate::performer > unknown unspecified CSI: "0%m"
10:35:41.226 INFO wezterm_gui::termwindow::render > breaking on overflow 3492 > 1908
10:35:41.226 INFO wezterm_gui::termwindow::render > breaking on overflow 3501 > 1908
10:35:41.226 INFO wezterm_gui::termwindow::render > breaking on overflow 3501 > 1908
10:35:41.226 INFO wezterm_gui::termwindow::render > breaking on overflow 3510 > 1908
10:35:41.249 INFO wezterm_gui::termwindow::render > breaking on overflow 3483 > 1908
10:35:41.249 INFO wezterm_gui::termwindow::render > breaking on overflow 3492 > 1908
10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1755 > 1737
10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1764 > 1737
10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1773 > 1737
10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1782 > 1737
10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1791 > 1737
10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1791 > 1737
Anything else?
No response
The text was updated successfully, but these errors were encountered: