Memory leak when monitor sleeps? #4821
Labels
Comments
|
Re-reading the discussion, I realize I did not try to isolate the issue by disabling widgets. Let me do that and report back. |
|
I disabled all widgets and the issue persists. I actually don't have to force off my monitors: I can simply deactivate one of them in arandr, and every 10 seconds the memory taken by Qtile increases by the initial amount:
Is it because of screens then 🤔? I have two screens, one deactivated 🤔 The config file is only executed once on startup, right? |
|
OK I commented out almost everything in my config file. Now it's just two keys (to restart Qtile), one screen with an empty top bar, and the BSP layout. The memory starts leaking as soon as one monitor is deactivated (through arandr), and stops leaking as soon as the monitor is reactivated (through arandr again). So it looks like the issue lies somewhere low in the stack 🤔 Happy to try and provide logs if you can guide me 😄 |
|
A bit more weirdness 😂 And it's kinda getting us closer to my initial intuition on the GPU change: the memory only starts leaking if my right monitor is deactivated (lets hope its position on my desk is irrelevant though 🤣). Both are the same device (iiyama something), however one is plugged to my graphic card with an HDMI cable/port, while the other one, which is causing the memory leak when turned off, is plugged with a display-port (GPU) to HDMI (monitor) cable (I bought these monitors many years ago, and they don't have display ports, "only" HDMI, DVI and VGA, while my GPU has only one HDMI port and 3 display ports). Sure enough, if I actually unplug the HDMI/display-port monitor, the memory stops leaking. |
|
Just to be specific: the leak happens when the monitor is "sleeping" rather than turned off. Turning it off by pressing the power button of the monitor does not cause a leak. |
|
Hmmm would you be able to test other X11 WMs? E.g. i3 |
|
Sure, that's a good idea :) At this point it indeed looks like we're going lower in the stack than Qtile itself ^^ |
|
So I tried with Openbox 🙂 There's no memory leaks, but I witnessed something that matches my previous observation that the memory leaks every 10 seconds: in Openbox, with the problematic monitor disabled through arandr, every 10 seconds Openbox would pop up a dialog with screen configuration (mirror, only left/right, extend), as if it just detected that the monitor was plugged in. So it looks like my monitor (or rather, the way it's plugged to the GPU) "announces" itself every 10 seconds when it is not already active. In other WM it causes a pop-up, and in Qtile, a memory leak. Hopefully that gives Qtile maintainers a helpful hint at where to look in the code to avoid allocating more pixmaps when a monitor merely announces itself up? Not sure if that makes sense. In the coming days I'll try to buy another cable, or a display-port-to-HDMI adapter, and see if the issue persists. I really do not want to buy a new monitor to fix that... By the way, I have also tried to entirely disable sleeping and monitors going to sleep on my ArchLinux system, but failed to do so. I'll explore again what the internet says. |
|
Interesting. If there's some sort of notification then it's possible we're trying to reconfigure screens. Can you add a hooked function in your config to see if we're getting screen change events? e.g. from libqtile import hook
from libqtile.log_utils import logger
@hook.subscribe.screen_change
def screen_change(event):
logger.warning("Screen change event received.")Let's see if this is fired repeatedly. |
|
I wonder, are we leaking widgets here: Lines 451 to 454 in d9f4c08 Maybe the copy means that .finalize() is not explicitly called? |
|
If that's the case, wouldn't we see memory leaks every time we reloaded configs? |
|
@tych0 note that I witnessed the leaks with zero widgets in my single top bar. @elParaguayo here are the logs: First line = Qtile restarted to activate the hook you suggested. I waited a bit (one minute and a half), |
|
could you also start qtile in debug mode and try to find the relevant logs when this happens? to do that pass |
huh; looking at libqtile.config.Screen, the only other thing I see that might do an allocation is wallpaper painting. Did you have that enabled by chance? |
|
@jwijenbergh sure, let me do that. @tych0 I do set a wallpaper, though through pywal, in |
Let me know if this is enough, logs are still being written in DEBUG, I can paste more. |
|
Yeah, it is a thing via Screen(wallpaper="/path/to/image"), but if you're not using it, it must be something else. I'm just reading code in Screen() and trying to see any unfreed allocations it might perform... |
|
i think it is definitely true that at least the x11 wallpaper painting code leaks resources when screens are reconfigured, though. |
|
Any chance #4825 helps? |
|
Nope :/ EDIT: to explain, I applied the changes directly in /usr/lib/python3.12/site-packages/libqtile, and both logged out / logged back in, and rebooted, and I still see the leaks :/ |
|
Just randomly throwing out things to look for:
|
|
I added this to the screen change hook: @hook.subscribe.screen_change
def screen_change(event):
logger.warning("Screen change event received.")
logger.warning(str(gc.get_stats()))
logger.warning(str(gc.get_count()))
logger.warning(str(gc.garbage))Click to showIt's not very conclusive. |
|
I suspect that python's gc won't show much interesting, because the python objects are probably getting collected correctly, and we're just forgetting to do some x11 API call to free some x11 resource. Can you try without a bar and see if it still happens? I have read all this code several times and it looks okay to me, but the bar allocations are somewhat complicated, so it wouldn't surprise me if there's something lurking there. |
|
Oh, maybe this? tych0@96d4b60 |
|
Still no luck :( Will now try without a bar :) |
|
🎉 🎉 🎉 🎉 🎉 🎉 🎉 No bar, no leak! The leak seems proportional to the the number of bars and widgets within them. With a single empty bar, ~300K leak with each screen reconfiguration. With my complete bars, it's more like 8M. |
|
That one has a bug, so if it didn't crash, I think the patch wasn't applied correctly. I'm pretty hopeful this will fix the issue: 32ce136 Can you try that one instead? |
|
Unfortunately, no, it still leaks with a single empty bar 😕 |
|
Ok. Can you add some logging to confirm that it is applied correctly? I'm quite perplexed at this point. The above patch does fix a real leak... but maybe we have more? does the amount of memory leaked go down with that patch? |
|
Can you get an xtrace of the xrandr output? I'd like to see exactly the wire format if possible. |
|
Hmmm, as said previously, |
|
I guess you're on arch? they renamed it to x11trace: #4810 (comment) |
|
|
When monitor is disconnected: |
Merged
|
Thanks, can you try the two patches above as well? feels like we have to be getting close to fixing all of the leaks... right? right? :) |
|
ping @pawamoy, have you had a chance to test the above patches yet? are things still leaking? |
|
Ah sorry, I wasn't home for a few days and forgot about that, thanks for the reminder, let me check. |
|
@tych0 it would awesome if you could merge all the patches related to this issue in a single branch of yours, so that I can easily copy/paste files to my locally installed Qtile (or even install from your branch) instead of writing the changes by hand 🙂 Just the latest patch doesn't fix the leak, but maybe the combination of all of them will. |
|
All of them should be on master, but I think you are right that there is a bug in one of them. Can you send a patch to fix that? |
|
Do you mean the master branch of your fork, or the master branch here? |
|
The master branch here has everything that I'm aware of right now that would cause a leak. |
|
I have installed from master branch with Thank you so much for your help on this! |
|
Actually now that I'm on Qtile master, it leaked again while I was afk 😅 Damn. Maybe it's because I restarted Qtile before. Will see if it happens again without restarting Qtile. |
|
Nah, let's keep this open. I'll keep looking. Thank you for continuing to test! |
2 tasks
|
Do you know offhand if your config results in Mirror widgets being created? It looks like we have some leaks there as well, which I can send a patch for shortly. I am also looking for a way to see if we can automatically detect these leaks. I would love to get a big stack trace in CI when we do leak an object without finalizing, so we don't have to do this scavenger hunt in the future :) |
tych0
added a commit
to tych0/qtile
that referenced
this issue
Jun 15, 2024
We call _create_last_surface() on every draw() call. If last_surface previously existed, we will leak its backing resources since we do not finish() it. Let's finish() it. Found as part of qtile#4821. Signed-off-by: Tycho Andersen <tycho@tycho.pizza>
Looking at the code, it looks like only |
|
Not quite. If you use the same instance of a widget in multiple bars then a mirror will be created. WidgetBox just replicates this process as widgets aren't immediately added to the bar. |
tych0
added a commit
that referenced
this issue
Jun 15, 2024
We call _create_last_surface() on every draw() call. If last_surface previously existed, we will leak its backing resources since we do not finish() it. Let's finish() it. Found as part of #4821. Signed-off-by: Tycho Andersen <tycho@tycho.pizza>
|
Yeah, that's the one I was wondering about. I saw you have multiple bars, so it seems like it is potentially possible? Anyway, that one is in git master now, so maybe worth another go to see if that was it? |
|
So, this will not catch all of the flavors of bugs I have fixed, but it would have caught some, and is pretty simple: note that it's not guaranteed to work, since |
|
I think it would have caught stuff like a7f5d1c too? |
|
Nice. Are all our |
|
possibly, but things that aren't Configurable are Finalizable (e.g. drawer), so it's not a strict superset. it looks like we also do not finalize |
Thanks for the explanation. I vividly remember going out of my way to actually avoid reusing the same widget instances, because that caused issues (which ones exactly I don't remember though). Looking again at my config, I believe all widgets are unique in my bars. Even Click to see my bars confignum_monitors = get_num_monitors()
laptop_battery = []
if hostname == "laptop":
laptop_battery.append(widget.Battery(discharge_char="↓", charge_char="↑", format="{percent:2.0%}{char}"))
text_color = "#99c0de"
light_color = "#404552"
dark_color = "#2f343f"
separator_config = {"padding": 5, "linewidth": 0}
light_sep = lambda: widget.Sep(**separator_config, background=light_color)
dark_sep = lambda: widget.Sep(**separator_config, background=dark_color)
triangle_config = {"padding": 0, "fontsize": 28}
light_triangle_left = lambda: widget.TextBox(**triangle_config, text="", foreground=light_color, background=dark_color)
light_triangle_right = lambda: widget.TextBox(
**triangle_config, text="", foreground=light_color, background=dark_color
)
dark_triangle_left = lambda: widget.TextBox(**triangle_config, text="", foreground=dark_color, background=light_color)
dark_triangle_right = lambda: widget.TextBox(**triangle_config, text="", foreground=dark_color, background=light_color)
screens = [
Screen(
top=bar.Bar(
[
dark_sep(),
widget.TextBox(
text="¤",
margin=3,
fontsize=28,
background=dark_color,
),
dark_sep(),
widget.GroupBox(
highlight_method="line",
this_screen_border="#5294e2",
this_current_screen_border="#5294e2",
active="#ffffff",
inactive="#848e96",
background=dark_color,
),
dark_triangle_right(),
widget.Prompt(),
widget.WindowName(foreground=text_color),
widget.Chord(
chords_colors={
"launch": ("#ff0000", "#ffffff"),
},
name_transform=lambda name: name.upper(),
),
*laptop_battery,
widget.Systray(padding=8, icon_size=16),
light_sep(),
dark_triangle_left(),
widget.CurrentLayoutIcon(
scale=0.6,
foreground=text_color,
background=dark_color,
),
widget.Clock(
format="%Y-%m-%d %a %H:%M",
foreground=text_color,
background=dark_color,
),
dark_sep(),
light_triangle_left(),
widget.TextBox(
text=" ",
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn(os.path.expanduser("~/.config/rofi/powermenu.sh"))
},
foreground="#e39378",
),
],
size=bar_size,
background=light_color,
),
bottom=bar.Bar(
[
widget.Spacer(),
widget.TextBox(text="devboard", mouse_callbacks={"Button1": lambda: qtile.cmd_spawn("terminator -x zsh -c devboard")}),
widget.TextBox(text=" ", mouse_callbacks={"Button1": lambda: qtile.cmd_spawn("pavucontrol")}),
widget.Volume(mouse_callbacks={"Button1": lambda: qtile.cmd_spawn("pavucontrol")}, foreground=text_color),
dark_triangle_left(),
widget.Net(
interface=net_interface,
update_interval=2,
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn("terminator -e 'sudo nethogs'"),
},
),
widget.NetGraph(
start_pos="bottom",
bandwidth_type="down",
**graph_config,
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn("terminator -e 'sudo nethogs'"),
},
),
widget.NetGraph(
start_pos="bottom",
bandwidth_type="up",
**graph_config,
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn("terminator -e 'sudo nethogs'"),
},
),
dark_sep(),
light_triangle_left(),
widget.GenPollUrl(
url="http://ip-api.com/json/",
parse=get_ip_location,
foreground=text_color,
update_interval=3600 * 4,
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn("bash -c 'notify-send ifconfig \"$(ifconfig enp3s0)\"'"),
},
),
dark_triangle_left(),
dark_sep(),
widget.CPU(
format="{load_percent}%",
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn("terminator -x bash -ilc 'htop --sort-key=PERCENT_CPU'"),
},
),
widget.ThermalSensor(
tag_sensor="Core 0",
show_tag=True,
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn("terminator -e 'htop --sort-key=PERCENT_CPU'"),
},
),
widget.CPUGraph(
**graph_config,
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn("terminator -e 'htop --sort-key=PERCENT_CPU'"),
},
),
widget.Memory(
format="{MemUsed:.0f}{mm}/{MemTotal:.0f}{mm} ({MemPercent:.0f}%)",
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.cmd_spawn("terminator -e 'htop --sort-key=PERCENT_MEM'"),
},
),
widget.MemoryGraph(
**graph_config,
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.spawn("terminator -e 'htop --sort-key=PERCENT_MEM'"),
},
),
widget.DF(
format="{uf}{m}/{s}{m} ({r:.0f}%)",
visible_on_warn=False,
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.spawn("terminator -e 'htop --sort-key=IO_RATE'"),
},
),
widget.HDDBusyGraph(
**graph_config,
foreground=text_color,
background=dark_color,
mouse_callbacks={
"Button1": lambda: qtile.spawn("terminator -e 'htop --sort-key=IO_RATE'"),
},
),
dark_sep(),
],
size=bar_size,
background=light_color,
),
)
]
if num_monitors > 1:
for m in range(num_monitors - 1):
screens.append(
Screen(
top=bar.Bar(
[
dark_sep(),
widget.TextBox(
text="¤",
margin=3,
fontsize=28,
background=dark_color,
),
dark_sep(),
widget.GroupBox(
highlight_method="line",
this_screen_border="#5294e2",
this_current_screen_border="#5294e2",
active="#ffffff",
inactive="#848e96",
background=dark_color,
),
dark_triangle_right(),
widget.WindowName(foreground=text_color),
],
size=bar_size,
background=light_color,
)
)
) |
fjpavon
pushed a commit
to fjpavon/qtile
that referenced
this issue
Jun 20, 2024
We call _create_last_surface() on every draw() call. If last_surface previously existed, we will leak its backing resources since we do not finish() it. Let's finish() it. Found as part of qtile#4821. Signed-off-by: Tycho Andersen <tycho@tycho.pizza>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue description
I'm experiencing memory leaks with Qtile when my monitors go to sleep. More specifically, Qtile seems to allocate a lot of things (pixmaps) within Xorg.
Please see #3866. Copy/pasting some "evidence" here:
I have not been able to obtain traces as @tych0 suggested, and I can't find anything suspicious in drawer.py either.
Any kind soul to help here 🙏? This is very consistent: memory fills up every time my monitors go off.
Not sure why
qtile --versionsays0.25.1.dev0+g005da458.d20240413instead of0.25.2:Version
0.25.1.dev0+g005da458.d20240413
Backend
X11 (default)
Config
Click to show config
Logs
Last 100 log lines:
Required
The text was updated successfully, but these errors were encountered: