Xpra Server Crash with GLib >= 2.76

[weingo2]

Describe the bug
I run xpra on Arch Linux on a desktop computer at home, so I can attach a thin client to it wherever I go. I start an xpra server with an xterm then open and close windows from that xterm (running tmux too). It may seem a bit archaic, but I primarily use cli tools that bring up plot windows. I also normally bind a websocket session to a port on the desktop machine then, forward that port over ssh to my local machine then attach to that port. This method has been easier for me than using the native xpra ssh.

Recently, I updated Arch Lunix on the desktop, and the xpra server started sporadically crashing when I open or close windows. When xpra crashes it seems to be segment faulting somewhere inside the GTK library and specifically inside glib. The session becomes "UNKNOWN" and I have to restart the session.

The stack trace from one of these crashes is here:
gdb.txt
A crash does not occur with every window open or close but seems to occur randomly.

To Reproduce
Steps to reproduce the behavior:

1. xpra start --start=xterm --bind-ws=localhost:10001
2. Forward the port from the server to the client with ssh -L ...
3. xpra attach ws://localhost:10001
4. open and close windows from xterm

System Information (please complete the following information):

• Arch Linux
• Arch Linux or Windows 11
• 4.4.4
• 4.4.4

[totaam]

FYI: unless you also attach with a browser session and it is having problems with tcp socket upgrades to websockets, using bind-ws is counter productive and just a waste of bandwidth.

TIL:

#0  0x00007fd2cfb8089f in gdk_x11_display_translate_event (translator=<optimized out>, display=0x5590dd7e6380, event=0x5590dde1ab20, xevent=0x7ffe0ffec570) at ../gtk/gdk/x11/gdkdisplay-x11.c:653

GDK is choking on parsing of an X11 event.

You may be able to get more details by running the xpra server with -d all, though the log will grow very big, very quickly..

[weingo2]

Thank you for the information! I just figured websockets would be better because of the bidirectional channel.

So, I haven't been able to get it to crash in debug mode. I even started a server without debug mode again to check if the bug got resolved somehow, and it still crashed. I'm going to keep using my normal setup in debug mode to see if it ever crashes. I should have enough disk space to handle the log file. Any other things to try?

[totaam]

I just figured websockets would be better because of the bidirectional channel.

They all are - xpra cannot work without.
https://github.com/Xpra-org/xpra/tree/master/docs/Network

Any other things to try?

Yes, the usual: disable every feature and every encoder (without debug mode) and see if it still crashes.

[weingo2]

server_tail.log

It finally died while in debug mode. I attached the last 1000 lines of the large log file that was produced. It should contain all the context of the error.

[totaam]

Looks like you pressed control-C in window 0x80000c:

2023-04-13 10:00:38,525 handle_key((1, True, 'c', 99, 54, ['control', 'mod2'], False, True))

Which caused a screen update:

2023-04-13 10:00:38,529 parse_event(..)=<X11:DamageNotify {'send_event': '0', 'serial': '0x35fe9', 'delivered_to': '0x80000c', 'window': '0x80000c', 'damage': '4194331', 'x': '34', 'y': '1282', 'width': '16', 'height': '32'}>

And window 0x600001 to close:

2023-04-13 10:00:38,539 parse_event(..)=<X11:UnmapNotify {'send_event': '0', 'serial': '0x35ff8', 'delivered_to': '0x600001', 'window': '0x600001'}>
...
2023-04-13 10:00:38,542 unmanaging window: WindowModel(0x600001) (<GdkX11.X11Window object at 0x7fe19c0fa480 (GdkX11Window at 0x55dc118b3680)> - <GdkX11.X11Window object at 0x7fe19c0fb080 (GdkX11Window at 0x55dc118b3920)>)

The window model cleanup code hits a bunch of errors we ignore since the window is already gone - I think that's safe:

xpra.gtk_common.error.XError: XError: BadDamage (invalid Damage parameter)

Then we get some more screen updates (DamageNotify on 0x80000c) - presumably some terminal window?

What does look a little bit suspicious is a bunch of cannot get gdk window for ..:

2023-04-13 10:00:38,562 cannot get gdk window for <GdkX11.X11Display object at 0x7fe1c769b880 (GdkX11Display at 0x55dc1119c7f0)>, 0x402007: constructor returned NULL

Perhaps b1d2e9e introduced a bug - in which case the commit above should fix that.
Does that help? (without -d all to try to trigger it more readily)

[weingo2]

Often I just click control-c on my plotting to clear everything. However, it does crash sometimes when I close with window with the window X close button. I did only have one other window open at the time, an xterm window.

Your explanation of the log matches the observed behavior. When it crashes, the window I am trying to close closes, and the remaining windows flash before everything disappears. Naively I would think it still expects the window that just closed to exist and it seg faulting on a null pointer for that window. This doesn't explain why it only happens sometimes, though.

I just built xpra with that commit and installed it. It crashed using the same methodology...

What I have been trying to figure out is what changed about my setup. I've been using this xpra setup for a couple of years now with no issues then all the sudden, a month ago, this issue showed up. Is it a gtk issue? A gtk python binding issue? All these packages seem to have received small updates over the past month, so I couldn't narrow it down that way.

[totaam]

What I have been trying to figure out is what changed about my setup.

I believe that Arch must have upgraded an X11 or GTK library that makes it much more likely to hit the race condition that triggers this bug.

[weingo2]

Race condition makes sense. It's also a lot more reliable in debug mode, so the act of printing out a log message must slow down the right thread preventing the race condition. Any other debug suggestions? Honestly, I'm happily running in debug mode right now. I have plenty of disk space to handle the large log file it produces and don't notice any performance hits.

[totaam]

This may still trigger the bug and give us more details:

XPRA_X11_DEBUG_EVENTS="*" xpra start -d x11,gtk ...

[weingo2]

I ran this
XPRA_X11_DEBUG_EVENTS="*" xpra start -d x11,gtk --start=xterm --bind-tcp=localhost:10001
and spawned and closed windows in a loop until it crashed.
Here is the last 1000 lines of the log file it produced server_1000.log

[nentibusarchitectura]

I can trigger the same crash in Arch by opening and closing the SciTE editor, it happens all the time, can't tell for how long this has been the case, it happens with xpra 4.4.4-1 on my Arch system.

Simply opening SciTE and then File->Exit is enough to crash xpra on my machine.

[totaam]

Assuming that I find the time to install ArchLinux in a VM, are there any applications in the default repositories that can trigger this bug?
Normally, this type of problem can be reproduced with gtkperf -a - does it reproduce it for you?

[weingo2]

I think any program that spawns windows has the potential to trigger this bug. I ran gtkperf -a from installed from the AUR (https://aur.archlinux.org/gtkperf.git) inside a shell script while loop until it crashed. It took a minute but eventually exhibited the bug's behavior.

Is there any other debug information I can provide you? If there is anything else you want to try, I would be happy to check out a development branch (or however you would want to distribute code) you create and test it.

[nentibusarchitectura]

Assuming that I find the time to install ArchLinux in a VM, are there any applications in the default repositories that can trigger this bug? Normally, this type of problem can be reproduced with gtkperf -a - does it reproduce it for you?

I can check this later too, I think (not hundred percent sure) that Firefox also triggered it for me, but SciTE was 100% consistent, it was to led me check if this issue had been reported.

[nentibusarchitectura]

Just did extensive testing xpra segfaults pretty much when closing or exiting any application, examples:

open xterm, type exit = crash
open firefox go to file->quit = crash
open firefox close window from x buton = crash
open glxgears exit by pressing esc = crash
etc...

Sometimes it doesn't crash, but only some times like 1 in 20.

I can provide a full core dump if required via private link.

xpra_crash_log.txt

At some point in the past it used to work fine in Arch, but since I do not do work in my Arch box often I have lost track of when and what version used to work.

It was by chance that I tested it a couple of days ago in Arch.

[thoefl]

Can confirm such segfaults occur on the current Gentoo testing (~amd64), but not on the stable branch. This might help narrowing down the version differences, e.g.: https://packages.gentoo.org/packages/dev-libs/glib

[totaam]

Gentoo might be easier as a testbed.
I remember seeing an X11 commit recently that had to do with enabling some threading by default (could be xorg-server or libX11?), emerge-ing the older X11 ebuild might be enough to resolve this bug and tell us where the regression comes from.
But it would be easier if we had a 100% reliable reproducer..

[thoefl]

libX11 and xorg-server are the same in stable and testing, but I just tried updating only glib (and its dependency gdbus-codegen) from 2.74.6 (stable) to 2.76.1 (testing), and that alone was enough to trigger segmentation faults in libgdk. I've attached the relevant dmesg output and the complete dependency trees of the broken and working configurations, but the only differences should be glib and gdbus-codegen.

This is using the older 4.3.4 release, but I had the same behaviour building from the current master branch.

dmesg.txt
deps_segfault.txt
deps_working.txt

EDIT:
Possibly related, segfaults also reported in XFCE using glib 2.76:
https://gitlab.gnome.org/GNOME/glib/-/issues/2941
https://gitlab.xfce.org/xfce/xfce4-session/-/issues/166

[totaam]

@nentibusarchitectura @weingo2 can you try downgrading glib to see if that fixes it?

I've scoured those gitlab links and there's nothing related to what we're doing here.
The bugs seem to be related to gio and the file API whereas we're doing X11 window management..

For the record, the X11 threading change I was thinking about is this one: XInitThreads in library constructor breaks Motif! and I really don't think that this is relevant here: GTK does not call it and we never call X11 functions from other threads anyway, we even have thread checks in all the Cython bindings.

[nentibusarchitectura]

can you try downgrading glib to see if that fixes it?

My box uses glibc 2.37, I can only downgrade without screwing the system down to 2.36 which doesn't fix the issue, the crashes are still there.

Tried to downgrade xpra I can go as low as 4.3.3-r0 and the crashes are still there too.

Anything lower than 4.3.3 gives me python errors due to the system not meeting requirements due to rolling nature of dependencies in Arch.

Hope this helps.

[weingo2]

The arch changelog shows an upgrade from glib2 2.74.6-1 to glib2 2.76.1-2 over the last couple months.

I, too, tried to downgrade glib2 according to https://wiki.archlinux.org/title/Downgrading_packages by using the package cache method. That didn't work, and I quickly wound up with too many dependency conflicts. I then used the Arch Linux Archive method from https://wiki.archlinux.org/title/Arch_Linux_Archive to downgrade me to March 1st, 2023. This date has the older glib2-2.74.6-1 package in it. I also had to remove libgirepository with sudo pacman -Rd --nodeps libgirepository since it seems to have split out of another package and has conflicting files in it. This seemed to work but after rebooting my machine, Arch won't boot. The kernel won't load with efi for some reason... I am using a different distro with grub to boot arch, so that could be the problem.

Needless to say, I am out of commission until I solve the booting issues. If I don't get to it today, I won't be able to get to it until the Monday after next.

[weingo2]

So, I managed to recover my system and get the March 12th, 2023 Arch Linux Archive working. The day after this archive version, the glib library was upgraded. My system does not crash with this archive, I have been running gtkperf -a in a loop for 30 min with no crashes. This means that glib2 2.74.6-1 was stable on Arch for xpra until it was upgraded after March 12th. I can't say for sure that there is no other library causing problems but based on the Gentoo information from @thoefl I think it's safe to say that the glib library is causing the trouble.

[totaam]

I think it's safe to say that the glib library is causing the trouble.

I concur.
I hope that versions between 2.74.x and 2.76.x are bisectable.

[borneoa]

Updating Arch installation with glib2 2.76.2-1 from "testing" repository does NOT solve the issue

[jpVm5jYYRE1VIKL]

Got same issue on Arch . before it was very old ubuntu switched to arch and now all broken cannot start any application. Constant crashes.

this is my startup command :

xpra start ssh:smth@10.10.10.254:10 --ssh=ssh --env=XPRA_FORCE_XSETINPUTFOCUS=1 --encoding=h264\
 --webcam=no --speaker=disabled --microphone=disabled --border=yellow --pulseaudio=no

Crash log :

(Xpra:13727): Gdk-CRITICAL **: 15:40:35.165: gdk_window_thaw_toplevel_updates: assertion 'window->update_and_descendants_freeze_count > 0' failed
(..)
  File "/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py", line 1689, in main
    return _Gtk_main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/xpra/x11/models/core.py", line 754, in do_xpra_client_message_event
    if not self.process_client_message_event(event):
  File "/usr/lib/python3.10/site-packages/xpra/x11/models/window.py", line 654, in process_client_message_event
    return super().process_client_message_event(event)
  File "/usr/lib/python3.10/site-packages/xpra/x11/models/base.py", line 611, in process_client_message_event
    log.info("Unhandled _NET_WM_STATE request: '%s'", event, atom1)
  File "/usr/lib/python3.10/site-packages/xpra/log.py", line 461, in info
    self.log(logging.INFO, msg, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/xpra/log.py", line 452, in log
    global_logging_handler(self.logger.log, self.level_override or level, msg, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/xpra/log.py", line 142, in standard_logging
    log(level, msg, *args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1547, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/usr/lib/python3.10/site-packages/xpra/log.py", line 504, in emit
    self.handleError(record)
Message: "Unhandled _NET_WM_STATE request: '%s'"
Arguments: (<X11:ClientMessage {'send_event': '1', 'serial': '0x2993', 'delivered_to': '0x800004', 'window': '0x800004', 'message_type': '_NET_WM_STATE', 'format': '32', 'data': '(0, 432, 0, 1, 0)'}>, '_KDE_NET_WM_STATE_SKIP_SWITCHER')
2023-05-04 15:43:48,849  event<X11:ClientMessage {'send_event': '1', 'serial': '0x2993', 'delivered_to': '0x800004', 'window': '0x800004', 'message_type': '_NET_WM_STATE', 'format': '32', 'data': '(0, 432, 0, 1, 0)'}>
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:1689: Warning: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
  return _Gtk_main(*args, **kwargs)
2023-05-04 15:44:02,491 client   1 @36.728 window 3 has been moved to monitor 0: GSM DP-2 2560x1440 at 0,1440
2023-05-04 15:44:03,785 client   1 @38.020 Warning: window 3 requested static gravity
2023-05-04 15:44:03,785 client   1 @38.020  this is not implemented yet

Client show following :

2023-05-04 15:40:32,252 running, 1 window
2023-05-04 15:43:59,618 window 3 has been moved to monitor 0: GSM DP-2 2560x1440 at 0,1440
2023-05-04 15:44:00,909 Warning: window 3 requested static gravity
2023-05-04 15:44:00,910  this is not implemented yet
2023-05-04 15:44:16,021 Connection lost
2023-05-04 15:44:16,057 removing unix domain socket '/run/user/1000/xpra/clients/ktotoPC-97332'
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:1689: Warning: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
  return _Gtk_main(*args, **kwargs)
CONNECTION_LOST, reconnecting
xpra initialization error:

[totaam]

So, I tried the GTK approach, which had the merit of being simple to implement but I didn't get very far with it because it caused too many intractable problems with focus and other events going MIA.
The minimal fix above is somewhat bigger than most stable branch fixes, but most of it is boilerplate Cython headers cut&pasted, the actual change to the window model code is minimal: a0cc722#diff-a4160a51eb3e2d697991cbebafcd5941baa16ee796c86f6282271fb1e3a41091

@neuhalje @frej @chewi @thesamesam @weingo2 please give this commit a try, if I don't hear anything in the next 48 hours then 4.4.5 will be released with this fix as its headline.

[frej]

Can confirm that e2dfae9 fixes the segfault I experienced with glib 2.76.2.

[frej]

Unfortunately the fix in e2dfae9, although it fixes the the segfault experienced with glib 2.76.2, appears to prevent Firefox from creating its initial window.

/usr/bin/python3 /usr/bin/xpra start :20 --exit-with-children=yes --no-daemon --start-child=xterm
followed by
xpra attach gives me an xterm. Starting new xterms and exiting them works fine (it previously segfaulted xpra). Starting Firefox from the initial xterm does not produce a new Firefox window. No errors are shown, either from the two instances of xpra nor in the xterm. pstree on the xterm shows the normal set of firefox threads and processes.

This is with 4.4.4 [edit: corrected version] with e2dfae9 cherry picked, as well as with the complete v4.4.x branch at 4a4f45d.

[totaam]

Saw this with XPRA_LOG_MANAGE_FAILURES=1:

2023-05-22 11:07:47,324 failed to setup 0xa0003d
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/xpra/x11/models/core.py", line 311, in call_setup
    self.setup()
  File "/usr/lib64/python3.11/site-packages/xpra/x11/models/window.py", line 192, in setup
    prop_set(self.corral_xid, "_NET_WM_NAME", "utf8", "Xpra-CorralWindow-%#x" % self.xid)
  File "/usr/lib64/python3.11/site-packages/xpra/x11/gtk_x11/prop.py", line 102, in prop_set
    raw_prop_set(xid, key, dtype, dformat, data)
  File "/usr/lib64/python3.11/site-packages/xpra/x11/gtk_x11/prop.py", line 107, in raw_prop_set
    with xsync:
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 196, in __exit__
    trap.Xexit()
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 121, in Xexit
    raise XError(error)
xpra.gtk_common.error.XError: XError: BadWindow (invalid Window parameter)
2023-05-22 11:07:47,325 cannot manage Window 0xa0003d: XError: BadWindow (invalid Window parameter)
(..)
xpra.x11.common.Unmanageable: XError: BadWindow (invalid Window parameter)
2023-05-22 11:07:47,327  window name: Mozilla Firefox
2023-05-22 11:07:47,327  window info: unmapped window Mozilla Firefox (0, 0, 1282, 1026)

That's because Firefox (wrongly - but that's a different issue) uses a 32-bit window with transparency and these are hard to create with the correct arguments.
We were trying to copy the window attributes from the client window, now we just get most of them from the root window instead. (24-bit depth and we inherit the visual)

[weingo2]

The seg fault crash seems to be fixed but I also cannot open firefox, thunderbird, or google-chrome (from the aur). Basic guis I run like tcl/tk programs or runelite seem to be fine.

Running gimp closed the xpra connection but the server was still in the LIVE state, requiring me to reconnect. But once I reconnected, it worked fine.

So, I'd say this specific bug is fixed by a0cc722#diff-a4160a51eb3e2d697991cbebafcd5941baa16ee796c86f6282271fb1e3a41091

Are the firefox, thunderbird, chrome issues worth a separate bug or is it just something you will be aware of in the future?

@totaam Thank you for the bug fix!

[totaam]

Are the firefox, thunderbird, chrome issues worth a separate bug or is it just something you will be aware of in the future?

@weingo2 you need c7ddee5 as per #3822 (comment)

[weingo2]

@totaam That was it. Thank you again for resolving this!

[frej]

v4.4.5 works for me, thanks for the quick fix @totaam!

[borneoa]

Thanks for the fix @totaam ! I cannot replicate anymore the crash with current master 99731ac.

But the log reports some error that looks partially caused by the fix/rework.
(..)
Edit: moved to #1995 (comment)

[totaam]

.. with current master 99731ac.
But the log reports some error that looks partially caused by the fix/rework.

Those are part of #1995, not a huge priority right now as this is a WIP - thanks for reporting those anyway.
The important thing is that these should not be present in 4.4.5

[borneoa]

@totaam I go ahead reporting here; please let me know if you prefer I open new issues.

Edit: moved to #1995 (comment)

[neuhalje]

Late to the party, but my issues have been (nearly) resolved. Only very few (one window closing in about 50) crashes left.

Thanks for the quick fix!

[btasdelen]

I have this issue persisting very frequently. I guess some software is more prone to crashing (MATLAB in my case), but I also had other software cause the crash as well. Both the client and the server are at version 4.4.6. The end of the log reads the following after the crash:

2023-06-28 12:59:01,495 client   1 @00.941 forcing grab for OR window 21, matches {'DIALOG': ['sun-awt-X11']}
2023-06-28 13:00:05,121 client   1 @04.570 forcing grab for OR window 25, matches {'DIALOG': ['sun-awt-X11']}
2023-06-28 13:01:29,268 Warning: failed to resize corral window 0x400438
2023-06-28 13:01:29,268  XError: BadDrawable (invalid Pixmap or Window parameter)
2023-06-28 13:01:30,293 Warning: failed to resize corral window 0x4003c7
2023-06-28 13:01:30,293  XError: BadWindow (invalid Window parameter)

(Xpra:3257847): Gdk-ERROR **: 13:03:32.928: The program 'Xpra' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadWindow (invalid Window parameter)'.
  (Details: serial 111302 error_code 3 request_code 7 (core protocol) minor_code 0)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the GDK_SYNCHRONIZE environment
   variable to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)

@totaam Let me know if you need anything to squash the rest of the bug. It is somewhat unusable at the moment as I keep losing my work due to the crash.

[totaam]

The refactoring that morphed gdk windows into xids was incomplete and caused this bug: c2dfebc

[totaam]

Also broke sync-xvfb: 115fb08

[totaam]

Also: #4195