Frequent bubblewrap crashes on Fedora Linux 43

[Hasshu]

Multiple users are encountering frequent bubblewrap crashes on Fedora Linux 43. See, e.g., this Fedora Discussion topic.

The cause is not yet understood, but it might be related to a certain glycin update. From my own experience, locking the screen (on GNOME 49) while Steam is running is a sure way to crash bubblewrap.

For what it's worth, the crashes do not seem to happen on F42. Aside from glycin, the only obvious difference would be the bubblewrap version: 0.10 vs. 0.11 on F43.

Incidentally, there are some relevant open RHBZ issues assigned to @cgwalters, so if he could take a look at these, it would be appreciated.

[smcv]

See, e.g., this Fedora Discussion topic

The only concrete information I found by skim-reading that topic, other than a lot of "it crashes" and "it shouldn't crash", was that one reporter saw a backtrace in which bwrap was killed by SIGSYS during __GI_execve(). SIGSYS usually means that some larger component has set up a seccomp filter (bwrap --seccomp or bwrap --add-seccomp-fd, or using libseccomp itself) but the filter does not allow all of the syscalls that are required.

If it's that, then altering bwrap will not fix the issue. Instead, whatever component provided the seccomp filter (possibly glycin?) will need to be updated to either allow the syscalls that are required, or deny unknown syscalls by making them fail with ENOSYS instead of killing the process with SIGSYS, or both.

The seccomp filter is mostly applied to the command that bubblewrap wraps, rather than bubblewrap itself, but as a result of how the relevant kernel interfaces work, the last few lines of bubblewrap code also get run under the filter. This means that the filter needs to allow write(2) and execvp(3).

One possible reason for this to regress is if an updated version of glibc has changed the implementation of execvp(3), so that it uses different syscalls behind the scenes (perhaps it's now using execveat(2) or something): in glibc, function calls that look like they ought to be a syscall do not necessarily map 1:1 to syscalls. The seccomp filter being added by a larger component might have included the syscalls used by the old implementation, but not the syscalls used by the new implementation.

From my own experience, locking the screen (on GNOME 49) while Steam is running is a sure way to crash bubblewrap.

Please get a backtrace for the specific crash that you are experiencing. If you can find out what parent process launched bubblewrap, that is also important information.

[smcv]

I see that https://gitlab.gnome.org/GNOME/glycin/-/blob/main/glycin/src/sandbox.rs only allows execve(2) and not execveat(2), so if an updated glibc implements execvp(3) using execveat(2), that would have the described symptom.

[smcv]

--setenv HOME /tmp-home --setenv XDG_RUNTIME_DIR /tmp-run

Yes, this looks like glycin.

[Hasshu]

First of all, thanks for the prompt response.

Please get a backtrace for the specific crash that you are experiencing. If you can find out what parent process launched bubblewrap, that is also important information.

A noob question: would the gnome-abrt entries have the required information, or should I look elsewhere?

Yes, this looks like glycin.

Aha... Is this something that should be fixed on their end?

[smcv]

A noob question: would the gnome-abrt entries have the required information, or should I look elsewhere?

The stack traces in https://discussion.fedoraproject.org/t/everytime-i-boot-into-fedora-43-i-get-an-error-about-bwrap-crashing/172742/4 and https://discussion.fedoraproject.org/t/everytime-i-boot-into-fedora-43-i-get-an-error-about-bwrap-crashing/172742/10 are the sort of thing I would need to see to confirm or deny your crashes being the same ones seen by Jon Brown. abrt can probably give you similar information, but I don't use Fedora and abrt is a Fedora-specific tool, so I don't know.

Yes, this looks like glycin.

Aha... Is this something that should be fixed on their end?

If the crashes you're seeing have the same root cause as the stack traces from Jon Brown linked above, then yes. If my diagnosis is correct, then there is no change that bubblewrap can make that would resolve this. Instead, its caller (which I'm 95% sure is glycin) will need to give it a slightly different seccomp filter that doesn't result in bubblewrap being killed.

I would recommend reporting this to the Fedora / Red Hat bug tracking system as well: the maintainers of Fedora know what versions of each relevant component they have packaged, whether they have applied any patches, and so on.

[Hasshu]

Alright, I hope it's going to be readable... Here's a relevant excerpt from journalctl after triggering a "Steam crash":

gnome-shell[4016]: Type GITypeInfo of property AccountsService.User::password-mode does not match return type GITypeInfo of getter get_password_mode. Falling back to slow path
audit[32150]: SECCOMP auid=1000 uid=1000 gid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=32150 comm="bwrap" exe="/usr/bin/bwrap" sig=31 arch=c000003e syscall=59 compat=0 ip=0x7f35703bad4b code=0x0
audit[32149]: SECCOMP auid=1000 uid=1000 gid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=32149 comm="bwrap" exe="/usr/bin/bwrap" sig=31 arch=c000003e syscall=61 compat=0 ip=0x7f3570368c5e code=0x0
audit[32149]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=32149 comm="bwrap" exe="/usr/bin/bwrap" sig=31 res=1
audit[32150]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=32150 comm="bwrap" exe="/usr/bin/bwrap" sig=31 res=1
systemd-coredump[32151]: Process 32149 (bwrap) of user 1000 terminated abnormally with signal 31/SYS, processing...
systemd-coredump[32152]: Process 32150 (bwrap) of user 1000 terminated abnormally with signal 31/SYS, processing...
systemd[1]: Created slice system-systemd\x2dcoredump.slice - Slice /system/systemd-coredump.
audit: BPF prog-id=119 op=LOAD
audit: BPF prog-id=120 op=LOAD
audit: BPF prog-id=121 op=LOAD
systemd[1]: Started systemd-coredump@0-1-32151_32152-0.service - Process Core Dump (PID 32151/UID 0).
audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-1-32151_32152-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
audit: BPF prog-id=122 op=LOAD
audit: BPF prog-id=123 op=LOAD
audit: BPF prog-id=124 op=LOAD
systemd[1]: Started systemd-coredump@1-2-32152_32153-0.service - Process Core Dump (PID 32152/UID 0).
audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@1-2-32152_32153-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
systemd-coredump[32154]: [🡕] Process 32150 (bwrap) of user 1000 dumped core.
                                                   
                                                   Module /usr/bin/bwrap from rpm bubblewrap-0.11.0-2.fc43.x86_64
                                                   Module libpcre2-8.so.0 from rpm pcre2-10.47-1.fc43.x86_64
                                                   Module libcap.so.2 from rpm libcap-2.76-3.fc43.x86_64
                                                   Module libselinux.so.1 from rpm libselinux-3.9-5.fc43.x86_64
                                                   Stack trace of thread 2:
                                                   #0  0x00007f35703bad4b execve (libc.so.6 + 0xc0d4b)
                                                   #1  0x00007f35703baf82 __execvpe_common.isra.0 (libc.so.6 + 0xc0f82)
                                                   #2  0x00007f35703bb282 execvpe (libc.so.6 + 0xc1282)
                                                   #3  0x0000561bdc3f6f47 main (/usr/bin/bwrap + 0x2f47)
                                                   #4  0x00007f35702fd5b5 __libc_start_call_main (libc.so.6 + 0x35b5)
                                                   #5  0x00007f35702fd668 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3668)
                                                   #6  0x0000561bdc3f8875 _start (/usr/bin/bwrap + 0x4875)
                                                   ELF object binary architecture: AMD x86-64
systemd-coredump[32153]: [🡕] Process 32149 (bwrap) of user 1000 dumped core.
                                                   
                                                   Module /usr/bin/bwrap from rpm bubblewrap-0.11.0-2.fc43.x86_64
                                                   Module libpcre2-8.so.0 from rpm pcre2-10.47-1.fc43.x86_64
                                                   Module libcap.so.2 from rpm libcap-2.76-3.fc43.x86_64
                                                   Module libselinux.so.1 from rpm libselinux-3.9-5.fc43.x86_64
                                                   Stack trace of thread 1:
                                                   #0  0x00007f3570368c5e __internal_syscall_cancel (libc.so.6 + 0x6ec5e)
                                                   #1  0x00007f3570368c84 __syscall_cancel (libc.so.6 + 0x6ec84)
                                                   #2  0x00007f35703d8b4f wait4 (libc.so.6 + 0xdeb4f)
                                                   #3  0x0000561bdc3f7898 main (/usr/bin/bwrap + 0x3898)
                                                   #4  0x00007f35702fd5b5 __libc_start_call_main (libc.so.6 + 0x35b5)
                                                   #5  0x00007f35702fd668 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3668)
                                                   #6  0x0000561bdc3f8875 _start (/usr/bin/bwrap + 0x4875)
                                                   ELF object binary architecture: AMD x86-64
audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-1-32151_32152-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@1-2-32152_32153-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
systemd[1]: systemd-coredump@0-1-32151_32152-0.service: Deactivated successfully.
steam[30043]: WARNING: Glycin running without sandbox.
systemd[1]: systemd-coredump@1-2-32152_32153-0.service: Deactivated successfully.
audit: BPF prog-id=124 op=UNLOAD
fedora audit: BPF prog-id=123 op=UNLOAD
fedora audit: BPF prog-id=122 op=UNLOAD
fedora audit: BPF prog-id=121 op=UNLOAD
fedora audit: BPF prog-id=120 op=UNLOAD
fedora audit: BPF prog-id=119 op=UNLOAD
fedora abrt-dump-journal-core[1874]: Failed to obtain all required information from journald
fedora abrt-dump-journal-core[1874]: Failed to obtain all required information from journald
fedora abrt-server[32208]: Unsupported container technology
fedora abrt-server[32208]: Lock file '.lock' was locked by process 32212, but it crashed?
fedora abrt-applet[4462]: g_app_info_should_show: assertion 'G_IS_APP_INFO (appinfo)' failed
fedora abrt-notification[32276]: [🡕] Process 32150 (bwrap) crashed in execve()
fedora abrt-server[32209]: Unsupported container technology
fedora abrt-server[32209]: Lock file '.lock' was locked by process 32281, but it crashed?
fedora abrt-applet[4462]: g_app_info_should_show: assertion 'G_IS_APP_INFO (appinfo)' failed
fedora abrt-notification[32323]: [🡕] Process 32149 (bwrap) crashed in __internal_syscall_cancel()

As for gnome-abrt...

reason              bwrap killed by SIGSYS
crash_function      __internal_syscall_cancel
cmdline             bwrap --unshare-all --die-with-parent --chdir / --ro-bind /usr /usr --dev /dev --ro-bind-try /etc/ld.so.cache /etc/ld.so.cache --ro-bind-try /nix/store /nix/store --tmpfs /tmp-home --tmpfs /tmp-run --clearenv --setenv HOME /tmp-home --setenv XDG_RUNTIME_DIR /tmp-run --setenv XDG_RUNTIME_DIR /run/user/1000 --symlink /usr/lib /lib --symlink /usr/lib64 /lib64 --seccomp 133 /usr/bin/true

And here's "core_backtrace":

{   "signal": 31
,   "executable": "/usr/bin/bwrap"
,   "stacktrace":
      [ {   "crash_thread": true
        ,   "frames":
              [ {   "address": 139867492617310
                ,   "build_id": "f410d74336e800b1795d2fd97ddfd36c16ad6546"
                ,   "build_id_offset": 453726
                ,   "function_name": "__internal_syscall_cancel"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 139867492617348
                ,   "build_id": "f410d74336e800b1795d2fd97ddfd36c16ad6546"
                ,   "build_id_offset": 453764
                ,   "function_name": "__syscall_cancel"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 139867493075791
                ,   "build_id": "f410d74336e800b1795d2fd97ddfd36c16ad6546"
                ,   "build_id_offset": 912207
                ,   "function_name": "wait4"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 94677659252888
                ,   "build_id": "2a24855f56b62fc4e24688e54093f5f81e274c98"
                ,   "build_id_offset": 14488
                ,   "function_name": "main"
                ,   "file_name": "/usr/bin/bwrap"
                }
              , {   "address": 139867492177333
                ,   "build_id": "f410d74336e800b1795d2fd97ddfd36c16ad6546"
                ,   "build_id_offset": 13749
                ,   "function_name": "__libc_start_call_main"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 139867492177512
                ,   "build_id": "f410d74336e800b1795d2fd97ddfd36c16ad6546"
                ,   "build_id_offset": 13928
                ,   "function_name": "__libc_start_main@@GLIBC_2.34"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 94677659256949
                ,   "build_id": "2a24855f56b62fc4e24688e54093f5f81e274c98"
                ,   "build_id_offset": 18549
                ,   "function_name": "_start"
                ,   "file_name": "/usr/bin/bwrap"
                } ]
        } ]
}

[Hasshu]

steam[30043]: WARNING: Glycin running without sandbox.

Hmm...

I would recommend reporting this to the Fedora / Red Hat bug tracking system as well: the maintainers of Fedora know what versions of each relevant component they have packaged, whether they have applied any patches, and so on.

As I mentioned earlier, there are open issues filed against bubblewrap on Red Hat Bugzilla, but its maintainer has yet to act on these.

[smcv]

audit[32150]: SECCOMP auid=1000 uid=1000 gid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=32150 comm="bwrap" exe="/usr/bin/bwrap" sig=31 arch=c000003e syscall=59 compat=0 ip=0x7f35703bad4b code=0x0

Looks like syscall 59 (which seems to be pipe2()) was prevented by a seccomp filter...

audit[32150]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=32150 comm="bwrap" exe="/usr/bin/bwrap" sig=31 res=1

... killing the process with SIGSYS.

audit[32149]: SECCOMP auid=1000 uid=1000 gid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=32149 comm="bwrap" exe="/usr/bin/bwrap" sig=31 arch=c000003e syscall=61 compat=0 ip=0x7f3570368c5e code=0x0

And the same thing happened for syscall 61, which seems to be getdents64().

The backtraces from systemd-coredump indicate that there might be different syscalls in progress, wait4() and execve(). I don't know why the inconsistency, but presumably one or the other is correct...

Changing bwrap is unlikely to solve this. pipe2(), getdents64(), wait4() and execve() are all basic, normal syscalls which programs ought to be able to use safely.

So this is a problem with some component that is either running bwrap under the influence of a seccomp filter that would forbid some of those syscalls, or telling bwrap to apply a seccomp filter that forbids some of those syscalls. It might be glycin, or it might be something else.

steam[30043]: WARNING: Glycin running without sandbox.

Hmm...

A plausible explanation for this is that Steam uses GTK, which uses gdk-pixbuf, which uses glycin (at least, it does on Fedora). And glycin tried to run its image processing in a bubblewrap sandbox, but bubblewrap crashed (or more precisely, was killed by a seccomp filter for using a syscall that the filter doesn't like), so glycin falls back to doing its image processing unsandboxed, with this warning.

[smcv]

I don't think there's anything that bubblewrap is going to be able to do to solve this.

[Hasshu]

@smcv Thanks for deciphering this! Looks like I'm going to bug (*cough*) the glycin devs.