Prebuilt Electron binaries segfault at startup on Arch Linux with glibc 2.28

[K900]

• Electron Version: 2.0.5, 2.0.6
• Operating System (Platform and Version): Arch Linux latest
• Last known working Electron version: ?

Expected Behavior
Electron runs correctly.

Actual behavior
Electron segfaults on start, before anything useful happens.

To Reproduce
Download the prebuilt Electron release (electron-2.0.x-linux-x64.zip); unzip it; run ./electron; observe the fireworks.

Additional Information
The issue seems to be caused by the glibc 2.28 upgrade on Arch - downgrading to glibc 2.27 makes Electron work. I'm not really sure what the issue itself is, as Electron crashes even before Breakpad kicks in, so no minidump is produced, and I don't have a machine powerful enough to produce a debug build in a reasonable amount of time (and it's likely that a debug build produced on my machine with glibc 2.28 will work fine anyway, being linked against the correct glibc). Distribution-provided binaries (the electron package in the repository) work fine, even though they were not rebuilt to match the glibc upgrade.

[welcome]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.

To help make it easier for us to investigate your issue, please follow the contributing guidelines.

[eli-schwartz]

The electron package from the Arch Linux community repository was actually built using glibc 2.27, nice try. ;) (glibc tries pretty hard to maintain ABI compatibility, you should never need to rebuild software in order to work with newer glibc -- the fact that this breaks indicates either a glibc bug or a something in the software stack that depends on private details of glibc when it shouldn't.)

$ bsdtar -xOf /var/cache/pacman/pkg/electron-2.0.6-1-x86_64.pkg.tar.xz .BUILDINFO| grep glibc
installed = glibc-2.27-3-x86_64

So, it's entirely likely that rebuilding it on Arch Linux with glibc 2.28 would have the same issues, and that the issue lies in the build process. For the record, our package is built using https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/electron and you can see various patches we apply in order to use system dependencies, in the same directory of the git repository as the PKGBUILD file.

[eli-schwartz]

More context: https://bugs.archlinux.org/task/59550 (Arch Linux bugreport).

it would appear to be coming from libnode.so, and it's not just electron either -- sone R packages do the same.

[K900]

So I've done some testing and posted the results on the VSCode issue linked: microsoft/vscode#55934

TL;DR:

• Electron + Arch patches + Arch sysroot = works fine
• Electron + no Arch patches + Debian sysroot (default) = segfault
• Electron + minimal patches + Arch sysroot = ? (wip)

[K900]

Another update: did a clean build outside of the Arch packaging tools, but with most of the Arch patches applied, and it worked (which isn't really surprising). I'm not quite sure where the Debian sysroot images are coming from or how they are produced, but upgrading glibc in those might be an interesting thing to try.

[K900]

I might actually just try bisecting glibc and seeing when it started crashing.

[K900]

I've bisected glibc to this commit. The versions before it run fine, the version including it cause a segfault. I'm not entirely sure what the reason is. Trying to build the latest glibc-2.28 tag now with it reverted.

Edit: disregard that, I messed up. Re-testing now...

[eli-schwartz]

Per discussion with azanella on freenode's #glibc channel:

I did some debug to check what is happening and it seems that the PLT to nghttp2_session_callbacks_set_on_frame_recv_callback (for libnode.so initialization) is branching to a bogus value, it came from symbol resolution in fact
there is really something strange with libnode.so files, on both skypeforlinux and electron
it is calling nghttp2_session_callbacks_set_on_frame_recv_callback through PLT (and also internally on the library)
but it does not seem to provide the symbol itself

$ nm -D /usr/share/skypeforlinux/libnode.so | grep nghttp2_session_callbacks_set_on_frame_
00000000009dbaf0 A nghttp2_session_callbacks_set_on_frame_not_send_callback
0000000000d20020 A nghttp2_session_callbacks_set_on_frame_recv_callback
0000000001100230 T nghttp2_session_callbacks_set_on_frame_send_callback

while nghttp2_session_callbacks_set_on_frame_send_callback is pointing to text segment (and thus code), nghttp2_session_callbacks_set_on_frame_recv_callback is an absolute symbol
this does not seem right
we changed how to handle absolute symbol to fix BZ #19818
yeap, reverting e7feec374c635b6a29d65c39ae5e1855528fed39 seems to avoid this specific issue, but loader fails now with another one...

And the conclusion was that this should be brought up for discussion on libc-alpha.

[zatrazz]

Could you check by reverting e7feec374c635b6a29d65c39ae5e1855528fed39 ? I would to get some freedback before address it on libc-alpha.

[K900]

This is the commit that I traced it down to when bisecting too. I'll try just reverting it and see what happens.

[K900]

Just reverting that commit (on top of the latest 2.28 tag) seems to fix things. Doesn't explain how the binary got so messed up though...

[K900]

Also, repo Electron's libnode.so seems fine:

❯ nm -D /usr/lib/electron/libnode.so | grep nghttp2_session_callbacks_set_on_frame_
00000000010363c0 T nghttp2_session_callbacks_set_on_frame_not_send_callback
0000000000bbd990 T nghttp2_session_callbacks_set_on_frame_recv_callback
00000000011cf540 T nghttp2_session_callbacks_set_on_frame_send_callback

[zatrazz]

Does Electron's libnode.so work correctly with glibc 2.28?

[K900]

The one in the official binary releases from this repo does not; the one in the Arch package does, as it's built with a different toolchain. I'm actually trying to build upstream Electron with Arch clang right now, will see where this goes, as it seems like an Electron issue and not a glibc issue to me.

[K900]

So yeah, it's definitely a toolchain issue, more specifically a linker issue. Simply replacing vendor/llvm-build/Release+Asserts/bin/lld with a symlink to system lld (version 6.0.1) results in a functional build. I'm guessing it's a linker bug, so glibc is correct in barfing here. Now to figure out where the Debian sysroot comes from...

[zatrazz]

Right, I saw reports where libnode.so is also failing on other projects and it seems to be the same issue. I was trying to reproduce the build failure with GNU linkers, but without much success.

[K900]

It seems to be an LLD bug. I'm trying a build with a newer prebuilt clang (and thus lld) version now.

[K900]

Good news: the tests pass with an updated lld, and the resulting libnode.so isn't horribly bugged.

[K900]

@pcc on #llvm IRC found the exact revision that fixed it: https://github.com/llvm-project/lld/commit/1adba6d5d2f5f51d0719e81f71461437ecc1bd59

[ghjm]

Electron apps including Slack are also broken on Fedora 29 for the same reason. I built a glibc with @mfonville's patch, which solves the problem for me. This build is available here: https://copr.fedorainfracloud.org/coprs/ghjm/glibc/. You can install the updated glibc using dnf copr enable ghjm/glibc && dnf update glibc (and then probably reboot).

[allanlewis]

@ghjm Fedora has released a new glibc: does this include the patch or do you need to update your COPR? I'll refuse that update for now.

[ghjm]

@allanlewis Fedora has said they are not going to apply this patch. See https://bugzilla.redhat.com/show_bug.cgi?id=1653832 for details.

I'm pretty sure I never signed up to maintain this forever, but I've started a new build here: https://copr.fedorainfracloud.org/coprs/ghjm/glibc/build/838399/. Assuming it succeeds, the new builds should be available in about an hour.

[allanlewis]

I'm pretty sure I never signed up to maintain this forever...

Absolutely - I'm grateful you started it at all!

...but I've started a new build here...

Thanks, @ghjm!

For now I've added exclude = glibc* to my updates repo config. For others' benefit:

# dnf install -y 'dnf-command(config-manager)'
# dnf config-manager --setopt 'updates.exclude=glibc*' --save updates

[perlun]

For reference, users of Slack should upgrade to version 3.3.7 released earlier this week, which has this fix: https://slack.com/release-notes/linux

[sofianguy]

Thank you for taking the time to report this issue and helping to make Electron better.

The version of Electron you reported this on has been superseded by newer releases.

If you're still experiencing this issue in Electron v4.2.x or later, please add a comment specifying the version you're testing with and any other new information that a maintainer trying to reproduce the issue should know.

I'm setting the blocked/need-info label for the above reasons. This issue will be closed 7 days from now if there is no response.

Thanks in advance! Your help is appreciated.

[eli-schwartz]

The electron 2.x release line was fixed in #13988 by updating the version of lld used to build the prebuilt electron release, and electron 3.x was never affected as it already used the newer lld.

So this is definitely resolved.

[cjdg]

There is an regression, latest prebuiilt clients of teams, electron, and rambox, shows a segfault

Ubuntu 20.04.1 x64

[nornagon]

@cjdg would you mind opening a new issue with some more details about the segfault? Ideally a stack trace or core dump.