New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prebuilt Electron binaries segfault at startup on Arch Linux with glibc 2.28 #13972

Open
K900 opened this Issue Aug 7, 2018 · 34 comments

Comments

Projects
None yet
@K900

K900 commented Aug 7, 2018

  • Electron Version: 2.0.5, 2.0.6
  • Operating System (Platform and Version): Arch Linux latest
  • Last known working Electron version: ?

Expected Behavior
Electron runs correctly.

Actual behavior
Electron segfaults on start, before anything useful happens.

To Reproduce
Download the prebuilt Electron release (electron-2.0.x-linux-x64.zip); unzip it; run ./electron; observe the fireworks.

Additional Information
The issue seems to be caused by the glibc 2.28 upgrade on Arch - downgrading to glibc 2.27 makes Electron work. I'm not really sure what the issue itself is, as Electron crashes even before Breakpad kicks in, so no minidump is produced, and I don't have a machine powerful enough to produce a debug build in a reasonable amount of time (and it's likely that a debug build produced on my machine with glibc 2.28 will work fine anyway, being linked against the correct glibc). Distribution-provided binaries (the electron package in the repository) work fine, even though they were not rebuilt to match the glibc upgrade.

@welcome

This comment has been minimized.

welcome bot commented Aug 7, 2018

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.

To help make it easier for us to investigate your issue, please follow the contributing guidelines.

@eli-schwartz

This comment has been minimized.

eli-schwartz commented Aug 7, 2018

The electron package from the Arch Linux community repository was actually built using glibc 2.27, nice try. ;) (glibc tries pretty hard to maintain ABI compatibility, you should never need to rebuild software in order to work with newer glibc -- the fact that this breaks indicates either a glibc bug or a something in the software stack that depends on private details of glibc when it shouldn't.)

$ bsdtar -xOf /var/cache/pacman/pkg/electron-2.0.6-1-x86_64.pkg.tar.xz .BUILDINFO| grep glibc
installed = glibc-2.27-3-x86_64

So, it's entirely likely that rebuilding it on Arch Linux with glibc 2.28 would have the same issues, and that the issue lies in the build process. For the record, our package is built using https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/electron and you can see various patches we apply in order to use system dependencies, in the same directory of the git repository as the PKGBUILD file.

@eli-schwartz

This comment has been minimized.

eli-schwartz commented Aug 7, 2018

More context: https://bugs.archlinux.org/task/59550 (Arch Linux bugreport).

it would appear to be coming from libnode.so, and it's not just electron either -- sone R packages do the same.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

So I've done some testing and posted the results on the VSCode issue linked: Microsoft/vscode#55934

TL;DR:

  • Electron + Arch patches + Arch sysroot = works fine
  • Electron + no Arch patches + Debian sysroot (default) = segfault
  • Electron + minimal patches + Arch sysroot = ? (wip)
@K900

This comment has been minimized.

K900 commented Aug 8, 2018

Another update: did a clean build outside of the Arch packaging tools, but with most of the Arch patches applied, and it worked (which isn't really surprising). I'm not quite sure where the Debian sysroot images are coming from or how they are produced, but upgrading glibc in those might be an interesting thing to try.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

I might actually just try bisecting glibc and seeing when it started crashing.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

I've bisected glibc to this commit. The versions before it run fine, the version including it cause a segfault. I'm not entirely sure what the reason is. Trying to build the latest glibc-2.28 tag now with it reverted.

Edit: disregard that, I messed up. Re-testing now...

@eli-schwartz

This comment has been minimized.

eli-schwartz commented Aug 8, 2018

Per discussion with azanella on freenode's #glibc channel:

I did some debug to check what is happening and it seems that the PLT to nghttp2_session_callbacks_set_on_frame_recv_callback (for libnode.so initialization) is branching to a bogus value, it came from symbol resolution in fact
there is really something strange with libnode.so files, on both skypeforlinux and electron
it is calling nghttp2_session_callbacks_set_on_frame_recv_callback through PLT (and also internally on the library)
but it does not seem to provide the symbol itself

$ nm -D /usr/share/skypeforlinux/libnode.so | grep nghttp2_session_callbacks_set_on_frame_
00000000009dbaf0 A nghttp2_session_callbacks_set_on_frame_not_send_callback
0000000000d20020 A nghttp2_session_callbacks_set_on_frame_recv_callback
0000000001100230 T nghttp2_session_callbacks_set_on_frame_send_callback

while nghttp2_session_callbacks_set_on_frame_send_callback is pointing to text segment (and thus code), nghttp2_session_callbacks_set_on_frame_recv_callback is an absolute symbol
this does not seem right
we changed how to handle absolute symbol to fix BZ #19818
yeap, reverting e7feec374c635b6a29d65c39ae5e1855528fed39 seems to avoid this specific issue, but loader fails now with another one...

And the conclusion was that this should be brought up for discussion on libc-alpha.

@zatrazz

This comment has been minimized.

zatrazz commented Aug 8, 2018

Could you check by reverting e7feec374c635b6a29d65c39ae5e1855528fed39 ? I would to get some freedback before address it on libc-alpha.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

This is the commit that I traced it down to when bisecting too. I'll try just reverting it and see what happens.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

Just reverting that commit (on top of the latest 2.28 tag) seems to fix things. Doesn't explain how the binary got so messed up though...

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

Also, repo Electron's libnode.so seems fine:

❯ nm -D /usr/lib/electron/libnode.so | grep nghttp2_session_callbacks_set_on_frame_
00000000010363c0 T nghttp2_session_callbacks_set_on_frame_not_send_callback
0000000000bbd990 T nghttp2_session_callbacks_set_on_frame_recv_callback
00000000011cf540 T nghttp2_session_callbacks_set_on_frame_send_callback
@zatrazz

This comment has been minimized.

zatrazz commented Aug 8, 2018

Does Electron's libnode.so work correctly with glibc 2.28?

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

The one in the official binary releases from this repo does not; the one in the Arch package does, as it's built with a different toolchain. I'm actually trying to build upstream Electron with Arch clang right now, will see where this goes, as it seems like an Electron issue and not a glibc issue to me.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

So yeah, it's definitely a toolchain issue, more specifically a linker issue. Simply replacing vendor/llvm-build/Release+Asserts/bin/lld with a symlink to system lld (version 6.0.1) results in a functional build. I'm guessing it's a linker bug, so glibc is correct in barfing here. Now to figure out where the Debian sysroot comes from...

@zatrazz

This comment has been minimized.

zatrazz commented Aug 8, 2018

Right, I saw reports where libnode.so is also failing on other projects and it seems to be the same issue. I was trying to reproduce the build failure with GNU linkers, but without much success.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

It seems to be an LLD bug. I'm trying a build with a newer prebuilt clang (and thus lld) version now.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

Good news: the tests pass with an updated lld, and the resulting libnode.so isn't horribly bugged.

@K900

This comment has been minimized.

K900 commented Aug 8, 2018

@pcc on #llvm IRC found the exact revision that fixed it: llvm-project/lld@1adba6d

@timdp

This comment has been minimized.

timdp commented Oct 15, 2018

I hope this comment doesn't derail the conversation into a stream of broken apps. If it does, I apologize.

So for me, this issue breaks the Slack desktop app on Ubuntu cosmic. For the same issue with Atom, an upgrade to the latest beta resolved it, so I assume I just need to wait for the Slack team to roll out a new version?

@solsticedhiver

This comment has been minimized.

solsticedhiver commented Oct 19, 2018

ubuntu 18.10 is released, and atom is broken here because of this issue. and I guess any electron app.

This is gonna be very bad buzz for electron. I am beginning to hate it myself just for this

@d4n3sh

This comment has been minimized.

d4n3sh commented Oct 22, 2018

slack segfaults on 18.10

@Masiosare

This comment has been minimized.

Masiosare commented Oct 22, 2018

For a workaround for ubuntu 18.10 users, see this comment. I can confirm it works.

https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1790966/comments/3

larivierec added a commit to larivierec/android-messages-desktop that referenced this issue Oct 23, 2018

Update Electron version
There is a bug with regards to all electron applications
See link here: electron/electron#13972
@mwik

This comment has been minimized.

mwik commented Oct 25, 2018

For a workaround for ubuntu 18.10 users, see this comment. I can confirm it works.

https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1790966/comments/3

Another workaround that worked for me in 18.10 was to simply replace libnode.so with a newer version. For slack replace /usr/lib/slack/libnode.so with e.g libnode.so found in https://github.com/electron/electron/releases/download/v2.0.12/electron-v2.0.12-linux-x64.zip

Edit: Celebrated too soon. Slack becomes unstable and crashes and restarts every now and then with the updated lib.

@brpaz

This comment has been minimized.

brpaz commented Nov 4, 2018

Just updated my laptop to Ubuntu 18.10 and now almost all my Electron based apps crash.

This is a very serious issue and I suspect it affects hundreds of applications, many of them might not be actively maintained anymore.

I dont really understand, is this an issue of Electron or glibc?

Still, we need a solution. ;(

@eli-schwartz

This comment has been minimized.

eli-schwartz commented Nov 4, 2018

It's literally in the release notes for Electron 2.0.8, that this bug has been fixed (by #13988 to be precise).

It was an electron bug, and now it is a bug in any application that has not updated their vendored Electron copies. At this point, the Electron developers have done everything they could in order to fix it.

The only other thing they could do, is to redesign the entire Electron ecosystem to use global, system electron installations by default, which would allow this to be fixed once and applied everywhere. But this is rather out of scope of the issue.

@Diftraku

This comment has been minimized.

Diftraku commented Nov 7, 2018

For Slack users who replaced libnode.so from somewhere else (and got the app running again) but still get the occasional crash, disable your notification sound for now.

It seems if Slack tries to play the notification sound for a new message, it simply restarts instead of ever getting around to playing the clip.

jhit added a commit to jhit/Boostnote that referenced this issue Nov 12, 2018

Update electron framework to version 3.0.8
On Ubuntu 18.10 there is a issue with glibc versions below 2.28 see
[Prebuilt Electron binaries segfault at startup on Arch Linux with glibc 2.28 #13972](electron/electron#13972)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment