New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prebuilt Electron binaries segfault at startup on Arch Linux with glibc 2.28 #13972

Open
K900 opened this Issue Aug 7, 2018 · 26 comments

Comments

Projects
None yet
9 participants
@K900

K900 commented Aug 7, 2018

  • Electron Version: 2.0.5, 2.0.6
  • Operating System (Platform and Version): Arch Linux latest
  • Last known working Electron version: ?

Expected Behavior
Electron runs correctly.

Actual behavior
Electron segfaults on start, before anything useful happens.

To Reproduce
Download the prebuilt Electron release (electron-2.0.x-linux-x64.zip); unzip it; run ./electron; observe the fireworks.

Additional Information
The issue seems to be caused by the glibc 2.28 upgrade on Arch - downgrading to glibc 2.27 makes Electron work. I'm not really sure what the issue itself is, as Electron crashes even before Breakpad kicks in, so no minidump is produced, and I don't have a machine powerful enough to produce a debug build in a reasonable amount of time (and it's likely that a debug build produced on my machine with glibc 2.28 will work fine anyway, being linked against the correct glibc). Distribution-provided binaries (the electron package in the repository) work fine, even though they were not rebuilt to match the glibc upgrade.

@welcome

This comment has been minimized.

Show comment
Hide comment
@welcome

welcome bot Aug 7, 2018

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.

To help make it easier for us to investigate your issue, please follow the contributing guidelines.

welcome bot commented Aug 7, 2018

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.

To help make it easier for us to investigate your issue, please follow the contributing guidelines.

@eli-schwartz

This comment has been minimized.

Show comment
Hide comment
@eli-schwartz

eli-schwartz Aug 7, 2018

The electron package from the Arch Linux community repository was actually built using glibc 2.27, nice try. ;) (glibc tries pretty hard to maintain ABI compatibility, you should never need to rebuild software in order to work with newer glibc -- the fact that this breaks indicates either a glibc bug or a something in the software stack that depends on private details of glibc when it shouldn't.)

$ bsdtar -xOf /var/cache/pacman/pkg/electron-2.0.6-1-x86_64.pkg.tar.xz .BUILDINFO| grep glibc
installed = glibc-2.27-3-x86_64

So, it's entirely likely that rebuilding it on Arch Linux with glibc 2.28 would have the same issues, and that the issue lies in the build process. For the record, our package is built using https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/electron and you can see various patches we apply in order to use system dependencies, in the same directory of the git repository as the PKGBUILD file.

eli-schwartz commented Aug 7, 2018

The electron package from the Arch Linux community repository was actually built using glibc 2.27, nice try. ;) (glibc tries pretty hard to maintain ABI compatibility, you should never need to rebuild software in order to work with newer glibc -- the fact that this breaks indicates either a glibc bug or a something in the software stack that depends on private details of glibc when it shouldn't.)

$ bsdtar -xOf /var/cache/pacman/pkg/electron-2.0.6-1-x86_64.pkg.tar.xz .BUILDINFO| grep glibc
installed = glibc-2.27-3-x86_64

So, it's entirely likely that rebuilding it on Arch Linux with glibc 2.28 would have the same issues, and that the issue lies in the build process. For the record, our package is built using https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/electron and you can see various patches we apply in order to use system dependencies, in the same directory of the git repository as the PKGBUILD file.

@eli-schwartz

This comment has been minimized.

Show comment
Hide comment
@eli-schwartz

eli-schwartz Aug 7, 2018

More context: https://bugs.archlinux.org/task/59550 (Arch Linux bugreport).

it would appear to be coming from libnode.so, and it's not just electron either -- sone R packages do the same.

eli-schwartz commented Aug 7, 2018

More context: https://bugs.archlinux.org/task/59550 (Arch Linux bugreport).

it would appear to be coming from libnode.so, and it's not just electron either -- sone R packages do the same.

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

So I've done some testing and posted the results on the VSCode issue linked: Microsoft/vscode#55934

TL;DR:

  • Electron + Arch patches + Arch sysroot = works fine
  • Electron + no Arch patches + Debian sysroot (default) = segfault
  • Electron + minimal patches + Arch sysroot = ? (wip)

K900 commented Aug 8, 2018

So I've done some testing and posted the results on the VSCode issue linked: Microsoft/vscode#55934

TL;DR:

  • Electron + Arch patches + Arch sysroot = works fine
  • Electron + no Arch patches + Debian sysroot (default) = segfault
  • Electron + minimal patches + Arch sysroot = ? (wip)
@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

Another update: did a clean build outside of the Arch packaging tools, but with most of the Arch patches applied, and it worked (which isn't really surprising). I'm not quite sure where the Debian sysroot images are coming from or how they are produced, but upgrading glibc in those might be an interesting thing to try.

K900 commented Aug 8, 2018

Another update: did a clean build outside of the Arch packaging tools, but with most of the Arch patches applied, and it worked (which isn't really surprising). I'm not quite sure where the Debian sysroot images are coming from or how they are produced, but upgrading glibc in those might be an interesting thing to try.

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

I might actually just try bisecting glibc and seeing when it started crashing.

K900 commented Aug 8, 2018

I might actually just try bisecting glibc and seeing when it started crashing.

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

I've bisected glibc to this commit. The versions before it run fine, the version including it cause a segfault. I'm not entirely sure what the reason is. Trying to build the latest glibc-2.28 tag now with it reverted.

Edit: disregard that, I messed up. Re-testing now...

K900 commented Aug 8, 2018

I've bisected glibc to this commit. The versions before it run fine, the version including it cause a segfault. I'm not entirely sure what the reason is. Trying to build the latest glibc-2.28 tag now with it reverted.

Edit: disregard that, I messed up. Re-testing now...

@eli-schwartz

This comment has been minimized.

Show comment
Hide comment
@eli-schwartz

eli-schwartz Aug 8, 2018

Per discussion with azanella on freenode's #glibc channel:

I did some debug to check what is happening and it seems that the PLT to nghttp2_session_callbacks_set_on_frame_recv_callback (for libnode.so initialization) is branching to a bogus value, it came from symbol resolution in fact
there is really something strange with libnode.so files, on both skypeforlinux and electron
it is calling nghttp2_session_callbacks_set_on_frame_recv_callback through PLT (and also internally on the library)
but it does not seem to provide the symbol itself

$ nm -D /usr/share/skypeforlinux/libnode.so | grep nghttp2_session_callbacks_set_on_frame_
00000000009dbaf0 A nghttp2_session_callbacks_set_on_frame_not_send_callback
0000000000d20020 A nghttp2_session_callbacks_set_on_frame_recv_callback
0000000001100230 T nghttp2_session_callbacks_set_on_frame_send_callback

while nghttp2_session_callbacks_set_on_frame_send_callback is pointing to text segment (and thus code), nghttp2_session_callbacks_set_on_frame_recv_callback is an absolute symbol
this does not seem right
we changed how to handle absolute symbol to fix BZ #19818
yeap, reverting e7feec374c635b6a29d65c39ae5e1855528fed39 seems to avoid this specific issue, but loader fails now with another one...

And the conclusion was that this should be brought up for discussion on libc-alpha.

eli-schwartz commented Aug 8, 2018

Per discussion with azanella on freenode's #glibc channel:

I did some debug to check what is happening and it seems that the PLT to nghttp2_session_callbacks_set_on_frame_recv_callback (for libnode.so initialization) is branching to a bogus value, it came from symbol resolution in fact
there is really something strange with libnode.so files, on both skypeforlinux and electron
it is calling nghttp2_session_callbacks_set_on_frame_recv_callback through PLT (and also internally on the library)
but it does not seem to provide the symbol itself

$ nm -D /usr/share/skypeforlinux/libnode.so | grep nghttp2_session_callbacks_set_on_frame_
00000000009dbaf0 A nghttp2_session_callbacks_set_on_frame_not_send_callback
0000000000d20020 A nghttp2_session_callbacks_set_on_frame_recv_callback
0000000001100230 T nghttp2_session_callbacks_set_on_frame_send_callback

while nghttp2_session_callbacks_set_on_frame_send_callback is pointing to text segment (and thus code), nghttp2_session_callbacks_set_on_frame_recv_callback is an absolute symbol
this does not seem right
we changed how to handle absolute symbol to fix BZ #19818
yeap, reverting e7feec374c635b6a29d65c39ae5e1855528fed39 seems to avoid this specific issue, but loader fails now with another one...

And the conclusion was that this should be brought up for discussion on libc-alpha.

@zatrazz

This comment has been minimized.

Show comment
Hide comment
@zatrazz

zatrazz Aug 8, 2018

Could you check by reverting e7feec374c635b6a29d65c39ae5e1855528fed39 ? I would to get some freedback before address it on libc-alpha.

zatrazz commented Aug 8, 2018

Could you check by reverting e7feec374c635b6a29d65c39ae5e1855528fed39 ? I would to get some freedback before address it on libc-alpha.

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

This is the commit that I traced it down to when bisecting too. I'll try just reverting it and see what happens.

K900 commented Aug 8, 2018

This is the commit that I traced it down to when bisecting too. I'll try just reverting it and see what happens.

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

Just reverting that commit (on top of the latest 2.28 tag) seems to fix things. Doesn't explain how the binary got so messed up though...

K900 commented Aug 8, 2018

Just reverting that commit (on top of the latest 2.28 tag) seems to fix things. Doesn't explain how the binary got so messed up though...

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

Also, repo Electron's libnode.so seems fine:

❯ nm -D /usr/lib/electron/libnode.so | grep nghttp2_session_callbacks_set_on_frame_
00000000010363c0 T nghttp2_session_callbacks_set_on_frame_not_send_callback
0000000000bbd990 T nghttp2_session_callbacks_set_on_frame_recv_callback
00000000011cf540 T nghttp2_session_callbacks_set_on_frame_send_callback

K900 commented Aug 8, 2018

Also, repo Electron's libnode.so seems fine:

❯ nm -D /usr/lib/electron/libnode.so | grep nghttp2_session_callbacks_set_on_frame_
00000000010363c0 T nghttp2_session_callbacks_set_on_frame_not_send_callback
0000000000bbd990 T nghttp2_session_callbacks_set_on_frame_recv_callback
00000000011cf540 T nghttp2_session_callbacks_set_on_frame_send_callback
@zatrazz

This comment has been minimized.

Show comment
Hide comment
@zatrazz

zatrazz Aug 8, 2018

Does Electron's libnode.so work correctly with glibc 2.28?

zatrazz commented Aug 8, 2018

Does Electron's libnode.so work correctly with glibc 2.28?

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

The one in the official binary releases from this repo does not; the one in the Arch package does, as it's built with a different toolchain. I'm actually trying to build upstream Electron with Arch clang right now, will see where this goes, as it seems like an Electron issue and not a glibc issue to me.

K900 commented Aug 8, 2018

The one in the official binary releases from this repo does not; the one in the Arch package does, as it's built with a different toolchain. I'm actually trying to build upstream Electron with Arch clang right now, will see where this goes, as it seems like an Electron issue and not a glibc issue to me.

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

So yeah, it's definitely a toolchain issue, more specifically a linker issue. Simply replacing vendor/llvm-build/Release+Asserts/bin/lld with a symlink to system lld (version 6.0.1) results in a functional build. I'm guessing it's a linker bug, so glibc is correct in barfing here. Now to figure out where the Debian sysroot comes from...

K900 commented Aug 8, 2018

So yeah, it's definitely a toolchain issue, more specifically a linker issue. Simply replacing vendor/llvm-build/Release+Asserts/bin/lld with a symlink to system lld (version 6.0.1) results in a functional build. I'm guessing it's a linker bug, so glibc is correct in barfing here. Now to figure out where the Debian sysroot comes from...

@zatrazz

This comment has been minimized.

Show comment
Hide comment
@zatrazz

zatrazz Aug 8, 2018

Right, I saw reports where libnode.so is also failing on other projects and it seems to be the same issue. I was trying to reproduce the build failure with GNU linkers, but without much success.

zatrazz commented Aug 8, 2018

Right, I saw reports where libnode.so is also failing on other projects and it seems to be the same issue. I was trying to reproduce the build failure with GNU linkers, but without much success.

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

It seems to be an LLD bug. I'm trying a build with a newer prebuilt clang (and thus lld) version now.

K900 commented Aug 8, 2018

It seems to be an LLD bug. I'm trying a build with a newer prebuilt clang (and thus lld) version now.

K900 added a commit to K900/electron that referenced this issue Aug 8, 2018

vendor: update clang/lld version to fix #13972
This seems like some sort of a weird linker bug, so just update the
toolchain a bit.
@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

Good news: the tests pass with an updated lld, and the resulting libnode.so isn't horribly bugged.

K900 commented Aug 8, 2018

Good news: the tests pass with an updated lld, and the resulting libnode.so isn't horribly bugged.

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 8, 2018

@pcc on #llvm IRC found the exact revision that fixed it: llvm-project/lld@1adba6d

K900 commented Aug 8, 2018

@pcc on #llvm IRC found the exact revision that fixed it: llvm-project/lld@1adba6d

@alberto2000

This comment has been minimized.

Show comment
Hide comment
@alberto2000

alberto2000 Aug 10, 2018

Temporary (unsafe) fix is to npm i -D electron@beta

alberto2000 commented Aug 10, 2018

Temporary (unsafe) fix is to npm i -D electron@beta

felixonmars-bot pushed a commit to felixonmars/archlinux-packages that referenced this issue Aug 10, 2018

bpiotrowski
2.28-2: backport fix for BZ#23497 and revert commit breaking propriet…
…ary electron apps

The problem is in fact in lld[1] used to build Electron but it's too serious
regression to fool around.

[1] electron/electron#13972 (comment)


git-svn-id: file:///srv/repos/svn-packages/svn@331347 eb2447ed-0c53-47e4-bac8-5bc4a241df78

pld-gitsync pushed a commit to pld-linux/glibc that referenced this issue Aug 11, 2018

@petergerten petergerten referenced this issue Aug 16, 2018

Closed

AppImage segmentation fault #539

1 of 3 tasks complete

jkleinsc added a commit that referenced this issue Aug 16, 2018

Merge pull request #13988 from K900/update-lld
fix: update clang/lld version to fix #13972
@faultylee

This comment has been minimized.

Show comment
Hide comment
@faultylee

faultylee Aug 17, 2018

fyi, the latest update for glibc-2.28-4 seems to have fixed this issue, at least for storage explorer

faultylee commented Aug 17, 2018

fyi, the latest update for glibc-2.28-4 seems to have fixed this issue, at least for storage explorer

@javmorin

This comment has been minimized.

Show comment
Hide comment
@javmorin

javmorin Aug 17, 2018

@faultylee Yes, but it's a temporary workaround expressly for this issue, not a permanent fix. The breaking change was identified and reverted in a number of distros, but will be restored once electron is using the updated version of LDD (See @K900's comments above)

See https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/glibc&id=f2aaaac68876f6959c62ea09dcdda5d441bf4ff7 for the comment from the Arch devs

javmorin commented Aug 17, 2018

@faultylee Yes, but it's a temporary workaround expressly for this issue, not a permanent fix. The breaking change was identified and reverted in a number of distros, but will be restored once electron is using the updated version of LDD (See @K900's comments above)

See https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/glibc&id=f2aaaac68876f6959c62ea09dcdda5d441bf4ff7 for the comment from the Arch devs

@K900

This comment has been minimized.

Show comment
Hide comment
@K900

K900 Aug 28, 2018

Should this be closed now?

K900 commented Aug 28, 2018

Should this be closed now?

K900 added a commit to K900/vscode that referenced this issue Aug 28, 2018

Electron 2.0.8
This fixes a crash on Linux systems running Glibc 2.28
(see electron/electron#13972)

@K900 K900 referenced this issue Aug 28, 2018

Closed

Electron 2.0.8 #57457

@mfonville

This comment has been minimized.

Show comment
Hide comment
@mfonville

mfonville Sep 6, 2018

I think it is best to keep this open, but rename it. Because I experience the same issue on Ubuntu Cosmic (pre-release), I filed a bug on launchpad with them: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1790966

mfonville commented Sep 6, 2018

I think it is best to keep this open, but rename it. Because I experience the same issue on Ubuntu Cosmic (pre-release), I filed a bug on launchpad with them: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1790966

@jkleinsc

This comment has been minimized.

Show comment
Hide comment
@jkleinsc

jkleinsc Sep 6, 2018

Contributor

@mfonville did you try Electron 2.0.8?

Contributor

jkleinsc commented Sep 6, 2018

@mfonville did you try Electron 2.0.8?

@mfonville

This comment has been minimized.

Show comment
Hide comment
@mfonville

mfonville Sep 6, 2018

I did not, since I use a pre-built version of Atom.

So I went the other way around, and compiled my own glibc with Arch Linux's patch for Ubuntu Cosmic, and that works well: https://launchpad.net/~maarten-fonville/+archive/ubuntu/ppa/

mfonville commented Sep 6, 2018

I did not, since I use a pre-built version of Atom.

So I went the other way around, and compiled my own glibc with Arch Linux's patch for Ubuntu Cosmic, and that works well: https://launchpad.net/~maarten-fonville/+archive/ubuntu/ppa/

@mfonville mfonville referenced this issue Sep 7, 2018

Closed

:arrow_up: :electron: electron@2.0.8 #17904

0 of 1 task complete

mfonville added a commit to mfonville/atom that referenced this issue Sep 10, 2018

Update to Electron 2.0.9
Updated Electron fixes segfault on glibc 2.28 (see electron/electron#13972 )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment