Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2495 - ucert rebuild within SDK segfaults on Debian 10 and Ubuntu 18.04 #7299

Open
openwrt-bot opened this issue Sep 15, 2019 · 12 comments
Open
Labels

Comments

@openwrt-bot
Copy link

@openwrt-bot openwrt-bot commented Sep 15, 2019

cshoredaniel:

Using the 19.07-SNAPSHOT SDK build Sept 12 (last successful build by the buildbots it looks like), and current openwrt-19.07 branch on the feeds, if doing signed build, the build fails due to ucert segfaulting (this is on Debian 10).

ldd staging_dir/host/bin/ucert shows:

    linux-vdso.so.1 (0x00007ffd03d63000)
    libubox.so => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/libubox.so (0x00007ff0db3b0000)
    libblobmsg_json.so => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/libblobmsg_json.so (0x00007ff0db3a8000)
    libjson-c.so.2 => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/libjson-c.so.2 (0x00007ff0db398000)
    libc.so.6 => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/libc.so.6 (0x00007ff0daff8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ff0db3c8000)

I'm wondering if the problem is due to using host instead SDK libraries.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Sep 16, 2019

ynezz:

build fails due to ucert segfaulting (this is on Debian 10).

Can you provide something reproducible, like failing commands including the dummy key files and/or stacktrace?

ldd staging_dir/host/bin/ucert shows

That's very good ldd in Debian10, here it shows:

ldd staging_dir/host/bin/ucert not a dynamic executable

file staging_dir/host/bin/ucert
ucert: Bourne-Again shell script, ASCII text executable

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Sep 16, 2019

cshoredaniel:

Aargh! Lost the my writeup due to 'wrongtoken' error from FS. Let's do this again:

daniel@buildserver:~/Build/sdk-19.07-ucert$ file staging_dir/host/bin/ucert staging_dir/host/bin/ucert: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=68de4d7aa97907c6d230b5a6ad9b87b55282417a, with debug_info, not stripped

To see the error in action:

  • Extract the ath79 19.07-SNAPSHOT SDK
  • cd to the sdk dir
  • ./scripts/feeds update && ./scripts/feeds install -a
  • make menuconfig
  • ensure CONFIG_SIGNED_PACKAGES=y
  • recommend CONFIG_BUILD_LOG
  • recommend CONFIG_ALL is unset
  • make -j9 (or whatever suits your build)
  • You'll get a mysterious, unhelpful error
  • make V=sc
rm -f /home/daniel/Build/sdk-19.07-ucert/build_dir/target-mips_24kc_musl/linux-ath79_generic/base-files/.configured_* rm -f /home/daniel/Build/sdk-19.07-ucert/staging_dir/target-mips_24kc_musl/stamp/.base-files_installed [ -s /home/daniel/Build/sdk-19.07-ucert/key-build -a -s /home/daniel/Build/sdk-19.07-ucert/key-build.pub ] || /home/daniel/Build/sdk-19.07-ucert/staging_dir/host/bin/usign -G -s /home/daniel/Build/sdk-19.07-ucert/key-build -p /home/daniel/Build/sdk-19.-c /home/daniel/Build/sdk-19.07-ucert/key-build.ucert -p /home/daniel/Build/sdk-19.07-ucert/key-build.pub -s /home/daniel/Build/sdk-19.07-ucert/key-build make[3]: *** [Makefile:221: /home/daniel/Build/sdk-19.07-ucert/build_dir/target-mips_24kc_musl/linux-ath79_generic/base-files/.configured_b19dd3aae2bcca394b7e6453144e9502_8e081b74cf069e1e6800a5bbcbb282f0] Segmentation fault make[3]: Leaving directory '/home/daniel/Build/sdk-19.07-ucert/feeds/base/package/base-files' time: package/feeds/base/base-files/compile#0.20#0.07#0.24

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Sep 16, 2019

cshoredaniel:

Note that on initial extraction your observation of a script holds true, but after compilation it no longer is.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Sep 17, 2019

cshoredaniel:

Also note that make V=sc package/index sometimes succeeds so this may not be completely deterministic.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Sep 17, 2019

ynezz:

I was able to reproduce it even with the snapshot SDK, so it's not related to just 19.07 SDK. The problem is the rebuild of ucert under SDK, which then uses dynamic linker from the host which probably doesn't work properly with the libc from the SDK (ABI version 2.6.32), host libc has ABI version 3.2.0.

==1425== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==1425== Bad permissions for mapped region at address 0x0 ==1425== at 0x0: ??? ==1425== by 0x40040DB: dl_main (rtld.c:2199) ==1425== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253) ==1425== by 0x4002117: _dl_start_final (rtld.c:415) ==1425== by 0x4002117: _dl_start (rtld.c:522) ==1425== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so) ==1425== Jump to the invalid address stated on the next line ==1425== at 0x1036: ??? ==1425== by 0x4F83AE4: ??? (in /build/openwrt-sdk-ath79-generic_gcc-7.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6) ==1425== by 0x4F0C866: time (in /build/openwrt-sdk-ath79-generic_gcc-7.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6) ==1425== by 0x400BAD7: elf_machine_lazy_rel (dl-machine.h:576) ==1425== by 0x400BAD7: elf_dynamic_do_Rela (do-rel.h:77) ==1425== by 0x400BAD7: _dl_relocate_object (dl-reloc.c:258) ==1425== by 0x40040DB: dl_main (rtld.c:2199) ==1425== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253) ==1425== by 0x4002117: _dl_start_final (rtld.c:415) ==1425== by 0x4002117: _dl_start (rtld.c:522) ==1425== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so) ==1425== Address 0x1036 is not stack'd, malloc'd or (recently) free'd

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Sep 18, 2019

jow-:

We should inhibit the building of host utilities in the SDK. The SDK is meant to provide a precompiled feature-fixed toolchain environment. Compilation of host utilities (not package host builds) or toolchain components within the SDK is undefined due to the use of a bundled and shipped libc.

I think the best course forward here would be to prevent enabling signed package lists on a non-ucert equipped SDK.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Jan 10, 2020

stangri:

On Ubuntu 18.04.3 and 19.07.0 I'm also getting a segmentation fault on ucert (while trying to build package base-files):

SDK_mvebu.19.07.0/staging_dir/host/bin/ucert -I -c SDK_mvebu.19.07.0/key-build.ucert -p SDK_mvebu.19.07.0/key-build.pub -s SDK_mvebu.19.07.0/key-build
Segmentation fault (core dumped)

jow -- any updates on this?

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Jan 10, 2020

ynezz:

any updates on this

Nope, someone has to fix it. Meanwhile disable signing of the packages //CONFIG_SIGNED_PACKAGES// under SDK or if you want signed packages, you've to build from scratch, not within SDK.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented May 24, 2021

recycler:

Same here for Ubuntu 20.04.2

Seems that downloaded ucert is relying on certain library versions.

ldd ./staging_dir/host/bin/ucert
linux-vdso.so.1 (0x00007ffd69dae000)
libubox.so => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libubox.so (0x00007f743222b000)
libblobmsg_json.so => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libblobmsg_json.so (0x00007f7432226000)
libc.so.6 => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6 (0x00007f7432065000)
libjson-c.so.5 => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libjson-c.so.5 (0x00007f7432052000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7432245000)

(gdb) run -I -c /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.ucert -p /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.pub -s /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build
Starting program: /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/bin/ucert -I -c /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.ucert -p /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.pub -s /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7e18133 in __libc_start_main () from /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6
(gdb) backtrace
#0 0x00007ffff7e18133 in __libc_start_main () from /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6
#1 0x0000555555556a7e in _start () at /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/build_dir/hostpkg/ucert-2020-05-24-00b921d8/ucert.c:625

A solution could be a static binary for download.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Sep 10, 2021

tmn505:

This will affect all distro versions which weren't used for building particular SDK. On Arch Linux it doesn't segfault and gives a little hint what's wrong:
/home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert: /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/libc.so.6: version GLIBC_2.33' not found (required by /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert) /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert: /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/libc.so.6: version GLIBC_2.33' not found (required by /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/libjson-c.so.5)
that's on ramips SDK.
I created a workaround (see attachment) partly following Jo's advice. The workaround copies install stamps when creating SDK. Then if SDK is used with package which has a host part, the host part build is omitted if install stamp is present.

@aloisklink
Copy link

@aloisklink aloisklink commented May 20, 2022

This problem is still happening on Ubuntu 20.04 when using the OpenWRT SDK v19.07.10.

Running make overwrites the working ucert binary from the SDK with a new one, which is broken and causes a segmentation fault when creating base-files.

(side-note, for me, it looks like fwtool, usign, and jshn are also compiled and overwrite the SDK bins, but only jshn and ucert have segmentation faults when running jshn --help)

Workaround

Unfortunately, it looks like @tmn505's workaround mentioned in #7299 (comment) was lost when migrating to GitHub Issues.

It's probably easier to just disable CONFIG_SIGNED_PACKAGES so you don't even need to use ucert, but if you do want to sign packages, my work around is:

# wait until OpenWRT build creates the broken ucert bin and stampfiles
make package/ucert/compile
# copies back the original working ucert bin from the SDK .tar.xz archive
tar xf "../<PATH_TO_YOUR_SDK__DOWNLOAD_HERE>.tar.xz" --strip-components 1 --directory "." --wildcards '*/staging_dir/host/bin/*'
# continues make
make

@tmn505
Copy link
Contributor

@tmn505 tmn505 commented May 20, 2022

This is the workaround: https://paste.debian.net/1241483
It needs to be applied before SDK is created, otherwise the stamps are not there and the workaround won't work. You could try to patch the existing SDK but You'll have copy/create stamps Yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants