Running apps with GDB segfaults

[jonnius]

• Device: any (last tested at least FP2 and Xperia X)
• Channel: any (last tested at devel and nightly CI builds)
• Build: any (last tested on CI server build from 6th february)

Steps to reproduce

• Install any app (commands below are for Webber 0.5.1)
• Launch app at least once
• Make system image writable
• Install gdb and some debug packages:

sudo apt update
sudo apt install gdb libssl1.0.0-dbg

• Enter a shell (e.g. via adb shell)
• Run the app with gdb:

gdb -ex run --args /opt/click.ubuntu.com/webber.timsueberkrueb/0.5.1/lib/arm-linux-gnueabihf/bin/webber --desktop_file_hint=/home/phablet/.cache/ubuntu-app-launch/desktop/webber.timsueberkrueb_webber_0.5.1.desktop

Expected behavior

The app starts.

Actual behavior

There is a segfault before the app starts with

Program received signal SIGSEGV, Segmentation fault.
0xf77cadde in ?? () from /lib/ld-linux-armhf.so.3

Backtrace:

#0  0xf77cadde in ?? () from /lib/ld-linux-armhf.so.3
#1  0xf77cadf6 in ?? () from /lib/ld-linux-armhf.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

after installing libc6-dbg (and libssl1.0.0-dbg to get more informative output), there is just a (recoverable) SIGILL:

Program received signal SIGILL, Illegal instruction.
_armv7_tick () at armv4cpuid.S:94
94	armv4cpuid.S: No such file or directory.

Logfiles and additional information

• When starting the app the same way without gdb, it works fine:
  /opt/click.ubuntu.com/webber.timsueberkrueb/0.5.1/lib/arm-linux-gnueabihf/bin/webber --desktop_file_hint=/home/phablet/.cache/ubuntu-app-launch/desktop/webber.timsueberkrueb_webber_0.5.1.desktop
• Other devs confirmed that starting apps with GDB has been working before (may have been back in 2017 or 2018).
• This bug is the reason we cannot implement on-device-debugging in Clickable

[jonnius]

Just found out, if libc6-dbg is installed, one can continue after the segfault and the app starts. Without that package, it is impossible to continue.

[dobey]

It is not a segfault (SIGSEGV), but illegal instruction (SIGILL), which libssl handles internally and is expected. You need to tell gdb to ignore SIGILL, or just continue with the debug packages installed. There's nothing special to be done here.

[jonnius]

Well I missed some info when gathering the logs. But without libc6-dbg installed, it actually segfaults (SIGSEGV). Will upload the log later.

[jonnius]

@dobey I just updated the description. Could you please reopen?

[dobey]

@dobey I just updated the description. Could you please reopen?

What is actually crashing? Is it the app running under gdb, or is it gdb itself? Because this really doesn't make sense, and if it is an issue, it seems like a problem in ld and should be reported against Ubuntu. Can the same issue be reproduced on x86 for example when not installing libc6-dbg there? Does it happen with other commands, for example doing gdb -ex run --args /usr/bin/ls -lh instead?

Also, if the problem is fixed by installing libc6-dbg (libc6 is the package that provides the /lib/ld-linux.so.2, though I'm not sure what provides ld-linux-armhf.so instead), it will be quite difficult to debug, given one won't be able to get a stack trace with proper symbols, since the crash will be gone.

On the subject of gdb working prior to ubports/ubuntu-touch-meta#30 being merged, I think libc6-dbg was in the rootfs too, but we needed room and so removed it. I hope the plan is not for clickable to be installing debs and resizing the rootfs partition and such. That would be bad. Let's get a proper fix and gdbserver included in the rootfs by default,

But I guess either way, I think this is an issue that should be pushed upstream, as I don't think UBports should get sucked into maintaining forks of glibc and/or gcc for example, to fix issues like this.

[jonnius]

GDB also segfaults with other commands like ls. So should I open a bug report at gdb or gcc or glibc then?

I cannot reproduce the issue on my Ubuntu 19.10 machine. Should I set up a VM with 16.04 to test whether gdb crashes for any command? I doubt it.

I could live with having gdbserver and libc6-dbg installed in the system by default. If that is what you would recommend, I will open a MR accordingly.

[dobey]

No, I would not recommend having libc6-dbg installed in the rootfs by default. It is way too large. I think clickable or whatever IDE could instead download, push, and unpack it somewhere under the home directory (perhaps install debug packages inside a libertine container?) and tell gdbserver to find the symbols there instead of normal default expected paths.

As for where the bug should be reported, it seems like https://bugs.launchpad.net/ubuntu/+source/gdb/+bug/1325503 might already cover the issue here. The issue isn't exactly the same, but it is probably very directly related.

[jonnius]

Thanks for the clarification. If I understand that bug report correctly, it is expected behavior for gdb to fail without debug symbols being installed.

I just tried and tested to ship all shared object files from libc6-dbg with the click package of the app (only those with matching arch triplet of course). It does not change anything about the segfault when starting this app via gdb though. Do you know what those *.debug files are for in the package and whether I need them and if yes, where?

[dobey]

it is expected behavior for gdb to fail without debug symbols being installed.

I don't think that's true. It will run the app, and any stack traces will just be mostly useless.

I just tried and tested to ship all shared object files from libc6-dbg with the click package of the app (only those with matching arch triplet of course). It does not change anything about the segfault when starting this app via gdb though. Do you know what those *.debug files are for in the package and whether I need them and if yes, where?

You shouldn't actually distribute these in a click package. I don't know what those specific files are for, no. However, in order for gdb to find the symbols installed into a different path, you'll need to do set solib-search-path to an appropriate value, inside gdb before the run command.

[jonnius]

Thanks for your help. I finally made the Clickable feature that does not need to use apt on the device, nor distribute anything with the click package.