[WIP] Documentation for developers: Debugging spk packages

[m4tt075]

Motivation

I'm maintaining a Tvheadend-testing version of the current master branch (4.3.x) on my spksrc fork in order to follow and support the development. Experiencing crashes and dead-locks I started looking into ways how to debug self-cross-compiled spk packages on my Synology NAS. I have not been able to find any reasonable documentation on how to do this and spend hours learning the hard way. I would like to share my findings with those who might run into similar issues in the future. Having said that, my way of approaching this problem might be far from ideal. If you are aware of better ways / approaches, please comment on this issue so that we can optimize the documentation.

Objective

1. Document how spk packages can be debugged.
2. Once this is fine-tuned enough, integrate it into the Wiki.

[WIP]Documentation

Debugging

Cross-compiling with debug symbols (on your VM or docker spksrc environment)

First, you have to ensure that your cross-compiled package contains debug symbols. This is usually not the case, as the spksrc framework routinely strips (i.e. removes) those symbols from packaged libraries and binaries. There are two ways to achieve that:

1. You (temporarily) remove the PLIST files from your spk and all available cross packages you would like to debug or...
2. You (temporarily) edit the spksrc/mk/spksrc.strip.mk file and comment out the do \ [...] done block under strip_target, which is easier if you have multiple dependencies and cross packages.

Once you have done this, just build your package as usual via make arch-[ARCH]-[toolchain version], e.g. make arch-evansport-6.1. The package will be substantially bigger as usual as it contains all the debug symbols. Do not forget to revert the changes above once you are done with debugging.

Debugging on XPEnology / your Synology NAS

It is recommended to test and debug your packages in a safe environment and not on a production system. You can run XPEnology in a VM to provide such an environment. Debugging on your NAS should be last resort if you cannot reproduce the problems otherwise.

Once your package is installed, ssh into your XPEnology / Synology NAS as admin user. Then...

1. sudo su and enter the admin / root password again (the password is identical)
2. The command synogear install will install diagnosis tools on your system, including the gdb debugger, which is what we are looking for.

Note: You will have to repeat 1. and 2. for each ssh debugging session as the diagnosis environment is not maintained after exiting your ssh session.

If you just need to debug the main thread, you are done here and can start debugging. However, frequently, you want to be able to debug multiple threads. If so, continue from here...

3. You need to be able to access the toolchains, you have cross-compiled your package with, on your XPEnology / NAS. For the example above, you would download the evansport-6.1 toolchains from the official Synology sources and untar them into some accessible directory, e.g. /volume1/public/toolchains on your XPEnology / NAS.
4. Then, still as root user, go into your home directory (cd ~) and create a file .gdbinit. Include the following lines (adapting directories as needed):
   set sysroot /volume1/public/toolchains/i686-pc-linux-gnu/i686-pc-linux-gnu/sys-root
add-auto-load-safe-path /volume1/public/toolchains/i686-pc-linux-gnu/i686-pc-linux-gnu/sys-root/lib/libthread_db-1.0.so
set solib-search-path /volume1/@appstore/tvheadend-testing/lib
   The last line should contain the path to the libraries of the application you have packaged and are trying to debug.

With this, you are finally ready to go: Start your application manually within gdb or start it via the package center and hook the debugger on via gdb [application] [PID]. You will be able to create back traces and core dumps as usual.

[KaraokeStu]

Hi,

This looks like some fantastic information, although it would be better suited on the WIKI

[m4tt075]

Thanks, @KaraokeStu. Yes, I'm planning to submit this to the Wiki (see Objective 2 above), but am looking for reviewers and possible improvements before I do this. I was hoping to reach more people here.
Not sure, @ymartin59 or @GuillaumeSmaha maybe: Any thoughts or improvements to the above?

[ymartin59]

Really interesting. My naive question is: why not running your application on a standard workstation Linux and debug it there ? By the way, there may exist situation where bug may only appear on DSM.

I have only one NAS, it is my "productive" one, so I am really reluctant to install any compilation/debug related stuff on it. To test packages, I use virtual machines - xpenology for DSM 5.2 and a DSM 6.0 x86_64 VM, this add some comfort: no risk for my production, discard changes and redo from scratch (or revert to snapshot)

I propose you to give a try to "remote debugging" with gdbserver and the comfort of C/C++ IDE running on your laptop, that way no need to copy toolchains on DSM, and no gdb commands to learn:

• http://www.thegeekstuff.com/2014/04/gdbserver-example/
• https://sourceware.org/gdb/onlinedocs/gdb/Server.html

[m4tt075]

Thanks, @ymartin59. This is exactly the kind of feedback I was hoping for. What you are writing makes total sense. I just ran into all of this because I cross-compile an application that is developed by somebody else. I experienced dead-locks and the developer asked me for backtraces, but I have not been continuously developing and debugging myself.

As to your questions: I don't have a Linux workstation, just the NAS and a Windows laptop with an Ubuntu VM, that I'm using to cross-compile Synology packages via the spksrc framework. The idea to run Xpenology VMs to test cross-compiled packages has never even occured to me, but is great and definitely easier and safer than the route I have been taking. To that extent, debugging on the NAS itself should probably just be a fall-back option for boarderline cases where bugs / crashes / dead-locks cannot be reproduced in an Xpenology environment. I will adapt the documentation accordingly.

gdbserver looks very interesting indeed, as it would not even require debug symbols to be available on the NAS / Xpenology VM, but would definitely have to test it before I should even think about documenting anything in that regard...

Again, many thanks @ymartin59. Very, very helpful.