Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable stacktrace support in gitian builds #3006

Merged
merged 22 commits into from
Jul 2, 2019

Conversation

codablock
Copy link

This PR enables stacktrace support in the release builds done by Gitian.

It expects debug information to be present in dashd.debug (linux) and dashd.exe.dbg (windows). These debug files are split/stripped as part of the Gitian build already. If debug information is not present, a serialized crash info is printed which can then be used later to extract the stacktrace. This allows easier debugging and support when users report unexpected crashes. They can now provide us with the serialized crash info and we are able to extract the stacktraces even when the user did not run with debug info. We might consider releasing debug files as well in the future.

With the debug file present, it looks like:

2019-06-26 20:18:37 Windows Exception: EXCEPTION_INT_DIVIDE_BY_ZERO
   0#: (0x007BD0B4) wallet.cpp:5147 - CWallet::InitAutoBackup()
   1#: (0x0041F929) init.cpp:1558   - AppInitMain(boost::thread_group&, CScheduler&)

Without the debug file present, it looks like:

2019-06-26 20:19:41 No debug information available for stacktrace. You should download debug information from Github and then run:
dashd.exe -printcrashinfo=bvcgc43iinzgc43ijfxgm3ybaaewiyltnbsc4zlymuxvo2lomrxxo4zaiv4ggzlqoruw63r2ebcvqq2fkbkest2ol5eu4vc7irevmskeivpuewk7ljcvetycwtihwaaaaaaaakpzieaaaaaaaaaa====

This can then be used to extract the stacktrace with the same executable but with the debug file present:

$ wine ./dashcore-0.14.1/bin/dashd.exe -printcrashinfo=bvcgc43iinzgc43ijfxgm3ybaaewiyltnbsc4zlymuxvo2lomrxxo4zaiv4ggzlqoruw63r2ebcvqq2fkbkest2ol5eu4vc7irevmskeivpuewk7ljcvetycwtihwaaaaaaaakpzieaaaaaaaaaa====
Windows Exception: EXCEPTION_INT_DIVIDE_BY_ZERO
   0#: (0x007BD0B4) wallet.cpp:5147 - CWallet::InitAutoBackup()
   1#: (0x0041F929) init.cpp:1558   - AppInitMain(boost::thread_group&, CScheduler&)

As you can see, this was invoked with wine. Serialized stacktraces can be taken from native Windows and then be extracted with wine, and vice versa.

@codablock codablock added this to the 14.1 milestone Jun 26, 2019
@codablock
Copy link
Author

Latest commits changes how stacktrace support is compiled. From now on, basic stacktrace and crash info support is compiled in unconditionally on all platforms. The only thing that is conditionally now is the hooking into exception/assert handling.

@codablock codablock force-pushed the pr_gitian_stacktraces branch 2 times, most recently from 4390979 to c131da3 Compare June 27, 2019 09:26
@codablock
Copy link
Author

Also added experimental support for MacOS, but as always, I'm unable to verify if it actually works. My hope here is that by manually loading the .debug file instead of the actual binary when initializing libbacktrace, we're able to get debug info to work. This worked fine for Windows builds, so lets hope it works here as well.

@UdjinM6 maybe you find some time to test this again? I can also provide you with pre-built binaries from a Gitian run. As an alternative, you could do your own Gitian build (./dash/contrib/gitian-build.py -u https://github.com/codablock/dash -c -n -Vdocker -j6 -b -om <signer> pr_gitian_stacktraces). It should also be possible to generate the .debug files in a local build by running make -C src osx_debug. I added a temporary commit that allows to provoke a crash by adding -crash=1 to dashd/dash-qt.

When debug info has not been generated yet and dashd is run with -crash=1, it should give no stacktrace but a -printcrashinfo line. When debug info has been generated, it should print the stacktrace. At that time, the -printcrashinfo option should also start to work.

@codablock codablock changed the title Enable stacktrace support in linux/win gitian builds Enable stacktrace support in gitian builds Jun 27, 2019
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See below + should add *.debug into .gitignore.

In general, stacktraces still don't work for me on mac - No debug information available for stacktrace when providing -printcrashinfo=<smth> (stackframeInfos is empty even after GetStackFrameInfos).

EDIT: this ^^^ is for a local build via make -C src osx_debug btw.
EDIT2: doesn't matter actually, same results for the gitian build.

src/dash-cli.cpp Outdated Show resolved Hide resolved
src/dash-cli.cpp Outdated Show resolved Hide resolved
src/stacktraces.cpp Outdated Show resolved Hide resolved
configure.ac Outdated Show resolved Hide resolved
src/Makefile.am Outdated Show resolved Hide resolved
@codablock
Copy link
Author

Rebased on develop, handled review comments, and also added printing of crash description in the case where no debug info is present.

@UdjinM6 did you also try the case where the .debug files were already present when the crash happened?

@UdjinM6
Copy link

UdjinM6 commented Jun 28, 2019

Yep. I also tried to specify the exact path to dash-qt.debug (thought that maybe it fails due to /./ in the path) - doesn't help either.

@codablock
Copy link
Author

Hmm ok, looks like I have no other choice but to actually look into the dSYM bundle structure and make generating that thing part of the build process...

@codablock
Copy link
Author

codablock commented Jun 28, 2019

After some discussion in Slack and some more testing done by @UdjinM6, we figured out that using a dSYM bundle works. I've added some commits so that the osx_debug make target creates a dSYM bundle instead.

Also, turned out that we have to always include at least some debug information for this to work instead of making it dependent on --enable-crash-hooks

@codablock codablock force-pushed the pr_gitian_stacktraces branch 3 times, most recently from aa10c2d to 0259927 Compare June 28, 2019 10:42
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job! 👍

Works for me on mac in both cases now (w and w/out debug files).

ACK after f89c24b is removed.

Instead of making it dependent on "--enable-crash-hooks". We will need the
debug info every time now, even in release builds.
@codablock
Copy link
Author

Removed f89c24b

Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 👍

@UdjinM6 UdjinM6 merged commit 780bffe into dashpay:develop Jul 2, 2019
Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, utACK

nmarley added a commit to nmarley/dash that referenced this pull request Sep 16, 2019
nmarley added a commit to nmarley/dash that referenced this pull request Sep 16, 2019
barrystyle pushed a commit to PACGlobalOfficial/PAC that referenced this pull request Jan 22, 2020
* Remove use of -rdynamic

This causes check-symbols to fail horribly and also turned out to be not
required when using libbacktrace. It was only required when using
"backtrace()" from "<execinfo.h>"

* Remove spurious ], from configure.ac

* Add -DENABLE_STACKTRACES=1 to CMakeLists.txt

* Remove unused method my_backtrace_simple_callback

* Use fs::path().filename() instead of basename()

* Add static g_exeFileName and g_exeFileBaseName

* Use .exe.dbg file when available

* Use uint64_t instead of uintptr_t

* Implement GetBaseAddress() for unix and win32

* Implement unified crash_info and use it everywhere before printing crash info

* Print a serialized version of crash_info when there is no debug info

* Implement "-printcrashinfo" command line option

* Compile stacktrace support unconditionally and only make crash hooks conditional

This also renames the --enable-stacktraces option to --enable-crash-hooks

* Enable crash hooks in win/linux Gitian builds

* Try to load .debug file on MacOS and enable crash hooks for osx Gitian builds

* Check for dsymutil and if it needs --flat

* Create .debug files in osx Gitian build

* Handle review comments

* Also print crash description when no stacktrace is available

* Unconditionally add -g1 debug information

Instead of making it dependent on "--enable-crash-hooks". We will need the
debug info every time now, even in release builds.

* Put MacOS debug info into dSYM symbols instead of plain .debug files

* Implement MacOS specific GetBaseAddress
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants