ARROW-7119: [C++][CI] Show automatic backtraces

[kszucs]

Automatically generate coredumps and backtraces for the failing C++ tests on both Linux (including docker) and macOS.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7119

[kszucs]

Example coredump https://github.com/apache/arrow/pull/6202/checks?check_run_id=399397059

[kou]

Example coredump https://github.com/apache/arrow/pull/6202/checks?check_run_id=399397059

It seems that the coredump isn't related.
Should we remove ~/Library/Logs/DiagnosticReports/*.core before running our scripts?

[kszucs]

That wasn't a segfault, no core file was genereted. Let me force a segfault.

[kou]

https://github.com/apache/arrow/pull/6202/checks?check_run_id=401063744

.crash exists but no core dump.
We may need to run ulimit -c unlimited -S in the Test step.
How about running ulimit -a in the Test step to debug ulimit values?

[kszucs]

Auto backtraces are working on linux within the docker containers, it also works in the GHA jobs.

I've tested it locally on macOS and it also works when the coredump gets generated, but the coredumps are not generated in the GHA job, despite that it is theoretically enabled.

@kou any ideas what am I missing on macOS?

[kszucs]

The coredumps should be available now for macOS as well https://github.com/apache/arrow/runs/408685826

[kszucs]

@kou we should reuse the print_coredumps function for the rest of the C++ dependent builds too, but in a follow-up PR.

[kou]

+1

I hope that this can work without changing kernel.core_pattern (core is the default) on Linux in a follow-up task!

[kszucs]

We need to change the core pattern because the tests are usually run in parallel, so we need to identify the right coredump for the test case.

This PR also print the coredumps for linux, not just macOS.

[kou]

We need to change the core pattern because the tests are usually run in parallel, so we need to identify the right coredump for the test case.

OK.

This PR also print the coredumps for linux, not just macOS.

Oh, sorry. I misunderstood.

[fsaintjacques]

It seems to me that run-test.sh is overly complicated. Most of the things it do is dealing with xml for jenkins, which we don't use. I'd simplify to do the following:

• setup sanitizers
• execute tests (and capture pid + return code), that might require some bash magic with jobs and waitpid.
• link coredump with pid instead of basename pattern.
• print coredump

Do we use the local stdout/stderr dumped text files @kou @wesm?

[kszucs]

• execute tests (and capture pid + return code), that might require some bash magic with jobs and waitpid.
• link coredump with pid instead of basename pattern.

The chance of the collision vs. required bash magic tradeoff made me choose the base name.

[kou]

Do we use the local stdout/stderr dumped text files

AFAIK, no.
We can simplify run-test.sh but I think that it's out-of-scope of this pull request. We can do it as a separated pull request.

[kszucs]

@fsaintjacques created a follow-up JIRA as suggested by @kou https://issues.apache.org/jira/browse/ARROW-7814

I think it is ready for review again.

[kou]

I confirmed that https://github.com/apache/arrow/pull/6202/checks?check_run_id=436229364 shows backtrace.
I'll merge this after I remove needless sudo.

[kou]

+1