ast_coredumper: Increase reliability #446

gtjoseph · 2023-11-13T21:09:30Z

Instead of searching for the asterisk binary and the modules in the
filesystem, we now get their locations, along with libdir, from
the coredump itself...

For the binary, we can use gdb -c <coredump> ... "info proc exe".
gdb can print this even without having the executable and symbols.

Once we have the binary, we can get the location of the modules with
gdb ... "print ast_config_AST_MODULE_DIR

If there was no result then either it's not an asterisk coredump
or there were no symbols loaded. Either way, it's not usable.

For libdir, we now run "strings" on the note0 section of the
coredump (which has the shared library -> memory address xref) and
search for "libasteriskssl|libasteriskpj", then take the dirname.

Since we're now getting everything from the coredump, it has to be
correct as long as we're not crossing namespace boundaries like
running asterisk in a docker container but trying to run
ast_coredumper from the host using a shared file system (which you
shouldn't be doing).

There is still a case for using --asterisk-bin and/or --libdir: If
you've updated asterisk since the coredump was taken, the binary,
libraries and modules won't match the coredump which will render it
useless. If you can restore or rebuild the original files that
match the coredump and place them in a temporary directory, you can
use --asterisk-bin, --libdir, and a new --moddir option to point to
them and they'll be correctly captured in a tarball created
with --tarball-coredumps. If you also use --tarball-config, you can
use a new --etcdir option to point to what normally would be the
/etc/asterisk directory.

Also addressed various "shellcheck" issues.

Resolves: #445

gtjoseph · 2023-11-13T21:09:49Z

cherry-pick-to: 18
cherry-pick-to: 20
cherry-pick-to: 21

gtjoseph · 2023-11-16T15:20:12Z

Found a few things debian/ubuntu related. need to fix.

InterLinked1 · 2023-11-16T21:53:03Z

Found a few things debian/ubuntu related. need to fix.

This is with the current core dumper, not with your change, but somehow it seems to fail on Debian sometimes, even with the standard paths:

root@debian:/usr/src/asterisk-21.0.0# ls -la core
-rw------- 1 root root 180887552 Nov 16 16:49 core
root@debian:/usr/src/asterisk-21.0.0# /var/lib/asterisk/scripts/ast_coredumper core
No coredumps found
root@debian:/usr/src/asterisk-21.0.0# ls -la core
-rw------- 1 root root 180887552 Nov 16 16:49 core

gtjoseph · 2023-11-28T19:14:10Z

@InterLinked1 Try this version and see if it resolves your issues in various environments.

contrib/scripts/ast_coredumper

seanbright · 2023-11-29T14:02:12Z

contrib/scripts/ast_coredumper

@@ -411,7 +458,7 @@ find_pid() {
 	# Now that we have the pids, let's get the command and
 	# its args. We'll add them to an array indexed by pid.
 	declare -a candidates
-	while read LINE ; do
+	while read -r LINE ; do


Could potentially avoid the regex with:

while read -r pid prog args ; do

I did it this way so if for some reason the result didn't match the regex, we'd skip that line and continue otherwise we won't be sure that we got a valid line.

InterLinked1 · 2023-11-29T17:50:26Z

@InterLinked1 Try this version and see if it resolves your issues in various environments.

~~Still no bueno:~~

root@debian:/usr/src/asterisk-21.0.0# ls -la core
-rw------- 1 root root 226508800 Nov 29 12:49 core
root@debian:/usr/src/asterisk-21.0.0# /var/lib/asterisk/scripts/ast_coredumper core
No coredumps found

Ignore me, I forgot to make install to move the scripts.

Works much better now:

root@debian:/usr/src/asterisk-21.0.0# /var/lib/asterisk/scripts/ast_coredumper core
readlink: missing operand
Try 'readlink --help' for more information.
Examining core
    Does appear to be an asterisk coredump
    Coredump indicates executable 'asterisk'
    Searching for asterisk module directory
    Found asterisk module directory '/usr/lib/asterisk/modules'
Processing /usr/src/asterisk-21.0.0/core
    ASTBIN: asterisk
    MODDIR: /usr/lib/asterisk/modules
    ETCDIR: /etc/asterisk
    LIBDIR: /usr/lib
    Renaming /usr/src/asterisk-21.0.0/core to /usr/src/asterisk-21.0.0/core-asterisk-2023-11-29T17-49-18Z
    Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-thread1.txt
    Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-brief.txt
    Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-full.txt
    Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-locks.txt
    Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-info.txt

Instead of searching for the asterisk binary and the modules in the filesystem, we now get their locations, along with libdir, from the coredump itself... For the binary, we can use `gdb -c <coredump> ... "info proc exe"`. gdb can print this even without having the executable and symbols. Once we have the binary, we can get the location of the modules with `gdb ... "print ast_config_AST_MODULE_DIR` If there was no result then either it's not an asterisk coredump or there were no symbols loaded. Either way, it's not usable. For libdir, we now run "strings" on the note0 section of the coredump (which has the shared library -> memory address xref) and search for "libasteriskssl|libasteriskpj", then take the dirname. Since we're now getting everything from the coredump, it has to be correct as long as we're not crossing namespace boundaries like running asterisk in a docker container but trying to run ast_coredumper from the host using a shared file system (which you shouldn't be doing). There is still a case for using --asterisk-bin and/or --libdir: If you've updated asterisk since the coredump was taken, the binary, libraries and modules won't match the coredump which will render it useless. If you can restore or rebuild the original files that match the coredump and place them in a temporary directory, you can use --asterisk-bin, --libdir, and a new --moddir option to point to them and they'll be correctly captured in a tarball created with --tarball-coredumps. If you also use --tarball-config, you can use a new --etcdir option to point to what normally would be the /etc/asterisk directory. Also addressed many "shellcheck" findings. Resolves: asterisk#445

gtjoseph · 2023-11-29T18:36:04Z

root@debian:/usr/src/asterisk-21.0.0# /var/lib/asterisk/scripts/ast_coredumper core
readlink: missing operand
Try 'readlink --help' for more information.
Examining core
Does appear to be an asterisk coredump
Coredump indicates executable 'asterisk'
Searching for asterisk module directory
Found asterisk module directory '/usr/lib/asterisk/modules'
Processing /usr/src/asterisk-21.0.0/core
ASTBIN: asterisk
MODDIR: /usr/lib/asterisk/modules
ETCDIR: /etc/asterisk
LIBDIR: /usr/lib
Renaming /usr/src/asterisk-21.0.0/core to /usr/src/asterisk-21.0.0/core-asterisk-2023-11-29T17-49-18Z
Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-thread1.txt
Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-brief.txt
Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-full.txt
Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-locks.txt
Creating /tmp/core-asterisk-2023-11-29T17-49-18Z-info.txt

The readlink: missing operand should have been suppressed since you specified a core file on the command line. Just pushed up a fix for that.

jcolp · 2023-12-04T17:17:36Z

@InterLinked1 Is this now in a working state for you?

InterLinked1 · 2023-12-04T17:33:30Z

@InterLinked1 Is this now in a working state for you?

Sorry - yup, it is, and that other spurious warning is gone now, too.
I'm not sure if there was some recent regression but ast_coredumper was previously not working on any of my machines recently (it always had mostly in the past). With this PR though all is well again.

InterLinked1 · 2023-12-06T16:51:39Z

One more thing I just noticed now:

root@ess:/usr/src/asterisk-21.0.0# /var/lib/asterisk/scripts/ast_coredumper --RUNNING
Found a single asterisk instance running as process 848952
Dumping running asterisk process to /tmp/core-asterisk-running-2023-12-06T16-39-34Z
Dump is complete.
Couldn't get module directory from coredump!

However, it seems the backtrace was successfully extracted, so whatever happened didn't impede it from working.

github-actions · 2023-12-06T21:24:32Z

Successfully merged to branch master and cherry-picked to ["18","20","21"]

github-actions bot added the pr-submit-testing-in-progress label Nov 13, 2023

github-actions bot added pr-submit-tests-failed and removed pr-submit-testing-in-progress labels Nov 13, 2023

gtjoseph marked this pull request as draft November 16, 2023 15:19

gtjoseph force-pushed the master-coredumper-fix branch from 2a6b512 to dbd39a8 Compare November 28, 2023 19:13

github-actions bot added pr-submit-testing-in-progress pr-submit-tests-failed and removed pr-submit-tests-failed pr-submit-testing-in-progress labels Nov 28, 2023

seanbright reviewed Nov 29, 2023

View reviewed changes

contrib/scripts/ast_coredumper Outdated Show resolved Hide resolved

seanbright reviewed Nov 29, 2023

View reviewed changes

contrib/scripts/ast_coredumper Outdated Show resolved Hide resolved

seanbright reviewed Nov 29, 2023

View reviewed changes

gtjoseph force-pushed the master-coredumper-fix branch from dbd39a8 to b6cc3a2 Compare November 29, 2023 17:41

github-actions bot added pr-submit-testing-in-progress and removed pr-submit-tests-failed labels Nov 29, 2023

github-actions bot added pr-submit-tests-failed and removed pr-submit-testing-in-progress labels Nov 29, 2023

gtjoseph force-pushed the master-coredumper-fix branch from b6cc3a2 to bebfc2e Compare November 29, 2023 18:35

github-actions bot added pr-submit-testing-in-progress and removed pr-submit-tests-failed labels Nov 29, 2023

github-actions bot added pr-submit-tests-failed and removed pr-submit-testing-in-progress labels Nov 29, 2023

gtjoseph marked this pull request as ready for review November 30, 2023 16:39

InterLinked1 approved these changes Dec 4, 2023

View reviewed changes

jcolp approved these changes Dec 5, 2023

View reviewed changes

gtjoseph added the cherry-pick-test Trigger dry run of cherry-picks label Dec 6, 2023

gtjoseph added the merge-approved label Dec 6, 2023

github-actions bot added pre-merge-testing-in-progress pre-merge-checks-passed merge-in-progress and removed merge-approved pre-merge-testing-in-progress labels Dec 6, 2023

asterisk-org-access-app bot merged commit 1a0acab into asterisk:master Dec 6, 2023
55 of 62 checks passed

github-actions bot removed the merge-in-progress label Dec 6, 2023

gtjoseph deleted the master-coredumper-fix branch March 20, 2024 12:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ast_coredumper: Increase reliability #446

ast_coredumper: Increase reliability #446

gtjoseph commented Nov 13, 2023 •

edited

gtjoseph commented Nov 13, 2023

gtjoseph commented Nov 16, 2023

InterLinked1 commented Nov 16, 2023

gtjoseph commented Nov 28, 2023

seanbright Nov 29, 2023

gtjoseph Nov 29, 2023

InterLinked1 commented Nov 29, 2023 •

edited

gtjoseph commented Nov 29, 2023

jcolp commented Dec 4, 2023

InterLinked1 commented Dec 4, 2023

InterLinked1 commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

ast_coredumper: Increase reliability #446

ast_coredumper: Increase reliability #446

Conversation

gtjoseph commented Nov 13, 2023 • edited

gtjoseph commented Nov 13, 2023

gtjoseph commented Nov 16, 2023

InterLinked1 commented Nov 16, 2023

gtjoseph commented Nov 28, 2023

seanbright Nov 29, 2023

Choose a reason for hiding this comment

gtjoseph Nov 29, 2023

Choose a reason for hiding this comment

InterLinked1 commented Nov 29, 2023 • edited

gtjoseph commented Nov 29, 2023

jcolp commented Dec 4, 2023

InterLinked1 commented Dec 4, 2023

InterLinked1 commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

gtjoseph commented Nov 13, 2023 •

edited

InterLinked1 commented Nov 29, 2023 •

edited