-
Notifications
You must be signed in to change notification settings - Fork 332
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Applications may crash. If a crash happens on a remote system during CI run it's sometimes very hard to understand the reason. The most important means to analyze such is a stack trace. It's also very important to check whether there was a core dump or not, even a test passed. For Docker environment, the core dumps are collected by the host's systemd-coredump, which knows nothing about such containers (for now). To build an informative thread stack trace debuginfo packages should be installed. But they can't be installed on the host OS (ubuntu), That's why after all the tests completed an additional container should be up and the host's core dumps and host's journal should be passed into it. Even if there weren't enough debuginfo packages at CI-runtime, the core dump could be analyzed locally later. Fixes: https://pagure.io/freeipa/issue/8251 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com> Reviewed-By: Rob Crittenden <rcritten@redhat.com>
- Loading branch information
1 parent
aa5a333
commit d1b53de
Showing
5 changed files
with
150 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
#!/bin/bash -eu | ||
|
||
IPA_TESTS_ENV_WORKING_DIR="${IPA_TESTS_REPO_PATH}/ipa_envs" | ||
COREDUMPS_DIR="${IPA_TESTS_ENV_WORKING_DIR}/${COREDUMPS_SUBDIR}" | ||
|
||
since_time="$(cat '/coredumpctl.time.mark' || echo '-1h')" | ||
debugger="/debugger.sh" | ||
|
||
cat > "$debugger" <<EOF | ||
#!/bin/bash -eux | ||
debug_info="\$@" | ||
gdb \ | ||
-ex 'set confirm off' \ | ||
-ex 'set pagination off' \ | ||
-ex 'thread apply all bt full' \ | ||
-ex 'quit' \ | ||
\$debug_info > "\${CORE_PID}.stacktrace" 2>&1 | ||
EOF | ||
chmod +x "$debugger" | ||
|
||
# make sure coredumpctl installed | ||
which coredumpctl | ||
coredumpctl \ | ||
--no-pager --directory="$HOST_JOURNAL" --since="$since_time" list ||: | ||
|
||
rm -rvf "$COREDUMPS_DIR" ||: | ||
mkdir "$COREDUMPS_DIR" | ||
cd "$COREDUMPS_DIR" | ||
|
||
pids="$(coredumpctl --no-pager --directory="$HOST_JOURNAL" --since="$since_time" -F COREDUMP_PID || echo '')" | ||
for pid in $pids; do | ||
# core dump | ||
{ coredumpctl \ | ||
--no-pager \ | ||
--since="$since_time" \ | ||
--directory="$HOST_JOURNAL" \ | ||
-o "${pid}.core" dump "$pid" && \ | ||
tar -czf "${pid}.core.tar.gz" --remove-files "${pid}.core" ; } ||: | ||
|
||
# stacktrace | ||
{ CORE_PID="$pid" \ | ||
coredumpctl \ | ||
--no-pager \ | ||
--since="$since_time" \ | ||
--directory="$HOST_JOURNAL" \ | ||
--debugger="$debugger" \ | ||
debug "$pid" && \ | ||
tar \ | ||
-czf "${pid}.stacktrace.tar.gz" \ | ||
--remove-files "${pid}.stacktrace" ; } ||: | ||
done | ||
|
||
chmod a+rw -R "$COREDUMPS_DIR" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
#!/bin/bash -eu | ||
|
||
function install_debuginfo() { | ||
dnf makecache ||: | ||
dnf install -y \ | ||
${IPA_TESTS_REPO_PATH}/dist/rpms_debuginfo/*.rpm \ | ||
gdb | ||
|
||
dnf debuginfo-install -y \ | ||
389-ds-base \ | ||
bind \ | ||
bind-dyndb-ldap \ | ||
certmonger \ | ||
gssproxy \ | ||
httpd \ | ||
krb5-server \ | ||
krb5-workstation \ | ||
samba \ | ||
sssd | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
#!/bin/bash -eu | ||
|
||
function install_debuginfo() { :; } | ||
|
||
# override install_debuginfo for the platform specifics | ||
source "${IPA_TESTS_SCRIPTS}/install-debuginfo-${IPA_PLATFORM}.sh" | ||
|
||
install_debuginfo |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters