must-gather: add more info for ceph crash #818

crombus · 2020-10-12T11:43:39Z

add command to check for ceph crash info
for every crash and collect core dump of
rook crash.

Signed-off-by: crombus pkundra@redhat.com

ashishkumsingh · 2020-10-12T12:03:34Z

must-gather/collection-scripts/gather_ceph_resources

@@ -159,4 +162,15 @@ for ns in $namespaces; do
        { timeout 120 oc debug nodes/"${node}" -- bash -c "test -f /host/var/lib/rook/log/${ns}/ceph-volume.log && cat /host/var/lib/rook/log/${ns}/ceph-volume.log" > "${NODE_OUTPUT_DIR}"/ceph-volume.log; } >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1
    done
    oc delete -f pod_helper.yaml
+
+    # Collecting ceph prepare volume logs


The comment needs to be changed to capturing crash dumps.

leseb · 2020-10-12T13:47:38Z

must-gather/collection-scripts/gather_ceph_resources

@@ -159,4 +162,15 @@ for ns in $namespaces; do
        { timeout 120 oc debug nodes/"${node}" -- bash -c "test -f /host/var/lib/rook/log/${ns}/ceph-volume.log && cat /host/var/lib/rook/log/${ns}/ceph-volume.log" > "${NODE_OUTPUT_DIR}"/ceph-volume.log; } >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1
    done
    oc delete -f pod_helper.yaml
+
+    # Collecting ceph crash dump
+    for node in $(oc get nodes -l cluster.ocs.openshift.io/openshift-storage='' --no-headers | grep -w 'Ready' | awk '{print $1}'); do


Suggested change

for node in $(oc get nodes -l cluster.ocs.openshift.io/openshift-storage='' --no-headers | grep -w 'Ready' | awk '{print $1}'); do

for node in $(oc get nodes -l cluster.ocs.openshift.io/openshift-storage='' --no-headers | awk '/Ready/ {print $1}'); do

leseb · 2020-10-12T13:48:04Z

must-gather/collection-scripts/gather_ceph_resources

+        printf "collecting crash logs  from node %s \n"  "${node}" | tee -a  "${BASE_COLLECTION_PATH}"/gather-debug.log
+        CRASH_OUTPUT_DIR=${CEPH_COLLECTION_PATH}/crash_${node}
+        mkdir -p "${CRASH_OUTPUT_DIR}"
+        oc debug nodes/"${node}"  --to-namespace="${ns}" -- bash -c "sleep 5m" &  >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1


Suggested change

oc debug nodes/"${node}" --to-namespace="${ns}" -- bash -c "sleep 5m" & >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1

oc debug nodes/"${node}" --to-namespace="${ns}" -- bash -c "sleep 5m" & >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1

leseb · 2020-10-12T13:48:40Z

must-gather/collection-scripts/gather_ceph_resources

+        mkdir -p "${CRASH_OUTPUT_DIR}"
+        oc debug nodes/"${node}"  --to-namespace="${ns}" -- bash -c "sleep 5m" &  >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1
+        sleep 60
+        oc rsync -n "${ns}" `oc get pods -n "${ns}"| grep debug| awk '{print $1}'`:/host/var/lib/rook/openshift-storage/crash/ "${CRASH_OUTPUT_DIR}"


Suggested change

oc rsync -n "${ns}" `oc get pods -n "${ns}"| grep debug| awk '{print $1}'`:/host/var/lib/rook/openshift-storage/crash/ "${CRASH_OUTPUT_DIR}"

oc rsync -n "${ns}" $(oc get pods -n "${ns}"|awk '/debug/ {print $1}'):/host/var/lib/rook/openshift-storage/crash/ "${CRASH_OUTPUT_DIR}"

leseb · 2020-10-12T13:49:56Z

must-gather/collection-scripts/gather_ceph_resources

@@ -136,6 +136,9 @@ for ns in $namespaces; do
        for i in $(timeout 120 oc -n "${ns}" exec "${HOSTNAME}"-helper -- bash -c "ceph osd lspools --connect-timeout=15"|awk '{print $2}'); do
            { timeout 120 oc -n "${ns}" exec "${HOSTNAME}"-helper -- bash -c "rbd ls -p $i" >> "${COMMAND_OUTPUT_DIR}/pools_rbd_$i"; } >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1;
        done
+        for i in $(timeout 120 oc -n "${ns}" exec "${HOSTNAME}"-helper -- bash -c "ceph crash ls --connect-timeout=15"|awk '{print $1}'); do
+            { timeout 120 oc -n "${ns}" exec "${HOSTNAME}"-helper -- bash -c "ceph crash info $i" >> "${COMMAND_OUTPUT_DIR}/pools_rbd_$i"; } >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1;


don't you need --connect-timeout=15 here as well?

leseb · 2020-10-13T10:47:30Z

must-gather/collection-scripts/gather_ceph_resources

+        mkdir -p "${CRASH_OUTPUT_DIR}"
+        oc debug nodes/"${node}" --to-namespace="${ns}" -- bash -c "sleep 5m" &  >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1
+        sleep 60
+        oc rsync -n "${ns}" `oc get pods -n "${ns}"| awk '/debug/{print $1}'`:/host/var/lib/rook/openshift-storage/crash/ "${CRASH_OUTPUT_DIR}"


backticks are deprecated use $() instead

openshift-bot · 2020-10-13T15:28:50Z

/retest

Please review the full test history for this PR and help us cut down flakes.

jarrpa · 2020-10-13T15:38:50Z

must-gather/collection-scripts/gather_ceph_resources

@@ -136,6 +136,7 @@ for ns in $namespaces; do
        for i in $(timeout 120 oc -n "${ns}" exec "${HOSTNAME}"-helper -- bash -c "ceph osd lspools --connect-timeout=15"|awk '{print $2}'); do
            { timeout 120 oc -n "${ns}" exec "${HOSTNAME}"-helper -- bash -c "rbd ls -p $i" >> "${COMMAND_OUTPUT_DIR}/pools_rbd_$i"; } >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1;
        done
+<<<<<<< HEAD


Remove this.

add command to check for ceph crash info for every crash and collect core dump of rook crash. Signed-off-by: crombus <pkundra@redhat.com>

crombus · 2020-10-13T18:40:33Z

/retest

agarwal-mudit

lgtm

openshift-ci-robot · 2020-10-14T06:29:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: agarwal-mudit, jarrpa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~must-gather/OWNERS~~ [agarwal-mudit,jarrpa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

agarwal-mudit · 2020-10-14T06:29:46Z

/retest

openshift-ci-robot · 2020-10-14T07:50:02Z

@crombus: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/red-hat-storage-ocs-ci-e2e-aws	`66aabd7`	link	`/test red-hat-storage-ocs-ci-e2e-aws`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2020-10-14T07:55:14Z

/retest

Please review the full test history for this PR and help us cut down flakes.

crombus · 2020-10-14T08:59:00Z

/cherry-pick release-4.6

openshift-cherrypick-robot · 2020-10-14T08:59:07Z

@crombus: new pull request created: #830

In response to this:

/cherry-pick release-4.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

crombus requested a review from jarrpa October 12, 2020 11:43

openshift-ci-robot requested review from obnoxxx and umangachapagain October 12, 2020 11:43

ashishkumsingh reviewed Oct 12, 2020

View reviewed changes

crombus force-pushed the crash branch from 4572b14 to a2cfe98 Compare October 12, 2020 13:06

leseb reviewed Oct 12, 2020

View reviewed changes

crombus force-pushed the crash branch from a2cfe98 to bcd9391 Compare October 13, 2020 10:38

crombus requested a review from leseb October 13, 2020 10:39

leseb reviewed Oct 13, 2020

View reviewed changes

crombus force-pushed the crash branch from bcd9391 to a1d3d7d Compare October 13, 2020 11:05

crombus requested a review from leseb October 13, 2020 11:16

crombus force-pushed the crash branch from a1d3d7d to 15fc9d8 Compare October 13, 2020 12:46

jarrpa approved these changes Oct 13, 2020

View reviewed changes

openshift-ci-robot assigned jarrpa Oct 13, 2020

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 13, 2020

crombus force-pushed the crash branch from 15fc9d8 to 706529d Compare October 13, 2020 15:31

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Oct 13, 2020

jarrpa reviewed Oct 13, 2020

View reviewed changes

crombus force-pushed the crash branch 7 times, most recently from 061008f to 53dc021 Compare October 13, 2020 16:03

must-gather: add more info for ceph crash

66aabd7

add command to check for ceph crash info for every crash and collect core dump of rook crash. Signed-off-by: crombus <pkundra@redhat.com>

crombus force-pushed the crash branch from 53dc021 to 66aabd7 Compare October 13, 2020 16:05

jarrpa approved these changes Oct 13, 2020

View reviewed changes

agarwal-mudit approved these changes Oct 14, 2020

View reviewed changes

openshift-ci-robot assigned agarwal-mudit Oct 14, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 14, 2020

openshift-merge-robot merged commit 966b77e into red-hat-storage:master Oct 14, 2020

openshift-cherrypick-robot mentioned this pull request Oct 14, 2020

bug 1869411: [release-4.6] must-gather: add more info for ceph crash #830

Merged

crombus deleted the crash branch October 19, 2020 04:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

must-gather: add more info for ceph crash #818

must-gather: add more info for ceph crash #818

crombus commented Oct 12, 2020

ashishkumsingh Oct 12, 2020

crombus Oct 12, 2020

leseb Oct 12, 2020

crombus Oct 13, 2020

leseb Oct 12, 2020

crombus Oct 13, 2020

leseb Oct 12, 2020

crombus Oct 13, 2020

leseb Oct 12, 2020

crombus Oct 13, 2020

leseb Oct 13, 2020

crombus Oct 13, 2020

openshift-bot commented Oct 13, 2020

jarrpa Oct 13, 2020

crombus Oct 13, 2020

crombus commented Oct 13, 2020

agarwal-mudit left a comment

openshift-ci-robot commented Oct 14, 2020

agarwal-mudit commented Oct 14, 2020

openshift-ci-robot commented Oct 14, 2020

openshift-bot commented Oct 14, 2020

crombus commented Oct 14, 2020

openshift-cherrypick-robot commented Oct 14, 2020

	for node in $(oc get nodes -l cluster.ocs.openshift.io/openshift-storage='' --no-headers \| grep -w 'Ready' \| awk '{print $1}'); do
	for node in $(oc get nodes -l cluster.ocs.openshift.io/openshift-storage='' --no-headers \| awk '/Ready/ {print $1}'); do

	oc debug nodes/"${node}" --to-namespace="${ns}" -- bash -c "sleep 5m" & >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1
	oc debug nodes/"${node}" --to-namespace="${ns}" -- bash -c "sleep 5m" & >> "${BASE_COLLECTION_PATH}"/gather-debug.log 2>&1

	oc rsync -n "${ns}" `oc get pods -n "${ns}"\| grep debug\| awk '{print $1}'`:/host/var/lib/rook/openshift-storage/crash/ "${CRASH_OUTPUT_DIR}"
	oc rsync -n "${ns}" $(oc get pods -n "${ns}"\|awk '/debug/ {print $1}'):/host/var/lib/rook/openshift-storage/crash/ "${CRASH_OUTPUT_DIR}"

must-gather: add more info for ceph crash #818

must-gather: add more info for ceph crash #818

Conversation

crombus commented Oct 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-bot commented Oct 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crombus commented Oct 13, 2020

agarwal-mudit left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Oct 14, 2020

agarwal-mudit commented Oct 14, 2020

openshift-ci-robot commented Oct 14, 2020

openshift-bot commented Oct 14, 2020

crombus commented Oct 14, 2020

openshift-cherrypick-robot commented Oct 14, 2020