must-gather: Add kernel level logs

[rajatcing]

Signed-off-by: RAJAT SINGH rajasing@redhat.com
Fixes #840

[rajatcing]

@jarrpa I have some doubts that need some clarification.

[rajatcing]

Well, collection from paths /var/log works. But the script keeps looping fo one of the pod.

sent 138 bytes  received 160035 bytes  24642.00 bytes/sec
total size is 159588  speedup is 1.00
collecting finished for node ip-10-0-131-8.ec2.internal
collecting kernel logs  from node ip-10-0-244-172.ec2.internal
Starting pod/ip-10-0-244-172ec2internal-debug ...
To use host binaries, run `chroot /host`
waiting for the debug pod to be in ready state
waiting for the debug pod to be in ready state
waiting for the debug pod to be in ready state
waiting for the debug pod to be in ready state
waiting for the debug pod to be in ready state
waiting for the debug pod to be in ready state

cc @crombus

[rajatcing]

/hold

[crombus]

Thanks for the PR.

[crombus]

based on comments on the bug. Create func for kernel level logs which will include whole of the /val/log folder using rsync and journalctl -o verbose cmd. Call that func inside the crash_collection() before rsync cmd and make sure to start a different process. This will work fine for now.

[rajatcing]

That makes sense but again, there's some issue that I'm hitting that you might need to take a look at.

[rajatcing]

Added some code to take a look for you 👍

[rajatcing]

@crombus While I see what you're going for with requesting the functions be moved to the cluster-scoped file, I think it makes sense to leave them in the Ceph file. Though they operate on Nodes, which are cluster-scoped, they are being run for the purpose of collecting system-level information to troubleshoot Ceph problems specifically. And we certainly want to avoid looping through nodes more times than is strictly necessary.

Right Jose, that is somewhat I was thinking, It does not make sense to start debug pods twice, once for journal collection and then, once again for ceph related collection. We can keep it as it is for now and will start making changes in the next PRs as I briefed above.

[crombus]

@rajatsing I tested in my local I had some strange issues

[must-gather-5mnnl] POD rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1650) [Receiver=3.1.2]
[must-gather-5mnnl] POD rsync: [Receiver] write error: Broken pipe (32)
[must-gather-5mnnl] POD error: exit status 23

[rajatcing]

@rajatsing I tested in my local I had some strange issues

[must-gather-5mnnl] POD rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1650) [Receiver=3.1.2]
[must-gather-5mnnl] POD rsync: [Receiver] write error: Broken pipe (32)
[must-gather-5mnnl] POD error: exit status 23

I believe that these are coming because the path does not exist

[must-gather-h7hzm] POD rsync: link_stat "/host/var/log/sysstat" failed: No such file or directory (2)
[must-gather-h7hzm] POD rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1659) [Receiver=3.1.3]
[must-gather-h7hzm] POD rsync: [Receiver] write error: Broken pipe (32)

[rajatcing]

I don't understand why CI fails when it passes locally make ocs-operator-ci. Are there any issues with the CI, it also does not give me some proper error info to work on.

error: some steps failed:
  * could not run steps: step ocs-operator-ci failed: test "ocs-operator-ci" failed: the pod ci-op-rxlt4gb8/ocs-operator-ci was deleted without completing after 30s (failed containers: )
time="2020-12-10T10:17:07Z" level=info msg="Reporting job state 'failed' with reason 'executing_graph:step_failed:running_pod'"

[crombus]

overall look good. just one request I think you can merge the two functions journal_collection and kernel_collections

[crombus]

perfect.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: crombus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• must-gather/OWNERS [crombus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-merge-robot]

@rajatsing: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/red-hat-storage-ocs-ci-e2e-aws	69903ed	link	/test red-hat-storage-ocs-ci-e2e-aws

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[crombus]

/hold until the decision is made on this.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-ci-robot]

@rajatsing: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.