Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PESDLC-995 Create means to search logs in cloud pods #17250

Merged
merged 6 commits into from
Apr 3, 2024
Merged

PESDLC-995 Create means to search logs in cloud pods #17250

merged 6 commits into from
Apr 3, 2024

Conversation

savex
Copy link
Contributor

@savex savex commented Mar 21, 2024

To eliminate code duplication and offload some of the methods to external class, new util classes is introduced with abstract approach to reuse checks for both local run and cloud

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x

Release Notes

  • none

@savex savex force-pushed the PESDLC-995-cloud-log-checking branch from fe02da2 to 2747c4d Compare March 21, 2024 22:53
@rpdevmp
Copy link
Contributor

rpdevmp commented Mar 22, 2024

LGTM

@savex savex force-pushed the PESDLC-995-cloud-log-checking branch from a6267cf to 48d25e0 Compare March 22, 2024 18:50
@savex savex force-pushed the PESDLC-995-cloud-log-checking branch from 48d25e0 to ad7ad79 Compare March 22, 2024 18:57
@savex savex requested a review from rpdevmp March 22, 2024 18:58
@savex savex marked this pull request as ready for review March 22, 2024 18:59
@savex
Copy link
Contributor Author

savex commented Mar 22, 2024

/ci-repeat 1

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 22, 2024 •
edited

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46633#018e67e1-00f2-4051-8a54-aa4ceeb20562

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46825#018e7c7b-1137-44a2-ad30-9d2ce35fed14

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46825#018e7ca3-6d4d-4980-a8b8-14e5ae6f17bb

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46825#018e7ca3-6d50-4351-8236-2aa5a24d6d86

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46841#018e7d7d-1a5f-4699-b0c8-4e5007d70905

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46841#018e7d7d-1a5c-4e11-bc4b-d3c287008c71

rpdevmp
rpdevmp reviewed Mar 26, 2024
View reviewed changes
tests/rptest/services/utils.py
super().__init__(test_context, allow_list, logger)
self.targetpath = targetpath

def _capture_log(self, node, expr) -> Generator[str, None, None]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It assumes that kubectl command would work. We might want to add an Exception in case it failed to capture logs for pod

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Will do.

@savex
PESDLC-995 handle possible logs getter exceptions
a7dfbec
@savex
PESDLC-995 Introduce KubeNodeShell class
dbffb35 
     Class uses nsenter tool to switch namespaces when running
     priviledged on specific kubernetes nodes. Can be used to
     access node for given pod.
@savex
PESDLC-995 Updated utils class to use KubeNodeShell
06ba7f6 
     Approach is searches for logs on the kube node itself. And
     grabs the ones that is corresponds to RP pod name.
@vbotbuildovich
Copy link
Collaborator

new failures in https://buildkite.com/redpanda/redpanda/builds/46841#018e7d7d-1a5c-4e11-bc4b-d3c287008c71:

"rptest.tests.internal_topic_protection_test.InternalTopicProtectionLargeClusterTest.test_schemas_topic"

@savex savex marked this pull request as draft March 27, 2024 13:24
@rpdevmp
Copy link
Contributor

rpdevmp commented Mar 27, 2024

LGTM so far

@savex savex marked this pull request as ready for review April 2, 2024 18:19
@savex
Copy link
Contributor Author

savex commented Apr 2, 2024

Investigation shows that Containerd stored logs only in /var/log/pods. Which is being searched by this PR

@savex
Copy link
Contributor Author

savex commented Apr 2, 2024

/ci-repeat 1

@savex savex requested a review from rpdevmp April 2, 2024 18:25
rpdevmp
rpdevmp approved these changes Apr 2, 2024
View reviewed changes
Copy link
Contributor

@rpdevmp rpdevmp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The only thing is we are reading the whole log file into memory with cat + grep, but that shouldn't be the issue

@savex
Copy link
Contributor Author

savex commented Apr 2, 2024

LGTM. The only thing is we are reading the whole log file into memory with cat + grep, but that shouldn't be the issue

Actual command is running without TTY on the node itself and the cat is actually streaming the data to stdout instead of loading the whole thing. So this will not affect RP that resides on the same node because we are making it in-between tests.

@savex savex merged commit 8f45f96 into dev Apr 3, 2024
16 checks passed
@savex savex deleted the PESDLC-995-cloud-log-checking branch April 3, 2024 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rpdevmp rpdevmp rpdevmp approved these changes

@travisdowns travisdowns Awaiting requested review from travisdowns

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@savex @rpdevmp @vbotbuildovich