Add experimental gather and must-gather commands

[tnozicka]

Description of your changes:
This PR introduces 2 new commands for the scylla-operator binary that help collecting debug dumps.

must-gather collects a predefined set of resources in the Kubernetes cluster that relate to scylla-operator APIs or the cluster state itself. It also has a flag (--all-resources) that can collect every namespaced and non-namespaced resource that is "listable" which is useful in case the default dump wouldn't be enough to identify the issue or for our CI.

gather is aimed at collecting individual resources (+ related objects by default) and helps to get extra data in case there would need to ask for an extra resource to be collected.

All commands redact Secret's data except for well known Kubernetes keys ("ca.crt", "tls.crt", "service-ca.crt") that hold public parts of certificates.

Which issue is resolved by this Pull Request:
Resolves #1365 #1394

Also shaves off about 35 minutes from our e2e suites!

/cc

[scylla-operator-bot]

@tnozicka: GitHub didn't allow me to request PR reviews from the following users: tnozicka.

Note that only scylladb members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Description of your changes:
This PR introduces 2 new commands for the scylla-operator binary that help collecting debug dumps.

must-gather collects a predefined set of resources in the Kubernetes cluster that relate to scylla-operator APIs or the cluster state itself. It also has a flag (--all-resources) that can collect every namesapced and non-namespaced resource that is "listable" which is useful in case the default dump wouldn't be enough to identify the issue or for our CI.

gather is aimed at collecting individual resources (+ related objects by default) and helps to get extra data in case there would need to ask for an extra resource to be collected.

All commands redact Secret's data except for well known Kubernetes keys ("ca.crt", "tls.crt", "service-ca.crt") that hold public parts of certificates.

Which issue is resolved by this Pull Request:
Resolves #1365

/cc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tnozicka]

/test images

[tnozicka]

/retest
(infra)

[rzetelskik]

/approve
lgtm, waiting for other reviewers

[tnozicka]

/hold
(for final manual check of gathered resources)

[zimnx]

/lgtm

[scylla-operator-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rzetelskik, tnozicka, zimnx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tnozicka,zimnx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tnozicka]

must-gather and namespace dump in https://gcsweb.scylla-operator.scylladb.com/gcs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1368/pull-scylla-operator-e2e-gke-parallel/1706672007518621696/ look ok
/hold cancel

[mykaul]

Is QA using it now in their tests? @rayakurl ?

[rayakurl]

Is QA using it now in their tests? @rayakurl ?

We are not using this in cloud (we are using the cloud CLI so perhaps it will be wrapped in CLI and we will use if needed), I think that this question should be addressed to @fruch and @vponomaryov , who are testing operator directly

[fruch]

Is QA using it now in their tests? @rayakurl ?

We are not using this in cloud (we are using the cloud CLI so perhaps it will be wrapped in CLI and we will use if needed), I think that this question should be addressed to @fruch and @vponomaryov , who are testing operator directly

@mykaul we aren't using those commands, I don't think we ever run into a situation we needed those.

We have a much more serve issue, that we don't have the instructions on how to be able to catch scylla coredump on operator setup...

[vponomaryov]

Is QA using it now in their tests? @rayakurl ?

We are not using this in cloud (we are using the cloud CLI so perhaps it will be wrapped in CLI and we will use if needed), I think that this question should be addressed to @fruch and @vponomaryov , who are testing operator directly

@mykaul we aren't using those commands, I don't think we ever run into a situation we needed those.

We have a much more serve issue, that we don't have the instructions on how to be able to catch scylla coredump on operator setup...

I totally agree with @fruch
Being able to get any coredump file is also needed to solve user's problems.

[fruch]

@mykaul we aren't using those commands, I don't think we ever run into a situation we needed those.

I take some of it back, we are not using those commands, since they are new ( I thought this is some fix)
and we have our own code that collects all the information we need (maybe this is even modeled base on that test code ??)

we wrote a task down, I'm not sure when we'll get to add it to the tests.

[tnozicka]

and we have our own code that collects all the information we need (maybe this is even modeled base on that test code ??)

yep, from what I recall QA collects artifacts based on our old bash script so that part should definitely migrate over, but they may wait a few days to get the docs (#1367) for it to have an easier transition

core dumps are kind of an off topic, but I've made #1436 to track it

[fruch]

and we have our own code that collects all the information we need (maybe this is even modeled base on that test code ??)

yep, from what I recall QA collects artifacts based on our old bash script so that part should definitely migrate over, but they may wait a few days to get the docs (#1367) for it to have an easier transition

it's not a bash script, but python, but it collects all the k8s resources information it can get.

core dumps are kind of an off topic, but I've made #1436 to track it

great, we should be able to explain users on how to setup it up for coredumps to be collected