Implement a comprehensive diagnosis tool

[ahrtr]

What would you like to be added?

Currently we have several scattered etcd diagnosis tools,

• etcd-diagnosis
• etcd-dump-logs
• etcd-dump-db
• bbolt

I think it would be better to deliver a comprehensive tool that integrates all of these capabilities. It should support both online and offline diagnosis:

• online diagnosis connects to a running etcd cluster and gathers diagnostic data.
• offline diagnosis, it analyzes etcd data directly when the etcd instance isn’t running.

A couple of use cases (stories)

• When an user raises an issue, we can request them to use the diagnosis tool to generate a report to gather all required info something like etcd_diagnosis_report.json to avoid long back and forth communication.
• When the etcd cluster is completely down, and users need to recover the cluster from one of the members using flag --force-new-cluster, they need to figure out the best member to restore the cluster from. In such case, they will need to use the diagnosis tool to figure out which member has the latest data.
• When the cluster runs out db space quota, and it's already down. Users will need to figure out which resources consume most of the space. They can use the diagnosis tool.
  • Refer to https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/
• For advanced users, they may want to make some offline analysis and gain deeper insights, they can use the diagnosis tool

We can add the comprehensive tool under https://github.com/etcd-io/etcd/tree/main/tools. Once it's done, we can deprecate the existing etcd-dump-logs and etcd-dump-db

cc @fuweid @ivanvc @jmhbnz @serathius

Why is this needed?

To improve users diagnosis experience.

[ahrtr]

In the first step, we will just integrate the scattered tools into one comprehensive tool etcd-diagnosis. Roughly it will have the following sub commands,

• scrape
  get all required info from a running etcd cluster, it's similar to etcd-diagnosis
• iterate-bucket
• list-bucket
• scan-keys
• hash
• log: similar to etcd-dump-logs
• bbolt: we can just integrate the bbolt commands. We need to finish the following two tasks first,
  • Migrate all commands to cobra style commands bbolt#472
  • seperate cli and cmd main package bbolt#564

In the next step, we can implement some advanced & useful feature, it's open to discussion. See one example below,

• https://github.com/ahrtr/etcd-diagnosis?tab=readme-ov-file#example-2-offline-analysis

[jmhbnz]

I feel quite strongly that for user experience reasons before we go and create yet another binary we should make better use of both etcdctl and etcdutl. These binaries are part of our release artifacts and shipped with our released container images so our users already have them available.

Reviewing the use cases:

* scrape
  get all required info from a running etcd cluster, it's similar to [etcd-diagnosis](https://github.com/ahrtr/etcd-diagnosis)

Cluster is running, so --> etcdctl

* iterate-bucket
* list-bucket
* scan-keys
* hash

These appear to be in relation to etcd data files, so --> etcdutl.

* log: similar to etcd-dump-logs

etcd-dump-logs works on etcd data files, so --> etcdutl.

* bbolt: we can just integrate the bbolt commands. 

bbolt commands work on offline data files, so --> etcdutl.

In summary: Let's make life easier for our users by providing this important functionality within our existing released cli utility binaries, rather than further complicate the landscape and expect our users to download and compile tools.

[ahrtr]

For the etcdutl related changes might be OK.

For the etcdctl related change, as mentioned in the community meeting, it's higher level command, which is actually a combination of several etcdctl commands. If others think it's OK, then we can go for this direction.

[ahrtr]

Based on the feedback from yesterday's community, the main comment is about enhancing the existing etcdctl and etcdutl instead of developing a new tool. I am OK to go for that direction.

Please anyone feel free to feedback if you have any other comments.

I'd like to start to implement it starting from next week. Also call for contributors. The tasks are planned in #20217 (comment)

[fuweid]

online diagnosis connects to a running etcd cluster and gathers diagnostic data.
offline diagnosis, it analyzes etcd data directly when the etcd instance isn’t running.

Based on this, I recommend enhancing the existing tool to support both online and offline diagnostic modes, rather than introducing a separate tool. This would help reduce maintenance overhead, promote consistency in diagnostics, and provide a unified user experience.

Anyway, for this tool, +1. Thanks @ahrtr for this sharing this.