☂️ Re-write etcd-backup-restore as etcd-steward, with EtcdMember updation capabilities

[shreyas-s-rao]

How to categorize this issue?

/area quality robustness usability open-source
/kind enhancement cleanup api-change task

What would you like to be added:
I would like etcd-backup-restore to be refactored, ie, re-written, with components that can be individually turned on/off and provide well-defined interfaces and channels that can be easily consumed by other components within and outside the system. I would like this new replacement of etcd-backup-restore to be lean, modular, well-tested and extensible. I would like the etcd bootstrapping process to be simplified and more maintainable. I would the memory footprint for etcd DB restoration to be reduced. I would like garbage collection of snapshots to be simple and easy to understand.

Why is this needed:
The issues mentioned above cannot be solved within the old repository, sue to various reasons such as testability, backward compatibility concerns, component-wise changes to the code, which require changes in every other component anyway. The only way to roll out a new version of etcd-backup-restore is to fully re-write its components from scratch, possibly re-using existing code snippets from the old code. The name "etcd-backup-restore" is no longer relevant since the code does much more than just backups and restorations - it takes care of maintenance of the etcd DB, along with updation of k8s resources such as member leases and EtcdMember resources that are used by druid for computing etcd cluster status and performing other actions such as compaction or remediations, as specified in the EtcdMember proposal. Accordingly, the name "etcd-backup-restore" needs to be evolved into something that correctly depicts what the code does, which is to manage or look after the etcd, like a steward.

Task List:

• Basic project structure
• Components (enabled/disabled by individual config flags on default command; metrics and metering registered per component; additional endpoints registered per component as required)
  • [Feature] Always-on HTTP server with default endpoints #2
  • [Feature] Observe etcd cluster leadership changes #3
  • [Feature] Implement etcd lock mechanism #4
  • [Feature] Coordinate etcd defragmentation amongst multiple steward instances #5
  • [Feature] Handle etcd alarms #6
  • [Feature] Implement lean and well-defined snapstore for object storage providers #7
  • [Feature] Implement snapshot compression/decompression #8
  • [Feature] Implement snapshotting #9
  • [Feature] Implement better garbage collection of snapshots #10
  • [Feature] Implement failure-tolerant etcd restoration #11
  • [Feature] Implement snapshot compaction #12
  • [Feature] Implement etcd data validation #13
  • [Feature] Implement simple bootstrapping of etcd #14
  • [Feature] Implement configurable updation of EtcdMember resource #15
  • [Feature] Update etcd member lease #16
• CLI commands
  • Default (no subcommand)
  • Compact subcommand
  • Copy-Backups subcommand
• Testing
  • Unit tests for all individual functions/methods (small functions, quick tests)
  • Integration tests - working of a component within isolation, using an embedded etcd as necessary
  • e2e tests - working of all components together, along with etcd-wrapper
  • Load/performance tests - to be run on every PR, to avoid regression in performance of steward components
• Create make targets
• Set up CI/CD pipelines, jobs for tests, builds
• Documentation
  • User - getting started, usage
  • Operator - getting started, deployment, operations
  • Developer - getting started, testing, adding new functionality
  • Concepts - design decisions
• Evaluate etcd v3.5.x - [Upgrade] Move the etcd from v3.4.26 to v3.4.32 to v3.5.x etcd-druid#445