[Draft] Reuse events used for syncing watchers

[serathius]

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

This reduced memory allocation of sync loop, when there are a lot of streams.

For #17562 with command go run ./tools/benchmark/main.go watch-latency --watch-per-stream 1000 --streams 1 --put-total 500 --val-size 100000 this reduced memory usage:

Name	etcd memory [GB]	etcd cpu [core]	90%ile watch event latency[s]	99%ile watch event latency[s]	90%ile put latency[s]	99%ile put latency[s]
base	11.5	1.07	24.4	25.9	0.002	0.011
reuse events	0.48672	1.13	24.4	25.9	0.003	0.05

Higher cpu is expected due to faster event processing.

TODO:

• Limit event prefix by slowest watch instead of compaction.
• Refactor code
• Implement tests

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[chaochn47]

Limit event prefix by slowest watch instead of compaction.

Do we expect sending events to compacted watchers (the slowest one) though? I think it should not. WDYT?

[serathius]

Do we expect sending events to compacted watchers (the slowest one) though? I think it should not. WDYT?

I expect that minRev and rev of victims is above compactedRev, if now we can just add a protection.

[k8s-ci-robot]

@serathius: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-etcd-unit-test-amd64	05db321	link	true	/test pull-etcd-unit-test-amd64

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.