Is there a way to batch bbolt commits caused by deletions?

[JeremyScott106]

Good afternoon folks,
My team has been facing an issue for a couple years and not too long ago pinpointed the cause of the problems on our end. Since ETCD v3.5.12, we have seen regression in our systems due to how we use the leasing system.

We have made some adjustments on our end, which have mitigated the problems we faced, but we would prefer our previous settings.

Our preferred use case is having 500-3000 leases that renew every 1.5 to 3 seconds using jetcd. These leases may be cleared in bulk, in some cases, 700+ leases all get revoked at once.

When this happens, we see all leases get a new grant with a TTL of 3 to 6 seconds. The TTL is based on variables we use at the time.

Problem:
On the versions of ETCD v3.5.12+, our maximum revoked lease capability takes a huge hit, and we get stuck in a loop of lease grants, no lease renewal, lease revoke. It took sometime to realize that the renewals were not happening because the revokes had the backend tied up entirely. During these revoke backlogs, our observed successful lease-renewal rate drops to zero. Previous versions of ETCD had these revokes batched and that allowed us to continue doing renewals during that timeframe that the mass loss of leases did not cause any issues.

This, by our best understanding, came from commit github.com/etcd-io/etcd/commit/bd5915f71086cdc7674618e662a750da53ab5cf6

We have since worked our leasing count down from 500-3000 leases down to just under 100 leases being renewed. It works without failure, however, it is not our preferred way of doing this.

Is there a flag or env setting that would allow multiple queued lease revokes to share a bbolt backend commit while preserving the correctness guarantee that the change was put in place to resolve?

During the previous failures, we saw each etcd member report metrics under 'sum(rate(etcd_debugging_lease_revoked_total[1m])) by (instance) show between 250 and 300 revokes per second per member consistently. If I logged into one of the member, I would see the active leases climbing rather than staying about the same as we could not revoke faster than the grants coming in. In the same timeframe, renewals reported 0 operations. Before this version of etcd, we see the same pods reach close to 750 for the same metric, and not cause the lease renewals to drop to 0.

What I would hope for, a flag or env setting that would batch the revokes that are stacked up and waiting to occur. Another option, allow renewals to still occur even while revokes are pending. I can think of reasons why that second option probably isn't preferred, and possibly the entire cause of the change I shared in this post, so I get that as a hard no if it is. In testing I have done, our bbolt commits go up at the same rate as revokes, this is why we believe this is our cause.

Thank you for any thoughts/guidance in the matter here,
Jeremy