quotapool,kvserver: extend quotapool.RateLimit, rate limit queue addition in SystemConfigUpdate

[ajwerner]

See individual commits.

This PR is a fancier implementation of #53603 that permits bursting and makes
setting a relatively low rate feel less risky in the face of a small burst of
changes.

Release justification: low risk, high benefit changes to existing functionality

Release note (performance improvement): Limited the frequency of an expensive
operation due to schema changes making workload which perform schema changes
at a high rate less resource intensive.

[cockroach-teamcity]

This change is 

[knz]

I just reviewed this and I feel rather comfortable by the functional correctness of the thing (i.e. I can't see it blocking unexpectedly)

Can you speak a bit more quantitatively about what motivated this change and the visible benefits?

Reviewed 2 of 2 files at r1, 2 of 2 files at r2.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @knz and @tbg)

[knz]

Reviewed 1 of 1 files at r4.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @knz and @tbg)

[ajwerner]

Prior to this change, over 70% of the CPU utilization of the typeorm roachtest was underneath those calls. The change decreases the runtime of the typeorm by 30% (after applying #53589, have not measured the impact of this change against master but I'd suspect it to be even larger as that PR reduces the amount of work done to run jobs). I'm rerunning on top of master and against master to compare those runtime and CPU differences.

Before:

--- PASS: typeorm (1444.19s)
--- PASS: typeorm (1463.58s)
--- PASS: typeorm (1530.58s)

After

--- PASS: typeorm (1094.87s)
--- PASS: typeorm (1094.73s)
--- PASS: typeorm (1084.94s)

[ajwerner]

According to a 30s profile from before and after, this change reduces the CPU utilization due to the calls to adding to the queue from 69.7% to 4.9% and reduces the CPU utilization overall dramatically. I'll post runtime differences when they finish. See the below charts:

Before:

After:

[knz]

this looks gorgeous!

I'm LGTM on this

[ajwerner]

TFTR!

bors r=knz

typeorm runtime:

Before:

--- PASS: typeorm (1851.56s)
--- PASS: typeorm (1866.40s)
--- PASS: typeorm (1927.17s)

After:

--- PASS: typeorm (1599.05s)
--- PASS: typeorm (1632.52s)
--- PASS: typeorm (1640.89s)

[craig]

Build succeeded:

• GitHub CI (Cockroach)