admission: introduce elastic io token exhausted duration metric

[aadityasondhi]

This patch adds a new metric elastic_io_tokens_exhausted_duration.kv. This is similar to the existing
admission.granter.io_tokens_exhausted_duration.kv, but for elastic traffic.

The patch also does some code refactoring to make it easier to use both regular and elastic equivalent metrics.

Informs #121574.

Release note: None

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[aadityasondhi]

In DB console:

[sumeerbhola]

Reviewed 4 of 4 files at r1, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @aadityasondhi)

---

pkg/util/admission/granter.go line 471 at r1 (raw file):

		updateExhaustedTokenDurationLocked(tokenCount, avail[w], sg.coordMu.availableIOTokens[w], settingAvailableTokens,
			&sg.exhaustedStart[w], sg.ioTokensExhaustedDurationMetric[w])
	}

This looping and separation of the updateExhausterTokenDurationLocked with all these params is hard to follow. I think the following alternative is simpler:

func (sg *kvStoreTokenGranter) subtractTokensLocked(
	count int64, elasticCount int64, settingAvailableTokens bool,
) {
	sg.subtractTokensLockedForWorkClass(admissionpb.RegularWorkClass, count, settingAvailableTokens)
	sg.subtractTokensLockedForWorkClass(admissionpb.ElasticWorkClass, elasticCount, settingAvailableTokens
	if !settingAvailableTokens {
		if count > 0 {
			sg.tokensTakenMetric.Inc(count)
		} else {
			sg.tokensReturnedMetric.Inc(-count)
		}
	}
}

func (sg *kvStoreTokenGranter) subtractTokensLockedForWorkClass(wc admissionpb.WorkClass, count int64, settingAvailableTokens bool) {
	sg.coordMu.availableIOTokens[wc] -= count
	sg.availableTokensMetric[wc].Update(sg.coordMu.availableIOTokens[wc])
	// Inline the code from updateExhaustedTokenDurationLocked.
  ...
}

[aadityasondhi]

TFTR, PTAL.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @sumeerbhola)

---

pkg/util/admission/granter.go line 471 at r1 (raw file):

Previously, sumeerbhola wrote…

This looping and separation of the updateExhausterTokenDurationLocked with all these params is hard to follow. I think the following alternative is simpler:

func (sg *kvStoreTokenGranter) subtractTokensLocked(
	count int64, elasticCount int64, settingAvailableTokens bool,
) {
	sg.subtractTokensLockedForWorkClass(admissionpb.RegularWorkClass, count, settingAvailableTokens)
	sg.subtractTokensLockedForWorkClass(admissionpb.ElasticWorkClass, elasticCount, settingAvailableTokens
	if !settingAvailableTokens {
		if count > 0 {
			sg.tokensTakenMetric.Inc(count)
		} else {
			sg.tokensReturnedMetric.Inc(-count)
		}
	}
}

func (sg *kvStoreTokenGranter) subtractTokensLockedForWorkClass(wc admissionpb.WorkClass, count int64, settingAvailableTokens bool) {
	sg.coordMu.availableIOTokens[wc] -= count
	sg.availableTokensMetric[wc].Update(sg.coordMu.availableIOTokens[wc])
	// Inline the code from updateExhaustedTokenDurationLocked.
  ...
}

Done. Thanks for the suggestion, code looks much cleaner (and more aligned with what i was looking to do here).

[sumeerbhola]

Reviewed 1 of 1 files at r2.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @aadityasondhi)

[aadityasondhi]

bors r=sumeerbhola

[craig]

Build succeeded:

• local_roachtest_fips
• unit_tests
• docker_image_amd64
• linux_amd64_fips_build
• check_generated_code
• macos_arm64_build
• lint
• linux_arm64_build
• linux_amd64_build
• windows_build
• examples_orms
• macos_amd64_build
• acceptance
• local_roachtest