release-23.2: kv: bound minimum raft scheduler workers per store

[blathers-crl]

Backport 1/1 commits from #120162 on behalf of @nvanbenschoten.

/cc @cockroachdb/release

---

In a448edc, we switched from replicating COCKROACH_SCHEDULER_CONCURRENCY on each store to evenly distributing the workers across each store. For example, in an 8-store, 32-vCPU node, the number of raft scheduler workers per store went from 96 to 12.

This was done to avoid scheduler thrashing and excessive memory usage in many-store nodes. Unfortunately, we have seen that this change could also lead to situations where workers were spread so thin across stores in a many-store system that any single store's worker pool could not keep up with temporary imbalanced load. In extreme cases where the PreIngestDelay mechanism is kicking in, this could lead to high scheduler latency across replica on the store.

This commit establishes an intermediate solution. We will continue to distribute workers across stores, but we will also ensure that each store has at least COCKROACH_SCHEDULER_MIN_CONCURRENCY_PER_STORE workers. This will prevents any single store from being able to absorb imbalanced load. The value defaults to GOMAXPROCS, so that in the previous example, each store would have at least 32 workers.

Epic: None

Release note (ops change): a minimum Raft scheduler concurrency is now enforced per store so that nodes with many stores do not spread workers too thin. This avoids high scheduler latency across replicas on a store when load is imbalanced.

---

Release justification: avoids cluster instability with too few raft workers per store.

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Backports should only be created for serious
  issues or test-only changes.
• Backports should not break backwards-compatibility.
• Backports should change as little code as possible.
• Backports should not change on-disk formats or node communication protocols.
• Backports should not add new functionality (except as defined
  here).
• Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
• All backports must be reviewed by the owning areas TL and one additional
  TL. For more information as to how that review should be conducted, please consult the backport
  policy.

If your backport adds new functionality, please ensure that the following additional criteria are satisfied:

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
• Your backport must be accompanied by a post to the appropriate Slack
  channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

[cockroach-teamcity]

This change is 

[github-actions]

Reminder: it has been 3 weeks please merge or close your backport!