release-23.2.0-rc: release-23.2: sql: prevent gateway from always being picked as the default

[blathers-crl]

Backport 1/1 commits from #115388 on behalf of @adityamaru.

/cc @cockroachdb/release

---

Backport 2/2 commits from #114537.

/cc @cockroachdb/release

---

Previously, the instance resolver would always
assign the partition span to the gateway if the
gateway was in the set of eligible instances and
we did not find an eligible instance with a better
locality match. In large clusters during backup/cdc
running with execution locality, this could cause
the gateway to get the lions share of work thereby
causing it to OOM or severely throttle performance.

This change make span partitioning a little more
stateful. Concretely, we now track how many partition
spans have been assigned to each node in the planCtx
that is used throughout the planning of a single statement.
This distribution is then used to limit the number of
partition spans we default to the gateway. Currently, by
default we allow the gateway to have:

2 * average number of partition spans across the other instances

If the gateway does not satisfy this heuristic we randomly
pick one of the other eligible instances. Note, if there
are no eligible instances except for the gateway, or the
gateway has received no spans yet, we will pick the gateway.

Fixes: #114079
Release note (bug fix): fixes a bug where large jobs running
with execution locality could result in the gateway being assigned
most of the work causing performance degradation and cluster
instability

Release justification: fixes a condition that would cause a hotspot for bulk jobs such as backup, cdc that occurs because all spans are assigned to the gateway node

---

Release justification:

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Backports should only be created for serious
  issues or test-only changes.
• Backports should not break backwards-compatibility.
• Backports should change as little code as possible.
• Backports should not change on-disk formats or node communication protocols.
• Backports should not add new functionality (except as defined
  here).
• Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
• All backports must be reviewed by the owning areas TL and one additional
  TL. For more information as to how that review should be conducted, please consult the backport
  policy.

If your backport adds new functionality, please ensure that the following additional criteria are satisfied:

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
• Your backport must be accompanied by a post to the appropriate Slack
  channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

[cockroach-teamcity]

This change is 

[yuzefovich]

Reviewed 9 of 9 files at r1, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt, @mgartner, and @rharding6373)