Label balancing

[KirillPamPam]

Label-based balancing

Summary

Adds a second, optional balancing mode on top of the existing per-method rating.

Until now every request to a chain was balanced across all of that chain's upstreams
using the score-policy rating (latency, error rate, availability, …). This PR introduces
priority-group balancing: operators tag upstreams with group-labels, declare an
order of those labels, and requests are served from the highest-priority group first,
falling through to lower-priority groups when the current group can't serve. Rating still
decides ordering within a group.

Typical use case: prefer cheap full nodes, fall back to archive nodes only when the
full group can't handle a request.

Configuration

label-balancing is configured as a global default under upstream-config and can be
overridden per chain under chain-defaults.<chain> (same pattern as failsafe-config).
When neither is set, behavior is unchanged (pure rating).

upstream-config:
  # global default — applies to every chain unless overridden per chain
  label-balancing:
    order:                  # group label names, highest priority first
      - full
      - archive
      - fast
    pass-on-error: false    # see semantics below; default false
    include-default: true   # unlabeled upstreams form a final fallback group; default true
  chain-defaults:
    polygon:
      label-balancing:      # per-chain override fully replaces the global block for polygon
        order:
          - archive
          - full
  upstreams:
    - id: up1
      chain: ethereum
      group-labels: [full, fast]
      connectors: [{ type: json-rpc, url: https://full-node.example.com }]
    - id: up2
      chain: ethereum
      group-labels: [archive]
      connectors: [{ type: json-rpc, url: https://archive-node.example.com }]

Fields

Field	Description
order	Ordered list of group label names, highest priority first. Required when label-balancing is set; entries must be non-empty and unique.
pass-on-error	Controls retry routing after a retryable error response (default false).
include-default	When true (default), upstreams carrying none of the order labels form a final fallback group tried after all configured groups. When false, they are excluded from routing while label-balancing is active.
group-labels (per upstream)	The priority-group labels this upstream belongs to. Config-defined and independent of the runtime labels produced by label detectors. An upstream may belong to several groups but is selected at most once per request.

Algorithm

• Groups are visited in order, then the default group (unlabeled upstreams) last.
• Within a group, upstreams keep the usual rating order.
• An upstream is selected at most once per request, even if it belongs to several
  groups (guaranteed by the strategy's accumulating selectedUpstreams set, which persists
  across failsafe retries/hedges within a request).
• pass-on-error: false (default): on a retryable error, retry within the current group
  (next-best untried upstream); advance to the next group only once the current group has no
  selectable upstream left.
• pass-on-error: true: on a retryable error, jump straight to the next group, skipping
  untried upstreams in the current group.
• In both modes, an entirely-dead group (every upstream unavailable / method banned /
  rate-limited) always falls through to the next group, regardless of pass-on-error.

Implementation

The feature reuses the existing selection machinery rather than duplicating it:

• LabelGroupStrategy (internal/upstreams/flow/label_group_strategy.go) — a new
  UpstreamStrategy. A single instance lives for one request; its SelectUpstream walks the
  ordered groups with a cursor and delegates within-group selection to the existing
  filterUpstreams (status/method/ws matchers, rate limits, already-tried skipping). The
  false/true pass-on-error modes are the only difference in how the cursor advances.
• NewLabelGroupStrategy builds the ordered groups itself from the rating registry
  (registry.GetSortedUpstreams → PartitionLabelGroups), mirroring how NewRatingStrategy
  pulls its sorted list from the registry. createStrategy stays a one-liner. A
  NewLabelGroupStrategyWithGroups constructor keeps the explicit-groups path for tests.
• createStrategy (internal/upstreams/flow/execution_flow.go) builds a
  LabelGroupStrategy when label-balancing is configured for the chain, otherwise keeps
  RatingStrategy. Quorum and subscribe paths are unaffected; sticky-send composes via the
  existing additional matcher.
• Config (internal/config) — LabelBalancingConfig, group-labels on upstreams, the
  global + per-chain fields, a LabelBalancingFor(chain) resolver, validation, and an
  include-default default of true.
• Upstream.GetGroupLabels() returns a mapset.Set[string] built once at
  construction, so partitioning does O(1) membership checks instead of re-scanning a slice.

Tests

• internal/upstreams/flow/label_group_strategy_test.go — group exhaustion before advancing,
  dead-group fall-through, pass-on-error group-jumping, selected-once across overlapping
  membership, empty-groups, and PartitionLabelGroups partitioning (incl. default group).
• internal/config/label_balancing_test.go — validation (empty/duplicate order),
  include-default defaulting, and LabelBalancingFor per-chain/global resolution.

Docs

docs/nodecore/05-upstream-config.md gains a label-balancing section plus group-labels
and per-chain label-balancing field entries.

Notes

• With pass-on-error: true and only a single group configured, a retryable error has
  nowhere to escalate, so only one upstream is tried per request before failing — the literal
  "jump on error" semantics. This is an unusual config.

[tonatoz]

Solid feature — clean reuse of filterUpstreams, mirrors the existing RatingStrategy/failsafe-config patterns, with docs and focused unit tests. A few non-blocking suggestions:

1. Retry-delay default change — please scope or document it

The PR drops the implicit retry.delay = 300ms default and relaxes validation to < 0. Since internal/resilience/failsafe.go only applies backoff when Delay > 0, this means every user with retries but no explicit delay now gets zero-delay, back-to-back retries — not just label-balancing users. This is a global behavior change that isn't mentioned in the PR description (only the "Default: 300ms" doc line was removed).

Suggestion: keep the 300ms default and let label-balancing configs set delay: 0 explicitly. If the change is intentional, please call it out in the PR description / changelog and add a note in the retry docs, since it's effectively a breaking change for retry timing.

2. pass-on-error vs. hedging

pass-on-error advances the group cursor on every non-first SelectUpstream call, which assumes calls are sequential retries (one call = one error). With hedging enabled, the concurrent second attempt from the initial fan-out also sees firstCall == false and jumps to the next group even though no error occurred — so initial hedges get spread across groups.

Suggestion: document that pass-on-error is designed around retries (not concurrent hedges), or guard the cursor advance so it only triggers on an actual error path. Even a comment on SelectUpstream clarifying the assumption would help future readers.

3. Test coverage suggestions

The strategy-level and PartitionLabelGroups tests are good. A few additions would raise confidence:

• NewLabelGroupStrategy (registry path): the GetSortedUpstreams → GetUpstream → GetGroupLabels → PartitionLabelGroups wiring (incl. the up == nil branch) is currently untested — only NewLabelGroupStrategyWithGroups is exercised.
• End-to-end through createStrategy/execution_flow: the actual feature behavior — falling through groups via retries — isn't covered at the flow level.
• include-default: false at the strategy level (only covered in the PartitionLabelGroups test today).
• Concurrency: a test exercising hedge + pass-on-error would lock down the behavior from point 2.

None of these block merge — point 1 is the one I'd most like a follow-up or a line in the description on.