load_balancing: Add Load Aware Locality LB Policy

[jukie]

Commit Message: load balancing: add load-aware locality-picking LB policy

Additional Description: New LB policy (envoy.load_balancing_policies.load_aware_locality) that uses ORCA utilization data to choose localities based on real-time headroom, as proposed in #43665.

Risk Level: low (new extension behind a new config proto, core changes are additive)

Testing: Unit tests for config validation, weight computation (EWMA, variance threshold, probe percentage, all overloaded fallback, topology changes), per-locality host partitioning, and weighted random selection. Integration test with a multi-locality cluster verifying traffic shifts in response to ORCA utilization reports.

Docs Changes: proto comments serve as initial documentation.

Release Notes: added new extension envoy.load_balancing_policies.load_aware_locality.

Platform Specific Features: n/a
Fixes #43665
[Optional API Considerations:]
API Considerations:

• New proto LoadAwareLocality in envoy/extensions/load_balancing_policies/load_aware_locality/v3/.
  • endpoint_picking_policy field accepts any endpoint-picking child policy.
  • metric_names_for_computing_utilization is declared but not yet honored (documented in proto comments as TODO).
• New virtual method orcaUtilization() on HostDescription interface.

AI was used during implementation and for writing tests but I fully understand the changes here

[repokitteh-read-only]

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #43784 was opened by jukie.

see: more, trace.

[jukie]

This approach removes the OrcaWeightManager extraction (#43695) as a prerequisite - the policy reads utilization directly via a new lightweight per-host OrcaUtilizationStore rather than reusing CSWRR's weight management machinery.

How it works:

• A main-thread timer periodically reads host->orcaUtilization() from healthy hosts, averages per locality, applies EWMA smoothing, and computes capacity-weighted headroom (healthy_hosts × (1 - utilization)).
• The resulting routing weights are pushed to workers via TLS.
• Workers select a locality by weighted random and delegate endpoint selection to per-locality child LB instances (e.g., round_robin, client_side_weighted_round_robin).

LoadAwareLocalityLoadBalancer (main thread, ThreadAwareLoadBalancer)
  |
  |-- weight_update_timer_ (plain Event::Timer)
  |     Fires periodically to recompute locality routing weights.
  |
  |-- on timer callback:
  |     computeLocalityRoutingWeights()
  |       Reads host->orcaUtilization() from each healthy host, averages
  |       per locality, applies EWMA smoothing, computes capacity weighted
  |       by healthy host count (healthy_hosts * headroom), checks variance
  |       threshold for local-zone preference, applies probe percentage,
  |       publishes immutable snapshot.
  |
  +-- WorkerLocalLbFactory (shared across all workers)
        |
        |-- child_thread_aware_lb_ (single shared child ThreadAwareLoadBalancer)
        |     Created and initialized on the main thread. Workers call
        |     factory()->create() from it to build per-locality worker LBs.
        |
        |-- tls_ (ThreadLocal::TypedSlot<ThreadLocalShim>)
        |     Main thread pushes RoutingWeightsSnapshot to workers via
        |     runOnAllThreads(). Workers read from their TLS slot with
        |     zero synchronization.
        |
        +-- create() --> WorkerLocalLb (one per worker thread)
              |
              |-- selectLocality() [weighted random by capacity]
              |     Weighted random across localities proportional to
              |     host-count-weighted headroom.
              |
              +-- per_locality_[] (one PerLocalityState per locality)
                    |-- PrioritySetImpl (hosts for this locality only)
                    +-- LoadBalancer (worker-local child instance, e.g., RoundRobin)

Design decisions:

• Locality selection and endpoint selection are fully decoupled — any endpoint-picking child policy works, including CSWRR (no lbPolicyData() conflicts since utilization flows through a separate OrcaUtilizationStore channel).
• Local-zone preference: when local utilization is within a configurable variance threshold of the remote average, routes 100% local.
• Probe percentage: ensures a minimum fraction of traffic reaches remote localities to keep ORCA data fresh even in all-local mode. If Out-Of-Band reporting is added this can be used as an alternative to completely avoid cross-zone traffic

[repokitteh-read-only]

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @wbpcode
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #43784 was ready_for_review by jukie.

see: more, trace.

[markdroth]

This approach looks really good from an xDS API perspective!

I'll let one of the Envoy maintainers review the implementation.

[markdroth]

/lgtm api

[jukie]

/retest

[agrawroh]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a new load balancing policy, load_aware_locality, which distributes traffic between localities based on real-time utilization data from ORCA reports. The implementation is robust, featuring EWMA smoothing for utilization metrics, a variance threshold to prefer the local locality when load is balanced, and a configurable probe percentage to ensure telemetry freshness. The code is well-structured, with clear separation between main-thread weight computation and per-worker load balancing logic. It also includes comprehensive unit and integration tests that cover a wide range of scenarios, including dynamic load shifts and various configuration parameters. The changes are additive and well-contained within the new extension. Overall, this is a high-quality contribution that adds significant new functionality to Envoy's load balancing capabilities.

_{Note: Security Review did not run due to the size of the PR.}

[jukie]

/retest

[phlax]

needs main merge - unrelated go problem is fixed there

[markdroth]

I'm not super familiar with Envoy's implementation, so I don't understand why we have this restriction, but I will note that it seems sub-optimal. In principle, the child policy and the parent policy should have exactly the same API, so delegation should be possible to any child policy. If that's not the case, then maybe we need some changes in Envoy's LB policy API to make that possible.

[jukie]

This new locality policy only works with child policies whose logic can be preserved on a locality-scoped set of hosts. This is similar to the limitations with combining weighted clusters with consistent hashing (#21675). When the host set is split before the child policy runs, policies like ring_hash and maglev lose their expected behavior because they're building their hash structures from an incomplete view of the host set.

If you'd prefer that choice be up to the user I can remove this restriction and document it as a limitation.

[markdroth]

load_aware_locality only works with child policies whose logic can be preserved on a locality-scoped set of hosts. Policies that depend on the full cluster's host set (such as the hashing-based policies rejected here) don't fit that model

I don't understand what that means, since I'm not that familiar with the Envoy LB policy API. But in principle, I would expect that the parent policy and child policy should both accept the list of endpoints in the same structure, since they are both implementing the same LB policy API. Given that, it's not clear to me what restrictions we would have here: I would expect that the parent policy would essentially just filter the list of endpoints it passes down to the child policy, and the child policy would just pick from among that filtered set of endpoints. What am I missing?

[jukie]

I would expect that the parent policy would essentially just filter the list of endpoints it passes down to the child policy

That's exactly what happens here but the problem is policies like ring_hash or maglev who depend on building a hash structure over the full cluster host set. Once the parent narrows that to a single locality, the child is no longer implementing the same policy.
To really support that, the parent would need to implement something like locality-aware hashing so a given hash key maps consistently to the same locality. That isn't necessarily impossible in the future but is a bit at-odds with the core goal of this policy which is to route based on load. I can add that on top if you feel strongly but I personally feel that would be better suited as a locality-aware hashing LB policy.

[jukie]

Again, this is similar to other limitations that exist in Envoy (a good list is in envoyproxy/gateway#5307 (comment)) so documenting it as a known limitation but still accepting these child policies could be reasonable as well. I'm open to changing this.

[markdroth]

If I'm understanding right, you're saying that the child's behavior won't be exactly the same underneath the locality-picking policy, because of things like the fact that ring_hash won't have the same set of hosts in its ring and will therefore not wind up picking the same endpoint for the same request hash?

If so, I think that's fine and is not something we should be disallowing. It's certainly something that people need to understand if they choose to configure those policies underneath a locality-picking policy, but I don't think we should go out of our way to disallow that.

I also think this hard-coded allow list is going to wind up being brittle, because no one will ever remember to update it when we add new LB policies over time.

[jukie]

Thanks for the feedback, updated.

[wbpcode]

Is this necessary here? I think we may prefer the solution in the client_side_weighted_round_robin? That's say, the LB will implement the specific HostLbPolicyData implementation and will compute the the necessary utilization?

The HostLbPolicyData is designed to do this task.

[wbpcode]

Maybe, Maybe we can move the requiresOrcaLoadReports to the LoadBalancerConfig, and we can add another absl::optional<double> onOrcaLoadReport(const Upstream::OrcaLoadReport& report); there to calculate the utilization based on the configuration.

Then combine the new OrcaUtilizationStore, we may could get better solution than previous HostLbPolicyData?

[jukie]

That makes sense, it would allow the LoadBalancerConfig to control which ORCA fields get extracted instead of the current hardcoded logic. I'll take a look at this.

I initially took a stab at extracting HostLbPolicyData into a central class with #43695 but I'll re-think this approach.

[wbpcode]

cc @jukie recently we do have lots of new LB policies. Orz. See #43588 where the LbPolicyData also be used.

I think we may also could enhance the current LbPolicyData mechanism, to support multiple LbPolicyData in the single host. 🤔

[paul-r-gall]

@jukie I agree with @wbpcode's recommendation for enhancing LbPolicyData to have multiple on a single host.

[jukie]

Sounds good, thanks for the feedback!

[jukie]

@paul-r-gall @wbpcode starting on the multi-entry approach in #43995 if you could take a look please. Let me know if you'd rather see that in this PR.

[wbpcode]

You may could remove me from the owner list because we may have no enough bandwidth to sponsor this new extension. 😞