You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Rethinking BackendConfigPolicy and BackendTLSPolicy merge semantics
#13998
This discussion is initiated by issue #13918, but while digging in I think the underlying problem is broader and needs the wider community input before we land a fix.
#13918 is the current visible user behaviour. The deeper question is how BackendConfigPolicy and BackendTLSPolicy should compose, both individually and against each other. In the current implementation, neither story is fully defined.
Problem 1: BackendConfigPolicy has no merge semantics
BackendTLSPolicy is single purpose. Its entire spec like validation (CA refs, hostname, SANs) is one cohesive TLS configuration. When two BackendTLSPolicies target the same backend, picking one winner (oldest creation timestamp, lexical tiebreak by name; losers reported Accepted=False, Reason=Conflicted) is the right model and is what Gateway API 1.4 mandates. kgateway already implements this via WinnerPolicyIndexByCreationTimeAndRef and MergePolicies in the BackendTLSPolicy plugin.
BackendConfigPolicy is fundamentally different. Its spec is a bunch of orthogonal sub features like connectTimeout, healthCheck, circuitBreakers, upstreamProxyProtocol etc. These are independent feature configs, not parts of a single atomic policy. A user reasonably wants to express "team wide circuit breakers in policy A" + "service specific connect timeout in policy B" and have both apply.
In the current implementation, BackendConfigPolicy plugin does not register a MergePolicies function and every BackendConfigPolicy applies, last writer wins per field. In my opinion, having something like a AugmentedShallowMerge per top level field is better for the BackendConfigPolicy as older policy's set fields take precedence, newer policies fill in unset fields.
Problem 2: BackendConfigPolicy spec.tls vs BackendTLSPolicy cross policy conflict resolution
Even after Problem 1 is solved, BackendConfigPolicy and BackendTLSPolicy overlap on exactly one field: TLS. Both produce an Envoy transport_socket for the cluster. In Gateway API, BackendTLSPolicy spec (targetRefs, validation, options) is entirely TLS scoped, so this is the only cross policy collision surface as every other BackendConfigPolicy field is orthogonal to BackendTLSPolicy and applies independently.
In the current implementation, the collision is undefined. So whichever of BackendConfigPolicy and BackendTLSPolicy runs last overwrites out.TransportSocket. This has an impact to the applyUpstreamProxyProtocol as wraps the existing transport socket.
In my opinion, BackendTLSPolicy should be given priority over the BackendConfigPolicy as BackendTLSPolicy is standard Gateway API resource.
There are 2 solutions that I can think of.
1. Defer transport socket generation to a new IR
Instead of having BackendConfigPolicy and BackendTLSPolicy write out.TransportSocket directly, introduce a BackendTransportSocketIR with explicit fields (TLS, UpstreamProxyProtocol, plus their sources). Each plugin contributes only into its own fields; neither can overwrite the other's fields. After all backend policies have run, a single assembly step composes the final out.TransportSocket. Precedence (BackendTLSPolicy wins for TLS) is enforced at field write time.
2. Make BackendConfigPolicy and BackendTLSPolicy processing conflict aware.
Keep the current "write directly to out.TransportSocket" model, but make both plugins aware about each other. BackendTLSPolicy becomes wrap aware: if it sees out.TransportSocket already wrapped in ProxyProtocolUpstreamTransport, it replaces the inner socket instead of overwriting the wrapper. BackendConfigPolicy checks for an attached BackendTLSPolicy
before writing TLS, and skips the TLS write when BackendTLSPolicy is present.
Would appreciate some feedback on this by the maintainers so that I can start implementing the fix for #13918.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
This discussion is initiated by issue #13918, but while digging in I think the underlying problem is broader and needs the wider community input before we land a fix.
#13918 is the current visible user behaviour. The deeper question is how BackendConfigPolicy and BackendTLSPolicy should compose, both individually and against each other. In the current implementation, neither story is fully defined.
Problem 1: BackendConfigPolicy has no merge semantics
BackendTLSPolicy is single purpose. Its entire spec like validation (CA refs, hostname, SANs) is one cohesive TLS configuration. When two BackendTLSPolicies target the same backend, picking one winner (oldest creation timestamp, lexical tiebreak by name; losers reported Accepted=False, Reason=Conflicted) is the right model and is what Gateway API 1.4 mandates. kgateway already implements this via
WinnerPolicyIndexByCreationTimeAndRefandMergePoliciesin the BackendTLSPolicy plugin.BackendConfigPolicy is fundamentally different. Its spec is a bunch of orthogonal sub features like connectTimeout, healthCheck, circuitBreakers, upstreamProxyProtocol etc. These are independent feature configs, not parts of a single atomic policy. A user reasonably wants to express "team wide circuit breakers in policy A" + "service specific connect timeout in policy B" and have both apply.
In the current implementation, BackendConfigPolicy plugin does not register a MergePolicies function and every BackendConfigPolicy applies, last writer wins per field. In my opinion, having something like a
AugmentedShallowMergeper top level field is better for the BackendConfigPolicy as older policy's set fields take precedence, newer policies fill in unset fields.Problem 2: BackendConfigPolicy
spec.tlsvs BackendTLSPolicy cross policy conflict resolutionEven after Problem 1 is solved, BackendConfigPolicy and BackendTLSPolicy overlap on exactly one field: TLS. Both produce an Envoy
transport_socketfor the cluster. In Gateway API, BackendTLSPolicy spec (targetRefs, validation, options) is entirely TLS scoped, so this is the only cross policy collision surface as every other BackendConfigPolicy field is orthogonal to BackendTLSPolicy and applies independently.In the current implementation, the collision is undefined. So whichever of BackendConfigPolicy and BackendTLSPolicy runs last overwrites
out.TransportSocket. This has an impact to the applyUpstreamProxyProtocol as wraps the existing transport socket.In my opinion, BackendTLSPolicy should be given priority over the BackendConfigPolicy as BackendTLSPolicy is standard Gateway API resource.
There are 2 solutions that I can think of.
1. Defer transport socket generation to a new IR
Instead of having BackendConfigPolicy and BackendTLSPolicy write
out.TransportSocketdirectly, introduce aBackendTransportSocketIRwith explicit fields (TLS, UpstreamProxyProtocol, plus their sources). Each plugin contributes only into its own fields; neither can overwrite the other's fields. After all backend policies have run, a single assembly step composes the finalout.TransportSocket. Precedence (BackendTLSPolicy wins for TLS) is enforced at field write time.2. Make BackendConfigPolicy and BackendTLSPolicy processing conflict aware.
Keep the current "write directly to out.TransportSocket" model, but make both plugins aware about each other. BackendTLSPolicy becomes wrap aware: if it sees
out.TransportSocketalready wrapped in ProxyProtocolUpstreamTransport, it replaces the inner socket instead of overwriting the wrapper. BackendConfigPolicy checks for an attached BackendTLSPolicybefore writing TLS, and skips the TLS write when BackendTLSPolicy is present.
Would appreciate some feedback on this by the maintainers so that I can start implementing the fix for #13918.
All reactions