PolicyAttachment is complex

[keithmattix]

Context: This was brought up during a GAMMA meeting and subsequently raised during a Gateway API community meeting.

PolicyAttachment is one of the most powerful yet intimidating parts of the Gateway API spec. While it allows the spec's fundamental resources (e.g. xRoutes, Gateways) to avoid bloat, the mechanism itself is burdensome for implementations and users alike.

CRD Proliferation

If an implementation wants to expose some data plane specific configuration (e.g. timeouts, h2 settings, etc.), they must commit to creating and maintaining a new CRD (possibly per config domain). At that point, all of the headache around CRD versioning, maintenance, and support kick in. This isn't typically desirable, and many ingress controllers and meshes alike have received negative feedback from users about numerous CRDs.

Defaults and Overrides

The defaults and overrides system of policy attachment is a powerful mechanism to enable the multi-persona capabilities of the Gateway API. However, it's a lot of mental capital for a user to consider all of the interactions between different levels of defaults + overrides if they just want to configure a timeout or set a TCP keepalive. Furthermore, for GAMMA implementations, much of the hierarchy is unnecessary since, in the draft GAMMA spec, there's no listeners, gateways, or gatewayclasses at play.

What are other's thoughts on PolicyAttachment? Should it be changed? Is there another metaresource that should be created?

[sunjayBhatia]

Reporting what Policy is in effect on a Route is also not well defined, so more complex UX and implementation details there (see discussion started here #1531)

[robscott]

I've been thinking about this a lot as well, and would like to find ways to make this better. It seems like we're already starting to explore a few paths forward:

1. Build more into the API itself. If we can be very intentional about finding commonly configured concepts that can be implemented by a variety of underlying implementations, we should attempt to include that in the spec.
2. Consider moving forward with @youngnick's proposal to formally define metaresources/lightweight policy attachment without any kind of hierarchy.
3. Consider sharing policy resources among implementations that are using a shared underlying dataplane (Envoy, NGINX, etc).

Open to additional approaches here, but if we were to move forward with all of these directions, would it help resolve some of the complexity issues?

[shaneutt]

It occurs to me that number 3 is just another way of saying number 1, but maybe with some additional context on top?

I'm interested in seeing GEP-713 progress and see if we can't shape it to make mutual sense for everyone.

[robscott]

I think they're slightly different. For example, with timeouts:

1. would result in Timeout config either being added to a resource like HTTPRoute or a new official policy resource intended to work for everyone.

2. would result in EnvoyTimeoutPolicy, NGINXTimeoutPolicy, etc. Essentially if 1 is not possible due to a lack of common config, then 3 still allows us to share config across implementations with the same underlying dataplane tech.

[shaneutt]

@keithmattix this post seems to be suggesting that Gateway API is leaning on PolicyAttachment to avoid having to maintain common functionality upstream. Generally speaking the opposite is desired: we want shared functionality to make it to the upstream CRDs and the non-shared functionality into downstream CRDs as efficiently as we can. Are there some concrete examples of where we've not done this well that you have in mind? Some things we should perhaps revisit?

[keithmattix]

Timeouts are the main thing, but I also think about low-level settings that are likely purposefully out of scope like arbitrary TCP connection config (e.g window sizes)

[shaneutt]

So perhaps it would be worth taking another pass at seeing what we might be able to do with timeouts in upstream? I get that the timing pressure of how long it takes to get something like this done still leaves some room for concern about CRD proliferation as implementations but I can't help but think that we'll find the best path in pursuit of one of the most important concrete issues 🤔

[youngnick]

I've lost some of my notes on Envoy timeouts, but recovered others, so I'm going to attempt a document on the differences in available timeouts between (at least) Envoy, Nginx, and HAProxy. Further data path implementation suggestions welcomed. That can then form the basis for a timeout-related discussion specifically.

[keithmattix]

Sounds good @youngnick , thanks! @shaneutt I agree; I think opening the timeouts discussion (again) will reveal the optimal solution

[arkodg]

We tried to experiment with PolicyAttachment in Envoy Gateway to solve the below use cases

• Handle personas - Ability for the admin as well app devs to configure policies without letting app dev override certain fields
• Add capability - Ability to attach policies on the Gateway resource

We ended up not using it due to the following reasons

• User experience - the final policy applied to the HTTPRoute or Gateway is hard to comprehend for the user.
• Merge Semantics - merge semantics for lists and maps are undefined in the spec
• ExtensionFilters - extension filters are easier to understand and troubleshoot by the end user, however they cannot be applied to the Gateway resource, blocking us from building out implementation specific features applicable to the Gateway/Listener

More details in envoyproxy/gateway#675

[shaneutt]

The user experience issue rings true for me as well. For me, it is a similar problem like we've heard with reference grant where the end users get confused about the effects because it's difficult to see or otherwise be cognizant of the references.

[pleshakov]

I'd add that it is also not clear when to use a Policy or a Custom filter.

More use case examples (for an advanced proxy):

• backend properties: timeouts, load balancing algorithm, health checks, retries, session persistence, max connections per backend pod...
• security features: rate-limiting, authentication, authorization, HSTS, CORS, WAF
• misc: log format, data plane specific configuration options and tunables

[youngnick]

This is on me, I've had the PR clarifying about Policy Attachment vs more generic metaresources pending for a long time.

Policy attachment is definitely not a completed idea. It's lacking definition around the exact things that @arkodg and @sunjayBhatia mentioned: the merge behavior and how it should interact with other things like custom filters. (I updated the wording about custom filters in #1565, but that's not merged yet.)

Policy Attachment is directly designed to handle use cases like timeouts, because when we were building Contour, every time we added a timeout to HTTPProxy, soon after we got a request from a proxy admin for them to have the ability to either default or override that timeout. I think the same will apply here, even if we add official fields for timeouts to HTTPRoute (for example), Gateway admins will want to do things like say "no infinite request timeouts", since that's really bad when you're running a shared proxy (connection table starvation produces difficult-to-troubleshoot symptoms).

Defaults are also intended to be used to help remove boilerplate, because many fields have very deployment-dependent defaults. Many Envoy timeouts are only enabled if you specify a value, because they can produce unexpected behavior when you set them, even if they are useful in some cases. A great example here is the HTTP max_connection_duration - this is used to ensure a maximum lifetime for any connection, which is useful in some cases, but really sucks in others (like long-running gRPC connections).

In terms of the proliferation of CRDs - I don't see many ways around this, we really only have a few options:

• We add full-fledged fields, with at least extended support, for things that would otherwise be governed by Policy (like timeouts) to Gateway API objects. This has advantages
  • Easy to test
  • Very clear
  • Behavior will be well defined.
    It has disadvantages too though
  • There's no way to do defaults or overrides without building something of a similar shape to a Policy metaresource
  • Having no configurable defaults means we need to choose a default that's suitable in a lot of cases (which is harder than it seems a lot of the time), or have no default and require the setting of the field in every object it needs to be used.
  • Finding the commonalities between enough different implementations is a nontrivial design exercise that will probably involve a few round of back-and-forth design.
• We add a standard set of Policy objects. These could be very generic (eg TimeoutPolicy) or dataplane-specific (eg EnvoyTimeoutPolicy, NginxTimeoutPolicy)
  Advantages:
  • Testable
  • Well-defined behavior
  • We would have defaults and overrides available in a standard way, which should make how the pattern works more clear to new users.
    Disadvantages:
  • Finding commonalities again.
• We have implementations do implementation specific things (this is the current state)
  Advantages:
  • We're deferring the hard design work until we have some proven designs.
    Disadvantages:
  • Puts a lot of load on implementations
  • The pattern is not clear
  • Doesn't solve the status and user information problem.

I'd argue that in the GAMMA use case, although there's not an obvious hierarchy like GatewayClass -> Gateway -> Route, there is an implied hierarchy that would probably be useful to have overrides and defaults available at: Cluster -> Namespace -> Service. It seems likely to me that people would, for example, want to say "no infinite client request timeouts allowed in this cluster", or "all Services in this namespace should default to a longer request timeout".

tl;dr I agree that Policy Attachment is complex, but it's solving a complex problem. I welcome further suggestions for simplifying it.

[howardjohn]

My gut feeling is a TimeoutPolicy is better than an EnvoyTimeoutPolicy. While they may not be 100% mappable, I hope we can make them close enough that its still useful to users.

Most users probably don't appreciate the subtle differences between 10 different timeouts - its plausible that we can have common fields in a common policy (example with out research: request timeout and connection timeout) and document how implementations should behave. The precise behavior may be slightly different between nginx and envoy, for example, but the intent from the user is still there.

If it can be reasonably documented with some wiggle room and can be tested, its likely "close enough" that we can manage to use the same field for not-exactly-the-same things.

[keithmattix]

I think I mostly agree with that @howardjohn. The issue I think becomes conformance tests; while a TimeoutPolicy is better than an EnvoyTimeoutPolicy (for the sake of naming if nothing else), NGINX doesn't want to be marked as non-conformant just because it's behavior differs slightly than Envoy. Does that mean different policies have different conformance levels? Maybe - that's probably not too awful for something like timeout. But I definitely don't want a plurality to win here, i.e. the policy uses Envoy semantics, naming, etc. because there happen to be more people in the room using envoy as a dataplane. Dataplane specific policies can be tricky, but resources name isn't the only way to differentiate; we could do it at the group level for example. Regardless, it's definitely going to require some more thought and conversation

[howardjohn]

I was hoping they are close enough we can have 1 timeout field, with a vague enough specification and tests that multiple slightly-different implementations can exist. There are some more subtles areas like this already in the current spec!

[bowei]

As long as the vague specification is useful and understandable for the user is the key. So would be good to get info from that angle rather than us (implementors).

[arkodg]

raised #1741 to focus and collaborate on the timeout discussion

[youngnick]

I think between Policy Attachment update in #1565 (hopefully to be merged soon), and the TimeoutPolicy in #1742, we should be clarifying Policy Attachment a lot. Once those are done, does everyone watching this feel like we've done enough to mark this as "answered"?