Depreciation of TrafficRoutes in favor of supporting Kubernetes Gateway API - WIP

[nicholasjackson]

This issue is to discuss the overlaps between SMI and Kubernetes Gateway API and the possibility that where there is an overlap SMI adopts Gateway API in replacement for its own resources concentrating purely on service mesh specific resources.

Gateway API Spec:
https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io

TrafficSpecs Comparison

The below example shows comparable functionality as could be implemented using SMI or Gateway spec, the core difference in this specification is that SMI TrafficTarget controls service to service authorization.

kind: HTTPRouteGroup
metadata:
  name: the-routes
spec:
  matches:
  - name: metrics
    pathRegex: "/metrics"
    methods:
    - GET
  - name: health
    pathRegex: "/ping"
    methods: ["*"]

---
kind: TrafficTarget
metadata:
  name: path-specific
  namespace: default
spec:
  destination:
    kind: ServiceIdentity
    name: my-api
    namespace: default
  rules:
  - kind: HTTPRouteGroup
    name: the-routes
    matches:
    - metrics
  sources:
  - kind: ServiceIdentity
    name: my-service
    namespace: gateway-api-example-ns1

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: my-route
  namespace: gateway-api-example-ns2
spec:
  parentRefs:
  - kind: ServiceIdentity
    name: my-service
    namespace: gateway-api-example-ns1
  rules:
  - backendRefs:
    - name: my-api
      port: 8080
  matches:
    - path:
        value: "/metrics"
      method: GET
    - path:
        value: "/ping"

Notes:

• Investigate if parentRefs could relate to proposed ServiceIdentity, Gateway spec seems loose enough
• How would service to service authorization be implemented with Gateway spec
• In the instance that Gateway spec is used to control a Kubernetes ingress controller, how do you determine if HTTPRoute should configure a gateway or east west service mesh traffic

[mikemorris]

For the proposed switch to HTTPRoute, it was discussed in last community meeting how the owner of a service should be configuring an HTTPRoute to allow incoming connections - we should clarify how the parentRefs and backendRefs fields would be used, as this is a bit of the inverse of how HTTPRoute is used in the Gateway API, where routes bind to a Gateway with a parentRef, and require a ReferencePolicy for a backendRef to direct traffic to a service in a different namespace.

Multiple backendRefs can be defined on an HTTPRouteRule, and each backendRef includes a weight which could be used for balancing traffic e.g. between v1 and v2 k8s services of a single logical service (this could point to separate IdentityBinding objects if we don't want to introduce weighting within the IdentityBinding?)

An HTTPRoute can also define multiple parentRefs, which I think would offer sufficient granularity to be used to allow traffic from, e.g. { kind: IdentityBinding, name: fooBinding, namespace: api } or simply from a specific namespace.

We've focused mostly on HTTPRoute for directing L7 traffic, would we also want TCPRoute for L4 traffic, or just use the default behavior of and HTTPRoute routing all traffic when the scope is not narrowed with an HTTPRouteMatch?

[keithmattix]

I have some questions about how we need to handle hostname for E/W traffic; I'm not sure that field makes total sense in a mesh context.

Additionally, I'm currently of the opinion that parentRef should only be a reference to the mesh and that identity shouldn't come into play here. In other words, I think traffic routing (HTTPRoute) and traffic policy (TrafficTarget) should stay separate

[howardjohn]

I have some questions about how we need to handle hostname for E/W traffic; I'm not sure that field makes total sense in a mesh context.

Drive by comment from an Istio developer (speaking without consensus from Istio community) - IMO it should work like this:

A Route is bound to a Service (TODO how exactly this fits in the API). What this means is "All requests to this Service should use this route". This should be identified by the Service IP.

Once we have a route selected, we follow the standard route matching. For HTTP hostname, that means we can match additional Host headers. For example,

kind: HTTPRoute
parentRef:
- Service foo # TODO on syntax
hostname: [bar.com]
...

This would mean curl foo would get a 404, and curl foo -H "Host: bar.com" would match.

Similar applies to SNI for TLSRoute.

Note that this means that using Hostname with mesh would likely be pretty rare. It could maybe be useful for a service like, say, nginx that is doing its own vhost routing so you would call it with different Host headers, but definitely an exception.

This is not what Istio is doing here, but I want to change it. Today Istio doesn't do any IP matching at all for HTTP, so everything is host header based. This has caused various issues described in https://docs.google.com/document/d/1NAccj8WyjBXOUsMdOHW9sWW6PeCnrIWpc3muBHd4Cs8/.

[keithmattix]

I never thought about binding the route to service, but the more I think about it, the more it makes sense... I also agree that too much reliance on host headers likely leads to an inconsistent experience for users. SMI still has yet to answer the question "what is a service" in the spec, and the future parentRef API implementation may give us that.

The natural question though is how to handle headless services, but I imagine we could develop a consistent spec for how mesh implementations should translate the headless service to its endpoint IPs for matching purposes.

[howardjohn]

I only mentioned Service, but I think binding to "workload"/Pod also makes sense. For example, I could want to apply some route to all requests to a specific pod (again, matched by IP). Now no one really wants to select a single Pod probably, but there are ways to group pods (Headless service is a great example, or even something that says "Workloads of this Service" or some of the higher constructs like Deployment, etc. Or a custom type - lot of options).

FWIW Today Istio doesn't even really handle traffic directly to pods (except in headless service, and even then pretty poorly), so all of this is pretty aspirational from our perspective.

[keithmattix]

Yeah and on top of all of that, we need to make sure our solution handles the Gateway -> Mesh case smoothly. What does it look like for an HTTPRoute that has a parentRef of Gateway and a backendRef to service A and another HTTPRoute with a parentRef of Service A? Do we allow nested traffic splitting, one policy at ingress and then another policy at the mesh level?

Lots of fun questions to tackle 😄

[sftim]

BTW @nicholasjackson the issue title has a wording error; it should start with “Deprecation”.