Enable setting headers on all routes with dynamic service values

[erwbgy]

Using Linkerd with Contour requires setting the l5d-dst-override header on every request to the destination service hostname and port - for example: kuard.default.svc.cluster.local:80.

I agree with @jpeach's comment on #2089 that Contour shouldn't have special support for Linkerd headers - like Ambassador's add_linkerd_headers option - but I think it should be possible to support it more generically including @stevesloka's idea of adding default headers.

My proposal is to enable request and response headers specified in the config file to be set on all routes with dynamic values for namespace, service name and service port:

1. Add support for %CONTOUR_NAMESPACE%, %CONTOUR_SERVICE_NAME% and %CONTOUR_SERVICE_PORT% dynamic variables and have these expanded like Envoy dynamic variables are in Support specific dynamic request headers #3234. (The CONTOUR_ prefix is used so that there is no possibility of a clash with a future Envoy dynamic variable.)

This would allow a header to be set in an HTTPProxy object:

      requestHeadersPolicy:
        set:
        - name: X-Envoy-Hostname
          value: "%HOSTNAME%"
        - name: l5d-dst-override
          value: "%CONTOUR_SERVICE_NAME%.%CONTOUR_NAMESPACE%.svc.cluster.local:%CONTOUR_SERVICE_PORT%"

If any of these variables are used where they can't be expanded then they are just passed through literally. For example if %CONTOUR_SERVICE_NAME% is used at the route level then it is not expanded and becomes %%CONTOUR_SERVICE_NAME%% when passed to Envoy. This avoids any Envoy configuration errors due to bad dynamic variables.

2. Allow default request and response headers added to every route to be set in the Contour config file:

headers:
  set-request-headers:
  - "X-Envoy-Hostname=%HOSTNAME%"
  - "l5d-dst-override=%CONTOUR_SERVICE_NAME%.%CONTOUR_NAMESPACE%.svc.cluster.local:%CONTOUR_SERVICE_PORT%"
  set-response-headers:
  - "X-Envoy-Response-Flags=%RESPONSE_FLAGS%"

This is the same as if the above requestHeadersPolicy was added to every route service. These headers would need to be set at the service level so that the service name and port are available.

With these changes the Linkerd header requirement is met but without anything specific for Linkerd, enabling similar requirements to be met in future.

Using a mutating webhook would be another way to get the l5d-dst-override header set on every request, but this makes it more difficult to use GitOps solutions like ArgoCD or Flux as the object changes, so is not great in practice. The webhook is also another component to operate and maintain.

If this design sounds reasonable then I have a pull request that implements these changes, that can be the basis of further discussion.

[youngnick]

I like the idea of Contour being able to interact with service meshes, that sounds awesome.

However, I think there's two things to add as part of this:

• add %CONTOUR_NAMESPACE%, %CONTOUR_SERVICE_NAME% and %CONTOUR_SERVICE_PORT% interpolation variables to the header expansion list.
• add default header policy to the Contour configuration file, which will be applied by default across all routes added.

I think the first is straightforward and worth doing regardless, but I think that the second needs a bit more thought.

@erwbgy, could you split your PR into two pieces?

The things I'm thinking for the second part are:

• Do we apply the default to all routes? Whatever the config method? So, for Ingress and HTTPRoute (from service-APIs) as well as HTTPProxy?
• Do we add a way to disable or wait for people to ask?
• What happens if someone defines a header policy in their HTTPProxy that conflicts with the globally-set one? That is, what happens if someone tries to set the same header, or tries to unset the default one?

[artificial-aidan]

This got added for HTTPProxy objects, but is there any plan for adding this to Ingress objects too? Just having it on HTTPProxy objects means you still can't enable linkerd on contour.

[youngnick]

Oh, I thought that Linkerd would work with HTTPProxy objects from the initial request. There are currently no plans, but please feel free to open an issue, and we can discuss how we could make it work. (It would be a lot of annotations, which we are generally reluctant to do, but we can definitely discuss).

[artificial-aidan]

#4005