[Explore] Replaceable networking component

[danail-branekov]

Currently Korifi creates contour HTTP proxies for app workloads. However, we get feedback from people that they would like to use istio instead of contour.

We have the idea to abstract networking via the Gateway API, however we know that it might be not that easy as websockets are not supported.

However, it is still worth exploring whether we could leverage the Gateway API to achieve what Korifi is capable of. As a result of this explore we should be able to answer the following questions:

• Can Korifi create Gateway API objects instead of contour ones?
• Is there something on top of Gateway API we need to implement the current behaviour on Contour? Can we put that extra bit in an optional component?
• Bonus: could we implement a parallel component that translates GatewayAPI objects into Istio ones?
• Other bonus: can we replace the Countour HTTPProxy we currently use for the API with a more generic Ingress object? This would relegate the Contour dependency to the CFRoute reconciler alone.

[tcdowney]

Sharing some bits from this Slack conversation for additional context:

1. The ingress controllers that support Gateway API resources reconcile them directly so we won't have an opportunity to mutate any controller-specific resources (e.g. Contour HTTPProxy, Istio VirtualService) since those either won't exist or their lifecycle is not controlled by us.
2. Eventually I think we may be able to only make Gateway API resources, but it might not have the features we need on a soon enough timeframe. So maybe we need to do something like what we've done with our "pluggable" build/run/task controllers and have things like a contour-cfroute-reconciler, istio-cfroute-reconciler, openshift-ingress-cfroute-reconciler, gateway-api-cfroute-reconciler, etc. in the meantime. I think the CFRoute has all of the information needed for this and since we eventually want to converge on Gateway API it's unlikely we need yet another intermediate route resource.
3. Back in the cf-for-k8s days we (well the CF Networking Program) was experimenting with making the cf-k8s-networking controller more flexible. It produced Istio resources by default, but you can see on the various contour- branches how they were making it produce other resources. Example: https://github.com/cloudfoundry/cf-k8s-networking/tree/contour-ingress/routecontroller/controllers/networking

[georgethebeatle]

This branch contains an experimental implementation that separates the routing controller in its own replaceable component. It also attempts to reimplement the contour routing using istio, but is not yet complete.

Here are the main findings so far:

• As we could not make istio route tls we have introduces a protocol value in the api helm chart so that we can make the api available on http. Exploring TLS routing is a next step, so this flag may not be required in the end.
• Istio gateways do not like localhost as a hostname so we are using api.vcap.me for the api and *.vcap.me for the apps. Note that this means we should not push an app called api, but that is ok for the spike.
• Istio needs to inject its sidecars in the app pods and these sidecars need additional capabilities so we need to switch off pod security standards enforcement. An enforcePodSecurityStandards value has been introduced to the controllers helm chart.
• Istio needs injection to be enabled in all cf space namespaces, which are created by the user. This can be achieved by making the cf space controller label the namespace on creation, but this couples it with istio, which is bad. We can avoid this coupling by adding an extraLables value and making the cf space controller apply those labels to cf space namespaces, but it is still not ideal.
• A better approach to allow sidecar injection in cf space namespaces is to deploy istio with modified injection configuration configmap istio-system/istio-sidecar-injector:

alwaysInjectSelector:
  - matchExpressions:
    - key: korifi.cloudfoundry.org/process-type
      value: web
neverInjectSelector:
  - matchExpressions:
    - key: kpack.io/build
      operator: Exists

This should make istio inject sidecars in apps, but not in kpack builds without adding any special labels to the actual namespace. However we haven't manage to make it work on kind so far

• The istio-routing dir contains the istio objects that the cf route controller needs to start creating in order to support istio. Potentially we can have a single gateway for *.vcap.me that is shared for all apps and multiple virtualsevices, one for each app. These configurations are working for http only. We need to make them work with TLS eventually, but that should be doable and probably does not hide any significant edges.

[georgethebeatle]

This commit contains a working prototype of the istio router! However it only works with plain http for both the api server and apps, which is not ideal. Next steps: make it work with tls.

All changes that need to be made in korifi to accommodate the new networker are more or less described above. We have decided to take the easy route and enable istio sidecar injection by labeling the cf space namespaces with the special istio label. In the productive world we will have to allow the user to specify custom labels that are going to be applied to each cf space namespace. This is generic enough to not couple us with istio and might eventually be useful for other stuff. Another upside of that approach is that we are going to have no requirements of how istio is installed, which gives us extra flexibility with users who have their service mesh already setup.

[georgethebeatle]

This branch contains the completed prototype that enables tls for both the api and apps. At this point we consider the explore to be done and are ready to write up the proposal.

[danail-branekov]

Proposal doc: https://docs.google.com/document/d/117buRig9YUm4z2njK5gdnqxBldn5fTZNCM_QTUy28lw/edit?usp=sharing