Proposal - How to run Kyma on top of Knative

[sjanota]

Created on 2018-12-05 by Szymon Janota (@sjanota)

Decision log

Name	Description
Title	How to run Kyma with Knative
Ultimate decision maker(s)	Kyma Council
Due date	2018-12-20
Input provider(s)	@kubadz @Abd4llA @sjanota
Group(s) affected by the decision	WG Knative Integration
Decision type	Binary
Earliest date to revisit the decision	2019-01-06
Affected decisions	None

Context

One of the features that is currently being implemented in Kyma is the integration with Knative. It must be established how to run Kyma and Knative in a single Kubernetes cluster.
The following factors need to be taken into consideration:

• The way Kyma has to works without Knative has to remain unchanged
• How difficult is it to upgrade to newer versions of Knative
• How difficult is it to extend the solution and integrate more Knative components
• How difficult is it to manage additional components in the running clusters
• How does the solution fit into the overall Kyma architecture

With these criteria in mind, solutions must be chosen for these problems:

1. How to run Kyma and Knative on a single Kubernetes cluster and have both tools fully funtional.
2. How to fit Knative into the Kyma installation process.
3. How to version Knative within Kyma.

Each of these problems can be solved several ways. The possible solutions are discussed in the next section.

How to run Kyma and Knative on a single Kubernetes cluster and have both tools fully functional

Kyma and Knative are both installed in their own Namespaces, so they are not conflicting with each other directly. The only component they share is Istio. Kyma introduces several changes to the default Istio installation, whereas Knative does not modify Istio in an intrusive manner. This allows Knative to work with the Kyma version of Istio without any problems. Knative runs on top of Istio, but uses its own ingress gateway, while Kyma uses the Istio default Gateway. The Gateways are configured so that they can be accessed under different Services. Both ingress gateways and their configurations are installed in the istio-system Namespace. There are two approaches to making both tools accessible:

1. Do not change the network configuration. Acces Kyma and Knative through their respective default ingress gateways. This solution was proved to be working by @kubadz in (POC) API exposure using istio gateway controller with knative installed kyma#1643

   • Advantages:
     • No changes in any of the systems
   • Disadvantages:
     • Two ingress gateways have to be maintained. This means more complex DNS management, deployment and debugging

2. Route all traffic through the Knative ingress gateway, abandon the default Kyma ingress gateway. This solution was proved to be working by @sjanota in (POC) knative serving for API exposure using knative gateway controller kyma#1598.

   • Advantages:

     • Easier network management thanks to having a single point of entry in the cluster
     • Easier management of TLS certificates
     • Closer integration with Knative

   • Disadvantages:

     • Requires modifying Knative and Kyma deployments
     • Knative must to be tightly integrated with Kyma installation process

How to fit Knative into Kyma's installation process

The "batteries included" principal is one of the most important ones for Kyma. That's why Knative integration needs to be available without requiring the users to install additional components on their own. Because of its dependency on Istio, Knative must be installed by the Kyma installer after Istio.

Knative team does not provide Helm charts for their releases. Instead, they provide yaml files containing all of the resources. Possible ways to integrate Knative installation into the Kyma installer:

1. Create a Kubernetes job which installs Knative from the yaml file and verifies the installation, just like Helm would do. This job also adds required customizations.

   • Advantages:

     • Easier upgrades of Knative

   • Disadvantages:

     • This is not how installer is intended to be used
     • Yet another Bash script we need to maintain. Goes against the general rule to avoid Bash scripts whenever possible

2. Create a chart which embeds Knative yaml files. Required customisations are added as generic parameters to the chart or changed using existing tools such as overrides or the istio-patch.

   • Advantages:

     • Easier extensibility
     • More in line with the installer
     • No additional tool introduced to the system

   • Disadvantages:

     • More difficult upgrades until Knative introduces a helm chart

How to version Knative within Kyma

1. Version URLs to specific versions of Knative.

   • Advantages:

     • Knative yaml file are not maintained in Kyma

   • Disadvantages:

     • Kyma installation depends on the availability of the external files
     • Cannot be used if Knative is embedded in a chart

2. Version all yaml files within the Kyma repository

   • Advantages:

     • Better control over what's being installed

   • Disadvantages:

     • These files become a part of Kyma and thus need to be maintained

Proposed solution

Proposed solution is to use the Knative ingress gateway as it allows for a better integration with Knative and makes cluster management easier. The other ingress gateway will still be deployed, but will remain unexposed and unused. As this solution requires changes in both Knative and Kyma deployments, extensibility is the main concern when thinking about installation. Therefore the chosen approach is to embed Knative in the chart. Packing Knative into a chart format eliminates the possibility to version only URLs. Instead, all of the required yaml files must be added to Kyma.

Installing Kyma with Knative is optional. Installation can be enabled with the global.knative global installer override paired with a boolean value. Kyma without Knative installed will continue working as before.

Decision

Approved

Status

Proposed on 2018-12-06

Consequences

Advantages:

• Debugging and maintanance of installation with Knative is easier as Kyma and Knative use the same entry point.
• Extending Kyma-Knative integration will be easier than with raw yaml files. This is important from the eventing and serverless perspective.
• We are independent of Knative release cycle
• We can contribute helm charts to Knative

Disadvantages:

• Different component works as entry point in normal operations and with Knative. This difference may disappear as Knative will use Istio ingress gateway in new releases (Allow config ingress gateway knative/serving#2434).
• Upgrading Knative is troublesome as we need to migrate customizations from one version of the yaml files to another.
• We need to maintain Knative charts in Kyma

[sjanota]

Implementation: kyma-project/kyma#1845

[joek]

Is it possible to contribute the helm chart as part of knative (or some other project)? To at least get some help from the community during update? It feels like it is a lot of work to manage the helm charts by our own.

Second question: Shouldn't come knative from the cloud provider like gke?

[sjanota]

Is it possible to contribute the helm chart as part of knative (or some other project)? To at least get some help from the community during update? It feels like it is a lot of work to manage the helm charts by our own.

Yes, we'd like to contribute the chart to Knative, but a lot more changes need to be made to make it usable by community. We're not there yet, unfortunately. I think of a followup to work on it when this proposal is accepted and solution is merged.

Second question: Shouldn't come knative from the cloud provider like gke?

Only serving is available as add-on on GKE and its's in early access. Despite that the situation is the same as for Istio. We have to deliver our version of it ("Batteries included"), but as a user I should also be allowed to use my existing version of Knative. With careful tampering with installation configuration it is possible now. The only thing that needs to be done is to remove Istio and Knative from component list in Installation CR and install them independently. I haven't tested it though.

[sjanota]

The proposal is approved.