Install Kyma on AKS

[PK85]

Description

Part of our release theme: Generalize deployment and operational aspects so that also Microsoft AKS can be supported.

Acceptance Criteria

• prepare installation instruction (use GKE installation guide as a template)
• make sure kyma works (installation process and test are green)
• create unified documentation about GKE and AKS with wildcard DNS

Attachments

• Kyma on GKE installation guide, see:
  • https://github.com/kyma-project/kyma/blob/master/docs/kyma/docs/032-inst-gke-installation.md
  • https://kyma-project.io/docs/master/root/kyma#installation-install-kyma-on-a-gke-cluster
• Kyma on GKE with wildcard DNS installation guide
  • https://github.com/kyma-project/kyma/blob/master/docs/kyma/docs/036-inst-xip.io-domain.md
  • https://kyma-project.io/docs/master/root/kyma#installation-install-kyma-on-a-gke-cluster-with-wildcard-dns

[adamwalach]

There are two issues with running Kyma on AKS.

1. During installation the readiness probe on service-catalog-apiserver fails with 403
   This is a known issue: Catalog controller crashing on 1.9.6  Azure/AKS#417
   A bit more description can be found here: Readiness probe failed: HTTP probe failed with statuscode: 403 voyagermesh/voyager#1296
   Suggested solution is to allow "system:unauthenticated" group to access ["/healthz", "/healthz/*"] endpoints: [stable/prometheus-adapter] 403 in readiness probe in AKS helm/charts#10222 (comment). This is not perfect cause this policy is too open
2. Kyma UI cannot access Pods, Services and all other resources that are fetched through K8S dashboard.
   This issue has two causes:

• istio network setup
  It is also known: Internal apiserver calls blocked by Istio-- only on AKS Azure/AKS#561.
  Directly after installation we've got error 403. When we added global.proxy.excludeIPRanges
  to the installation yaml
  
  the error changed to 503.
• build-in k8s dashboard is exposed using http instead of https.
  It causes the 503 error. This port cannot be changed in the SVC, we tried to modify that and it always preserves original values. The only solution is to add configuration parameter to Kyma so we can set url for K8s dashboard to a proper value during installation. Or we could stop using the dashboard as a datasource.

[adamwalach]

Logs from tests:

• standard installation test-on-aks-standard.log
• with xip enabled test-on-aks-xip.log
  Overall test results looks good, there are some failed checks but they are caused by the k8s dashboard issue.

[adamwalach]

Follow-ups:

• Description of the installation process on gcp is not clear #2273
• Incomplete steps in the xip.io documentation #2403

[adamwalach]

Stats:
AKS cluster creation ~ 6min
Kyma installation ~ 18min
AKS cluster deletion - 8min

[PK85]

Findings:

1. Kubectl configuration duplicated, see:

• point 2 - https://github.com/kyma-project/kyma/blob/master/docs/kyma/docs/04-05-aks-installation.md#prepare-the-aks-cluster
• point 2 - https://github.com/kyma-project/kyma/blob/master/docs/kyma/docs/04-05-aks-installation.md#deploy-kyma

2. point 2 - https://github.com/kyma-project/kyma/blob/master/docs/kyma/docs/04-05-aks-installation.md#prepare-the-aks-cluster mention that if error Error from server (MethodNotAllowed): error when creating "my-kyma.yaml": the server does not allow this method on the requested resource (post installations.installer.kyma-project.io) will be displayed, apply again before going to point no 2
3. point 5 - https://github.com/kyma-project/kyma/blob/master/docs/kyma/docs/04-05-aks-installation.md#deploy-kyma "To watch the installation progress, run:" is not enough, whole "Configure DNS for the cluster load balancer" section requires some things to be up and running. As you said is-isntalled script could be not available in release mode installation. Let me know what options we have.