Add the Nutanix CSI driver

[xmudrii]

What this PR does / why we need it:

This PR:

• Add the Nutanix CSI driver addon. The addon is deployed manually, on-demand, by enabling the csi-nutanix embedded addon
  • The reason for not using .cloudProvider.external (like for other providers) is that Nutanix doesn't have an external CCM, so enabling external renders the cluster useless (there's no CCM to initialize nodes, so nodes remain tainted)
• Add the default StorageClass for the Nutanix CSI driver. The StorageClass can be deployed by enabling the default-storage-class embedded addon
  • Note: this StorageClass might get changed before the final release to support additional features
  • This addon can take the following parameters in order to customize the StorageClass: storageContainer (default is Default), fsType (default is xfs), isSegmentedIscsiNetwork (default is false)
• Add new environment variables used to provide endpoint and credentials for Prism Element: NUTANIX_PE_ENDPOINT, NUTANIX_PE_USERNAME, NUTANIX_PE_PASSWORD. Prism Element access is required by the Nutanix CSI driver
• Rename the NUTANIX_ALLOW_INSECURE environment variable to NUTANIX_INSECURE
• Remove the allow_insecure variable from Terraform configs for Nutanix in favor of the NUTANIX_INSECURE environment variable
• Forbid enabling .cloudProvider.external for Nutanix clusters because Nutanix doesn't have an external CCM that would initialize nodes
• Install and enable iscsid and NFS on Nutanix machines (required by the CSI driver)
• Update machine-controller to v1.42.0
• Fix a bug with the addons applier applying all files when addons path is not provided
• Fixes control plane tolerations in Azure CCM and CSI addons (node-role.kubernetes.io/master doesn't have a value)

However, this PR doesn't implement the snapshot controller and functionalities. This will be tackled in a follow-up. It's also yet to test this on newer Kubernetes versions.

The following KubeOneCluster manifest can be used on Nutanix clusters (params are optional and can be omitted if you want to use default values):

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: 1.21.8
cloudProvider:
  nutanix: {}
addons:
  enable: true
  addons:
  - name: "csi-nutanix"
  - name: "default-storage-class"
    params:
      storageContainer: "Default"
      fsType: "xfs"
      isSegmentedIscsiNetwork: "false"

The following manifest has been used for testing:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pvc-ntnx
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ntnx-csi
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP
    volumeMounts:
      - mountPath: /var/lib/www/html
        name: csi-data-ntnx
  volumes:
  - name: csi-data-ntnx
    persistentVolumeClaim:
      claimName: csi-pvc-ntnx
      readOnly: false

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
xref #1726

Does this PR introduce a user-facing change?:

* Add the Nutanix CSI driver addon. The addon is deployed manually, on-demand, by enabling the `csi-nutanix` embedded addon (see the PR description for more details and examples)
* Add the default StorageClass for the Nutanix CSI driver. The StorageClass can be deployed by enabling the `default-storage-class` embedded addon (see the PR description for more details and examples)
* Add new environment variables used to provide endpoint and credentials for Prism Element: `NUTANIX_PE_ENDPOINT`, `NUTANIX_PE_USERNAME`, `NUTANIX_PE_PASSWORD`. Prism Element access is required by the Nutanix CSI driver
* Rename the `NUTANIX_ALLOW_INSECURE` environment variable to `NUTANIX_INSECURE`
* Remove the `allow_insecure` variable from Terraform configs for Nutanix in favor of the `NUTANIX_INSECURE` environment variable
* Forbid enabling `.cloudProvider.external` for Nutanix clusters because Nutanix doesn't have an external CCM that would initialize nodes
* Install and enable iscsid and NFS on Nutanix machines (required by the CSI driver)
* Update machine-controller to v1.42.0
* Fix a bug with the addons applier applying all files when addons path is not provided
* Fixes control plane tolerations in Azure CCM and CSI addons (`node-role.kubernetes.io/master` doesn't have a value)

[kubermatic-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[kubermatic-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [xmudrii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubermatic-bot]

LGTM label has been added.

Git tree hash: 14604f0ca71417e049c6b6f111dc3b7cac2567ad