tvk-preflight is a kubectl plugin which checks if all the pre-requisites are met before installing Triliovault for Kubernetes application in a Kubernetes cluster.
This plugin automatically generates log file for each preflight run(preflight-<date>T<time>.log
)
and cleanup(preflight_cleanup-<date>T<time>.log
) which can be used to get more information around operations being performed by this plugin.
krew
[optional] - kubectl-plugin manager. Install from herekubectl
- kubernetes command-line tool. Install from herebash
(>=v3.2.x) should be present on system
For openshift environments, if kubectl
is not installed and oc
binary is installed on host machine, then oc
binary
can be used to perform kubectl
operation by creating symlink with -
sudo ln -s /usr/local/bin/oc /usr/local/bin/kubectl
Note:
oc
binary path can found by executingwhich oc
- To delete/remove symbolic links use either
unlink
orrm
command -
unlink /usr/local/bin/kubectl
Supported OS:
- linux/amd64
- linux/x86
- linux/arm
- linux/arm64
- darwin/amd64
- darwin/arm64
Preflight plugin performs checks on system where this plugin is installed and few checks are performed on the K8s cluster where current-context of kubeconfig is pointing to.
Whenever a preflight check is performed, a 6-length smallcase alphabet
UID
is generated for that particular preflight check. This UID
is the value of the label preflight-run
which is set on every
resource created during the preflight check. Also the UID
is the suffix of name of every resource created during preflight check.
This UID
is particularly useful to perform cleanup of resources created during a particular preflight check.
The following checks are included in preflight:
-
check-kubectl
- Ensures kubectl utility is present on system. This check is skipped if--in-cluster
flag is enabled. -
check-cluster-access
- Ensures preflight can access the remote target cluster. -
check-helm-version
-- Ensures helm[version>=v3.x.x] utility is present on system and pointed to the cluster
- Aborts successfully for Openshift cluster
- This check is skipped if
--in-cluster
flag is enabled.
-
check-kubernetes-version
- Ensures minimum Kubernetes version >= 1.19.x -
check-kubernetes-rbac
- Ensures RBAC is enabled in cluster -
check-csi
-- Checks if the following CSI apis are present on the cluster -
- "volumesnapshotclasses.snapshot.storage.k8s.io"
- "volumesnapshotcontents.snapshot.storage.k8s.io"
- "volumesnapshots.snapshot.storage.k8s.io"
- If not present, creates the missing CSI apis as per the k8s server version. If k8s server version is 1.19, installs the above CSI apis that support v1beta1 version. If k8s server version is 1.20+, installs the above CSI apis that support both v1 and v1beta1 version. Also, if volumesnapshot CRDs don't exist, any provided volume snapshot class will be overridden with default value.
- Checks if the following CSI apis are present on the cluster -
-
check-pod capability
-- Ensures pods with the TVK capabilities can be provisioned in the cluster.
- The capability matrix for TVK described in the documentation is used to generate three pods, each identified as pod-capability-${INDEX}-${UID}.
- Ensures pods with the TVK capabilities can be provisioned in the cluster.
-
check-storage-snapshot-class
-- Ensures provided storageClass is present in cluster
- Provided storageClass's
provisioner
[JSON Path:storageclass.provisioner
] should match with provided volumeSnapshotClass'sdriver
[JSON Path:volumesnapshotclass.driver
] - If volumeSnapshotClass is not provided then, volumeSnapshotClass which satisfies condition
[i]
will be selected. If there's are multiple volumeSnapshotClasses satisfying condition[i]
, default volumeSnapshotClass[which has annotationsnapshot.storage.kubernetes.io/is-default-class: "true"
set] will be used for further pre-flight checks. If no volumeSnapshotClass matching with the storage class's provisioner is found, then a volumeSnapshotClass withdriver
as storageClass'sprovisioner
anddeletionPolicy
asDelete
will be created with a name that starts withpreflight-generated-snapshot-class
and has a random suffix. - If volumeSnapshotClass is provided and matches with storage class provisioner, only then that volumeSnapshotClass will be used for further operations, otherwise preflight will fail with not found error.
- Provided storageClass's
- Ensures at least one volumeSnapshotClass is marked as default in cluster if user has not provided volumeSnapshotClass as input.
- Ensures provided storageClass is present in cluster
-
check-dns-resolution
-- Ensure DNS resolution works as expected in the cluster
- Creates a new pod (dnsutils-${UID}) then resolves kubernetes.default service from inside the pod
- Ensure DNS resolution works as expected in the cluster
-
check_volume_snapshot
-- Ensure Volume Snapshot functionality works as expected for both mounted and unmounted PVCs
- Creates a Pod and PVC (source-pod-${UID} and source-pvc-${UID}).
- Creates Volume snapshot (snapshot-source-pvc-${UID}) from the mounted PVC(source-pvc-${UID}).
- Creates volume snapshot of unmounted PVC(source-pvc-${UID} [deletes the source pod before snapshotting].
- Restores PVC(restored-pvc-${UID}) from volume snapshot of mounted PVC and creates a Pod(restored-pod-${UID}) and attaches to restored PVC.
- Restores PVC(unmounted-restored-pvc-${UID}) from volume snapshot from unmounted PVC and creates a Pod(unmounted-restored-pod-${UID}) and attaches to restored PVC.
- Ensure data in restored PVCs is correct[checks for a file[/demo/data/sample-file.txt] which was present at the time of snapshotting].
- If
check-storage-snapshot-class
fails then,check_volume_snapshot
check is skipped.
- Ensure Volume Snapshot functionality works as expected for both mounted and unmounted PVCs
After all above checks are performed, cleanup of all the intermediate resources created during preflight checks' execution is done.
-
Add TVK custom plugin index of krew:
kubectl krew index add tvk-plugins https://github.com/trilioData/tvk-plugins.git
-
Installation:
kubectl krew install tvk-plugins/tvk-preflight
-
Upgrade:
kubectl krew upgrade tvk-preflight
-
Removal:
kubectl krew uninstall tvk-preflight
- List of available releases: https://github.com/trilioData/tvk-plugins/releases
- Choose a version of preflight plugin to install and check if release assets have preflight plugin's package[preflight.tar.gz]
- Set env variable
version=v1.x.x
[update with your desired version]. Ifversion
is not exported,latest
tagged version will be considered.
- Bash or ZSH shells
(
set -ex; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
if [[ -z ${version} ]]; then version=$(curl -s https://api.github.com/repos/trilioData/tvk-plugins/releases/latest | grep 'tag_name'| awk '{print $2}' | tr -d '",'); fi &&
echo "Installing version=${version}" &&
package_name="preflight_${version}_${OS}_${ARCH}.tar.gz" &&
curl -fsSLO "https://github.com/trilioData/tvk-plugins/releases/download/"${version}"/${package_name}" &&
tar zxvf ${package_name} && sudo mv preflight /usr/local/bin/kubectl-tvk_preflight
)
Verify installation with kubectl tvk-preflight --help
NOT SUPPORTED
Note for Dark Site Installation
-
For using the
local-registry
flag, it is mandatory to havebusybox:latest
anddnsutils:1.3
images (with the same tags) to be there in the private registry.Steps for pushing images to local registry
- Pull the images (dnsutils:1.3 & busybox) to local machine.
- use the following command to push it to the local registry
docker push <local registry/image>
- Example:
docker push localhost:5000/busybox
kubectl tvk-preflight [sub-command] [flags]
The preflight binary has three common flags to both the subcommands.
Parameter | Shorthand | Default | Description |
---|---|---|---|
--namespace | -n | default | Namespace of the cluster in which resources will be created, preflight checks will be performed or resources will cleaned. Default is 'default' namespace of the cluster (Optional) |
--kubeconfig | -k | ~/.kube/config | kubeconfig file path (Optional) |
--log-level | -l | INFO | Logging level for the preflight check and cleanup. Logging levels are FATAL, ERROR, WARN, INFO, DEBUG (Optional) |
--config-file | -f | yaml file path to provide inputs for run and cleanup subcommand (Optional) | |
--in-cluster | -i | false | Skip kubectl and helm binary check if running inside a container. |
#####Note: --in-cluster flag should only be set when running inside a container where kubectl and helm checks are not required.
The inputs for running preflight checks and cleanup can be provided through a single file. The format of data in a file should be according to the below example:
run:
storageClass: <storage-classs>
snapshotClass: <snapshot-class>
namespace: <perform preflight checks in the given namespace>
kubeconfig: <kubeconfig file path>
inCluster: <bool>
serviceAccount: <service-account>
localRegistry: <complete path of the registry to pull the images from>
imagePullSecret: <Name of the secret while pulling images from the local registry>
cleanupOnFailure: <Boolean. If true cleans the preflight resources after a failed preflight run>
pvcStorageRequest: <Storage request value of PVC for volume snapshot check>
resources:
requests:
memory: <pod memory request for snapshot check, e.g 64Mi>
cpu: <pod cpu request for snapshot check, e.g 250m>
limits:
memory: <pod memory limit for snapshot check, e.g 128Mi>
cpu: <pod cpu limit for snapshot check, e.g 500m>
cleanup:
namespace: <clean preflight in a particular namespace>
kubeconfig: <kubeconfig file path>
logLevel: <specify logging level for cleanup>
uid: <specify this field to clean preflight resources of a particular run>
- The uid field, if specified, will clean the preflight resources with the given uid in the given namespace. Otherwise, all the preflight resources present on the system in the given namespace will be cleaned.
- User can override the values given in file using CLI flags.
- The input fields should be present in the correct hierarchical order. An incorrect key or input field will result in an error and preflight checks will not performed.
Run a preflight check with predefined values using a sample file. Download the file using below commands:
By wget
wget https://github.com/trilioData/tvk-plugins/tree/main/docs/preflight/sample_input.yaml
By curl
curl https://github.com/trilioData/tvk-plugins/tree/main/docs/preflight/sample_input.yaml
- With
--namespace
:
kubectl tvk-preflight [sub-command] [sub-command flags] --namespace <namespace of the cluster>
By using shorthand notation:
kubectl tvk-preflight [sub-command] [sub-command flags] -n <namespace of the cluster>
- With
--kubeconfig
:
kubectl tvk-preflight [sub-command] [sub-command flags] --kubeconfig <kubeconfig file path>
By using shorthand notation:
kubectl tvk-preflight [sub-command] [sub-command flags] -k <kubeconfig file path>
- With
--log-level
:
kubectl tvk-preflight [sub-command] [sub-command flags] --log-level <logging level>
By using shorthand notation:
kubectl tvk-preflight [sub-command] [sub-command flags] -l <logging level>
- With
--config-file
kubectl tvk-preflight [sub-command] [sub-command flags] --config-file <yaml input file path>
By using shorthand notation:
kubectl tvk-preflight [sub-command] [sub-command flags] -f <yaml input file path>
There are two subcommands to the preflight binary:
- run: To perform preflight checks
- cleanup: To clean the resources generated during failed preflight checks
run sub-command performs the actual preflight checks on the system and on the kubernetes cluster where the system's kubeconfig is pointing to in the given namespace.
Parameter | Default | Description |
---|---|---|
--storage-class | Name of storage class being used in k8s cluster (Needed) | |
--volume-snapshot-class | Name of volume snapshot class being used in k8s cluster (Optional) | |
--local-registry | Name of the local registry from where the images will be pulled (Optional) | |
--image-pull-secret | Name of the secret for authentication while pulling the images from the local registry (Optional) | |
--service-account | Name of the service account (Optional) | |
--cleanup-on-failure | false | Deletes/Cleans all resources created for that particular preflight check from the cluster even if the preflight check fails. For successful execution of preflight checks, the resources are deleted from cluster by default (Optional) |
--requests | cpu=250m,memory=64Mi | Pod cpu and memory request for DNS and volume snapshot check. Memory and cpu values must be specified in a comma separated format. (Optional) |
--limits | cpu=500m,memory=128Mi | Pod cpu and memory limit for DNS and volume snapshot check. Memory and cpu values must be specified in a comma separated format. (Optional) |
--pvc-storage-request | 1Gi | PVC storage request for performing volume snapshot check. (Optional) |
--node-selector | Node selector labels for scheduling pods on a set of particular nodes of a cluster (Optional) |
Storage-class is a required flag for run subcommand.
- With
--volume-snapshot-class
: Performs preflight checks on the cluster with the given volumeSnapshotClass in the given namespace.
kubectl tvk-preflight --storage-class <storageclass name> --volume-snapshot-class <volumeSnapshotClass name>
- With
--local-registry
|--service-account
: Performs preflight checks on the cluster with the given local-registry and service-account in the given namespace.
kubectl tvk-preflight run --storage-class <storageclass name> --local-registry <local registry file path/name> --service-account <service account name>
- With
--image-pull-secret
: To use image-pull-secret, local-registry flag value must be provided. Vice-versa is not true.
kubectl tvk-preflight run --storage-class <storageclass name> --local-registry <local registry file path/name> --image-pull-secret <image pull secret name>
- With
--cleanup-on-failure
: If preflight checks fail, the resources generated during preflight will be cleaned.
kubectl tvk-preflight run --storage-class <storageclass name> --cleanup-on-failure
- With
--requests
: A resource request is specified in key-value format, joined by=
. Multiple resource requests can be specified in a comma separated format.
kubectl tvk-preflight run --storage-class <storageclass name> --requests cpu=200m,memory=96Mi
- With
--limits
: A resource limit is specified in key-value format, joined by=
. Multiple resource limits can be specified in a comma separated format.
kubectl tvk-preflight run --storage-class <storageclass name> --limits cpu=400m,memory=128Mi
- With
--node-selector
: Multiple labels for node selection can be specified in a comma separated format. Where each label can be specified in a format<label-key>=<label-value>
.
kubectl tvk-preflight run --storage-class <storageclass name> --node-selector <label-key1>=<label-value1>,<label-key2>=<label-value2>
The pods of preflight run can be made to schedule on a particular set of nodes of cluster by specifying the labels for node selection, node affinity, pod affinity/anti-affinity and taints and toleration.
Note: The labels except node-selector can only be specified through a config file of preflight run. Currently, it is not possible to specify pod and node affinity and tolerations through CLI flags.
Please refer below examples for specifying labels onto the pods of preflight run.
- Examples
Node selection
run:
...
podSchedulingOptions:
nodeSelector:
node-sel-key: node-sel-value
...
Node-affinity
run:
...
podSchedulingOptions:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: pref-node-affinity
operator: In
values:
- high
- key: pref-node-affinity
operator: NotIn
values:
- low
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: pref-node-affinity
operator: NotIn
values:
- medium
...
Pod-affinity/anti-affinity
run:
...
podSchedulingOptions:
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: pref-pod-affinity
operator: In
values:
- medium
- key: pref-node-affinity
operator: NotIn
values:
- high
topologyKey: preflight-topology
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
labelSelector:
matchExpressions:
- key: pref-pod-affinity
operator: In
values:
- low
topologyKey: preflight-topology
...
Taints & Toleration
run:
...
podSchedulingOptions:
tolerations:
- key: pref-node-taint
operator: Equal
value: pref-node-toleration
effect: NoSchedule
...
- cleanup subcommand cleans/deletes the resources created during failed preflight checks and not cleaned-up on failure.
- The cleanup command will clean all the resources generated due to preflight checks in the given namespace.
- User can clean resources of a particular preflight check by specifying the
uid
of the preflight check.
Parameter | Default | Description |
---|---|---|
--uid | A 6-length character string generated during preflight check |
- Without
uid
: Cleans all the preflight resources present on the cluster in the given namespace.
kubectl tvk-preflight cleanup --namespace <namespace of the cluster>
- With
uid
: Cleans the resources of particular preflight check on the cluster in the given namespace
kubectl tvk-preflight cleanup --uid <generated UID of the preflight check> --namespace <namespace of the cluster>
If namespace
is not specified then, cleanup will be performed in default namespace of the cluster.