OpenShift Validation

Table of Contents

• Validation Details
  • OpenShift Cluster Readiness
  • Application Readiness
  • Cluster Capacity Estimation
• Requirements
• Ansible Execution Environment
• Usage
• To Do

Validation Details

This repository contains the Ansible playbooks and roles for validating an OpenShift cluster.

OpenShift Cluster Readiness

The playbook performs the following opinionated validations:

oAuth server metadata	verify the oauth server’s /.well-known/oauth-authorization-server endpoint returns metadata.	documentation
	verify oAuth server issuer metadata is as expected.	documentation
cluster version	verify ClusterVersion is as specified.	documentation
cluster operators	verify all ClusterOperators are in the correct state (Available: true, Progressing: false, Degraded: false).	documentation
machine config pools	verify all nodes belong to a MachineConfigPool.	documentation
machines	verify all machines are in Running state.	documentation
	verify number of machines and nodes is equal.	documentation
nodes	verify number of control nodes is as specified.	documentation
	verify number of worker nodes is as specified.	documentation
	verify all nodes are in Ready state.	documentation
	verify all control nodes are unschedulable. Reference	documentation
	verify cAdvisor is collecting metrics for all nodes.	documentation
	verify cpu and memory metrics for nodes are collected. Reference	documentation
certificate signing request	verify no CertificateSigningRequests are pending.	documentation
wildcard DNS and certificate	verify default ingress DNS domain is as specified.	documentation
	verify wildcard DNS domain *.apps.<cluster-name>.<base-domain>.	documentation
	verify ingress serving certificate Subject Alternative Names include th wildcard DNS hostname *.apps.<cluster-name>.<base-domain>.	documentation
CoreDNS	verify DNS lookup of service A record.	documentation
	verify DNS lookup of service SRV record.	documentation
NTP	verify ntp is synchronized on all nodes.	documentation
kubeadmin user	verify kubeadmin user is removed.	documentation
oauth	verify accessTokenMaxAgeSeconds is configure.	documentation
	verify accessTokenInactivityTimeout is configured.	documentation
	verify at least one identity provider is configured.	documentation
	verify ldap identity provider uses secured connection.	documentation
	verify console login banner is configured.	documentation
CLI motd	verify motd is configured.	documentation
etcd	verify etcd is healthy.	documentation
	verify performance of disk hosting etcd.	documentation
	verify etcd data is encrypted.	documentation
	verify etcd data encryption is aes-gcm.	documentation
	verify etcd data encryption is completed.	documentation
TLS security profile	verify IngressController not configured with Old tlsSecurityProfile.	documentation
	verify IngressController Custom tlsSecurityProfile not configure with minTLSVersion set to VersionTLS10 or VersionTLS11.	documentation
	verify APIServer not configured with Old tlsSecurityProfile.	documentation
	verify APIServer Custom tlsSecurityProfile not configure with minTLSVersion set to VersionTLS10 or VersionTLS11.	documentation
	verify KubeletConfig not configured with Old tlsSecurityProfile.	documentation
	verify KubeletConfig Custom tlsSecurityProfile not configure with minTLSVersion set to VersionTLS10 or VersionTLS11.	documentation
audit profile	verify audit profile is configured.	documentation
project self provisioner	verify project self provisioner is not allowed for authenticated users.	documentation
project request template	verify project request template is configured.	documentation
	verify project request template defines: Project, LimitRange, ResourceQuota, NetworkPolicy.	documentation
storage class	verify default storage class is configured.	documentation
	verify only one default storage class is configured.	documentation
	verify default storage class is as specified.	documentation
operator hub	verify default CatalogSources are disable as specified.	documentation
	verify at least one CatalogSource is configured.	documentation
	verify all CatalogSources are in READY state.	documentation
	verify CatalogSources have PackageManifests.	documentation
	verify all ClusterServiceVersions are in Succeeded phase.	documentation
	verify all InstallPlan are approved.	documentation
monitoring stack	verify user workload monitoring is enabled.	documentation
	verify cluster monitoring stack is using persistent volume.	documentation
	verify user workload monitoring stack is using persistent volume.	documentation
	verify AlertManager has at least one receiver configured	documentation
	verify alert notifications are sent to external system	documentation
logging stack	verify openshift logging operator is deployed.	documentation
	verify ClusterLogging instance exist.	documentation
	verify ClusterLogging instance is in Managed state.	documentation
	verify ClusterLogging collector type is vector.	documentation
image controller config	verify image registries whitelist is configured.	documentation
	verify insecure image registries are not allowed.	documentation
	verify import from insecure image registries are not allowed.	documentation
	verify samplesRegistry is configured.	documentation
openshift image registry	verify image registry is not using emptyDir storage if it is in Managed state.	documentation
pods	verify all pods are in Running or Completed state.	documentation
	verify no pod in default or openshift namespaces.
	verify pods with too many restarts.
	verify cAdvisor is collecting metrics for all pods.	documentation
	verify cpu and memory metrics for pods are collected.	documentation

Application Readiness

The playbook is base on this OpenShift Learning path, it:

• stateful application:

  1. deploys mongodb StatefulSet and namegen Deployment.

  2. exposes the namegen service externally via Route.

  3. creates database, collection, and document in mongodb.

  4. stimulate failure by killing mongodb and namegeb pods; OpenShift will restart the pods.

  5. compare documents in mongodb before and after failure.

  6. deletes the resources created.

• stateless application:

  1. configure Role and RoleBinding to allow get and list access o Pods and ConfigMaps.

  2. deploys ose-tools image to run oc and curl commands against OpenShift API server, to get/list Pods, ConfigMaps, Secrets and Nodes.

Successful playbook execution confirms the following:

• create/delete OpenShift resources (Namespace, Secret, ConfigMap, PersistentVolumeClaim, PersistentVolume, NetworkPolicy, StatefulSet, Deployment, Job, Service, Route, Role, RoleBinding, etc).

• default storage class is configured.

• dynamic provisioning of persistent volume.

• persistent volume for storing stateful data; no data loss when application pods failed.

• consuming OpenShift Secret/ConfigMap ss volume or environment variables.

• service discovery via CoreDNS.

• ingress and egress network policies to allow only necessary traffic, denies the rest.

• externally exposed application service is accessible.

• RBAC to allow access to specific OpenShift resources from within the container.

Important

The playbook does not deploy the application with common best practices: images are pulled from quay.io without authentication. secrets store in environment variables. no TLS encryption for network communication. password stored in Git repository.

Cluster Capacity Estimation

The playbook estimates the capacity of the cluster using OpenShift Cluster Capacity Tool. The tool estimates the number of pods that can be scheduled on the cluster with this podSpec.

Requirements

System	RHEL v8.x or above Python v3.8 or above Ansible v2.13 or above ansible-navigator v3.0.0 or above oc command line tool (for oc debug)
Ansible content collections	kubernetes.core v2.4.0 community.crypto v2.15.1 community.general v7.5.0
Python modules	kubernetes v26.1.0 dnspython v2.6.1
OpenShift cluster	cluster-admin privileges, or custom privileges: read privileges for almost all resources, privileges to run oc debug node/nodename command, privileges to create/delete clusterRole and clusterRoleBinding, and privileges to create Namespaces and resources in the namespace.

Ansible Execution Environment

Instead of installing the dependencies on the host, use the execution environment image quay.io/bkhoo/ee-openshift-validation:1.0 to run the playbooks with ansible-navigator run command line.

See this document for the steps to build execution environment image.

Usage

1. Create a YAML file with the following settings:

   ---
   kubeconfig_file: /home/bkhoo/.kube/config
   
   cluster_version: 4.15.8
   
   cluster_ingress_domain: apps.cluster-12345.examle.com
   ingress_ip:
     - 192.168.192.168
   
   control_nodes: 3
   worker_nodes: 3
   
   default_storage_class: ocs-external-storagecluster-ceph-rbd
   
   # min_access_token_max_age_seconds: 600        # playbook default
   # min_access_token_inactivity_timeout: 600     # playbook default
   
   # etc_encryption_type: aesgcm                  # playbook default
   
   # allowed_registries:                          # playbook default
   #   - quay.io
   #   - registry.redhat.io
   #   - registry.access.redhat.com
   #   - image-registry.openshift-image-registry.svc:5000
   
   # acceptable_audit_profiles:                   # playbook default
   #   - WriteRequestBodies
   #   - AllRequestBodies
   
   # project_template_objects:                    # playbook default
   #   - Project
   #   - LimitRange
   #   - ResourceQuota
   #   - NetworkPolicy
   
   # disable_default_catalog_sources:             # playbook default
   #   - community-operators
   
   # pod_too_many_restarts: 100                   # playbook default
   ...

2. Execute ansible playbook (no inventory required):

   • OpenShift cluster readiness verification:

     ansible-navigator run openshift-readiness.yaml \
       --extra-vars @settings.yaml \
       --pull-policy missing \
       --execution-environment true \
       --execution-environment-image quay.io/bkhoo/ee-openshift-validation:1.0 \
       --execution-environment-volume-mounts /home/bkhoo/.kube:/home/bkhoo/.kube:Z
     
     ### alternate method to run the playbook; this method requires all dependencies installed
     ansible-playbook openshift-readiness.yaml --extra-vars @settings.yaml

   • Application readiness:

     ansible-navigator run application-readiness.yaml \
       --extra-vars @settings.yaml \
       --pull-policy missing \
       --execution-environment true \
       --execution-environment-image quay.io/bkhoo/ee-openshift-validation:1.0 \
       --execution-environment-volume-mounts /home/bkhoo/.kube:/home/bkhoo/.kube:Z
     
     ### alternate method to run the playbook; this method requires all dependencies installed
     ansible-playbook application-readiness.yaml --extra-vars @settings.yaml

   • Cluster capacity estimation:

     ansible-navigator run cluster-capacity.yaml \
       --extra-vars @settings.yaml \
       --pull-policy missing \
       --execution-environment true \
       --execution-environment-image quay.io/bkhoo/ee-openshift-validation:1.0 \
       --execution-environment-volume-mounts /home/bkhoo/.kube:/home/bkhoo/.kube:Z
     
     ### alternate method to run the playbook; this method requires all dependencies installed
     ansible-playbook cluster-capacity.yaml --extra-vars @settings.yaml

To Do

• infra nodes.

• infra workloads are deployed to infra nodes.

• logs are forwarded to external log aggregator.