This project deploys the OpenShift Assisted Installer in Minikube and spawns libvirt VMs that represent bare metal hosts.
Table of contents
- Test-Infra
- Prerequisites
- Installation Guide
- OS parameters used for configuration
- Instructions
- Usage
- Full flow cases
- Run full flow with install
- Run full flow without install
- Run only deploy nodes (without pre deploy of all assisted service)
- Redeploy nodes
- Redeploy with assisted services
- Cleaning
- Install cluster
- Create cluster and download ISO
- Deploy Assisted Service and Monitoring stack
deploy_assisted_serviceand Create cluster and download ISO- start_minikube and Deploy UI and open port forwarding on port 6008, allows to connect to it from browser
- Kill all open port forwarding commands, will be part of destroy target
- Test
assisted-serviceimage - Test installer, controller,
assisted-serviceand agent images in the same flow
- In case you would like to build the image with a different
assisted-serviceclient
- CentOS 8 or RHEL 8 host
- File system that supports d_type
- Ideally on a bare metal host with at least 64G of RAM.
- Run as a user with password-less
sudoaccess or be ready to entersudopassword for prepare phase. - Get a valid pull secret (JSON string) from redhat.com if you want to test the installation (not needed for testing only the discovery flow). Export it as
export PULL_SECRET='<pull secret JSON>'Check the Install Guide for installation instructions.
| Variable | Description |
|---|---|
| SERVICE_BRANCH | assisted-service branch to use, default: master |
| SERVICE_REPO | assisted-service repository to use, default: https://github.com/openshift/assisted-service |
| ISO | path to ISO to spawn VM with, if set vms will be spawn with this iso without creating cluster. File must have the '.iso' suffix |
| NUM_MASTERS | number of VMs to spawn as masters, default: 3 |
| WORKER_MEMORY | memory for worker VM, default: 8892MB |
| MASTER_MEMORY | memory for master VM, default: 16984MB |
| NUM_WORKERS | number of VMs to spawn as workers, default: 0 |
| SSH_PUB_KEY | SSH public key to use for image generation, gives option to SSH to VMs, default: ssh_key/key_pub |
| PULL_SECRET | pull secret to use for cluster installation command, no option to install cluster without it. |
| ROUTE53_SECRET | Amazon Route 53 secret to use for DNS domains registration. |
| CLUSTER_NAME | cluster name, used as prefix for virsh resources, default: test-infra-cluster |
| BASE_DOMAIN | base domain, needed for DNS name, default: redhat.com |
| BASE_DNS_DOMAINS | base DNS domains that are managed by assisted-service, format: domain_name:domain_id/provider_type. |
| NETWORK_CIDR | network CIDR to use for virsh VM network, default: "192.168.126.0/24" |
| CLUSTER_ID | cluster id , used for install_cluster command, default: the last spawned cluster |
| NETWORK_NAME | virsh network name for VMs creation, default: test-infra-net |
| NETWORK_BRIDGE | network bridge to use while creating virsh network, default: tt0 |
| OPENSHIFT_VERSION | OpenShift version to install, default: "4.5" |
| HTTP_PROXY_URL | A proxy URL to use for creating HTTP connections outside the cluster |
| HTTPS_PROXY_URL | A proxy URL to use for creating HTTPS connections outside the cluster |
| NO_PROXY_VALUES | A comma-separated list of destination domain names, domains, IP addresses, or other network CIDRs to exclude proxying |
| SERVICE_BASE_URL | update assisted-service config map SERVICE_BASE_URL parameter with given URL, including port and protocol |
| AGENT_DOCKER_IMAGE | agent docker image to use, will update assisted-service config map with given value |
| INSTALLER_IMAGE | assisted-installer image to use, will update assisted-service config map with given value |
| SERVICE | assisted-service image to use |
| DEPLOY_TAG | the tag to be used for all images (assisted-service, assisted-installer, agent, etc) this will override any other os parameters |
| DEPLOY_MANIFEST_PATH | the location of a manifest file that defines image tags images to be used |
| DEPLOY_MANIFEST_TAG | the Git tag of a manifest file that defines image tags to be used |
| IMAGE_BUILDER | image-builder image to use, will update assisted-service config map with given value |
| CONNECTIVITY_CHECK_IMAGE | connectivity-check image to use, will update assisted-service config map with given value |
| HARDWARE_INFO_IMAGE | hardware-info image to use, will update assisted-service config map with given value |
| INVENTORY_IMAGE | assisted-service image to be updated in assisted-service config map with given value |
| OC_MODE | if set, use oc instead of minikube |
| OC_TOKEN | token for oc login (an alternative for oc-user & oc-pass) |
| OC_SERVER | server for oc login, required if oc-token is provided, default: https://api.ocp.prod.psi.redhat.com:6443 |
| OC_SCHEME | Scheme for assisted-service url on oc, default: http |
| SERVICE_NAME | assisted-service target service name, default: assisted-service |
| KUBECONFIG | kubeconfig file path, default: /.kube/config |
| ENABLE_AUTH | configure assisted-service to authenticate API requests, default: false |
| OFFLINE_TOKEN | token used to fetch JWT tokens for assisted-service authentication (from https://cloud.redhat.com/openshift/token) |
| SSO_URL | URL used to fetch JWT tokens for assisted-service authentication |
| OCM_CLIENT_ID | ID of Service Account used to communicate with OCM and AMS for Agent Auth and Authz |
| OCM_CLIENT_SECRET | Password of Service Account used to communicate with OCM and AMS for Agent Auth and Authz |
| OCM_BASE_URL | OCM API URL used to communicate with OCM and AMS, default: https://api-integration.6943.hive-integration.openshiftapps.com |
On the bare metal host:
dnf install -y git make
cd /home/test # don't do it on /root it will breaks build image mounts and fail to run
git clone https://github.com/openshift/assisted-test-infra.gitWhen using this infra for the first time on a host, run:
make create_full_environmentThis will install required packages, configure libvirt, pull relevant Docker images, and start Minikube.
There are different options to use test-infra, which can be found in the makefile.
The following is a list of stages that will be run:
- Start Minikube if not started yet
- Deploy services for assisted deployment on Minikube
- Create cluster in
assisted-serviceservice - Download ISO image
- Spawn required number of VMs from downloaded ISO with parameters that can be configured by OS environment (check makefile)
- Wait until nodes are up and registered in
assisted-service - Set nodes roles in
assisted-serviceby matching VM names (worker/master) - Verify all nodes have required hardware to start installation
- Install nodes
- Download
kubeconfig-noingressto build/kubeconfig - Waiting till nodes are in
installedstate, while verifying that they don't move toerrorstate - Verifying cluster is in state
installed - Download kubeconfig to build/kubeconfig
Note: Please make sure no previous cluster is running before running a new one (it will rewrite its build files).
To run the full flow, including installation:
make run_full_flow_with_installOr to run it together with create_full_environment (requires sudo password):
make allTo run the flow without the installation stage:
make run_full_flowmake deploy_nodes or make deploy_nodes_with_installmake redeploy_nodes or make redeploy_nodes_with_installmake redeploy_all or make redeploy_all_with_installFollowing sections show how to perform cleaning of test-infra environment.
make destroymake destroy_nodesSometimes you may need to delete all libvirt resources
make delete_all_virsh_resourcesInstall cluster after nodes were deployed. Can take ClusterId as OS environment
make install_clustermake download_isomake run
make deploy_monitoringmake download_iso_for_remote_usestart_minikube and Deploy UI and open port forwarding on port 6008, allows to connect to it from browser
make deploy_uimake kill_all_port_forwardingsmake redeploy_all SERVICE=<image to test>
or
export PULL_SECRET='<pull secret JSON>'; make redeploy_all_with_install SERVICE=<image to test>make redeploy_all AGENT_DOCKER_IMAGE=<image to test>
or
make redeploy_all_with_install AGENT_DOCKER_IMAGE=<image to test>make redeploy_all INSTALLER_IMAGE=<image to test> CONTROLLER_IMAGE=<image to test>
or
export PULL_SECRET='<pull secret JSON>'; make redeploy_all_with_install INSTALLER_IMAGE=<image to test> CONTROLLER_IMAGE=<image to test>make redeploy_all INSTALLER_IMAGE=<image to test> AGENT_DOCKER_IMAGE=<image to test> SERVICE=<image to test>
or
export PULL_SECRET='<pull secret JSON>'; make redeploy_all_with_install INSTALLER_IMAGE=<image to test> CONTROLLER_IMAGE=<image to test> AGENT_DOCKER_IMAGE=<image to test> SERVICE=<image to test>Assisted-test-infra builds an image including all the prerequisites to handle this repository.
make image_buildmake image_build SERVICE=<assisted service image URL>