Skip to content

Openshift IPI

Jump to bottom
cnorman edited this page Oct 19, 2022 · 63 revisions

Power VS IPI - Doc

Note: The intention for this doc is to address instructions to install, todos, issues, anything we want to maintain as a running doc

How to trigger ipi installer:

Pre-reqs:

  • [optional] A resource group. If none is supplied the default group will be used.
  • A precreated DNS domain in IBM Cloud CIS (this can be reused by as many teams as wanted, as long as cluster names are unique)
  • A pre-created Power VS Service Instance

Note: Make sure that your DHCP CIDR doesn't overlap with any other network in the same zone in your Power VS account. If you're installing 4.11, which doesn't have the option to configure the CIDR, make sure to only install one IPI cluster per account per zone. For 4.12 installs, you make specify the network CIDR to ensure that no clusters' networks overlap.

Release Images

Option 1: From mirror site(suitable for the GA'ed builds)

Option 2: From ppc64le release page(For CI and nightly builds

Single arch ppc64le images: https://openshift-release-ppc64le.apps.ci.l2s4.p1.openshiftapps.com, clink on the relevant build)

Multi-arch images: https://multi.ocp.releases.ci.openshift.org (useful for project like hypershift)

Get the installer:

Option 1: Build from source(developer's flow, not suggested for the testing)

  • Git clone and build the code 
    $ git clone https://github.com/openshift/installer.git && cd installer
$ go mod vendor
$ DEFAULT_ARCH=ppc64le ./hack/build.sh

Option 2: From Nightly bits(suitable for the testing)

Get the installer from the latest nightly (e.g.https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.11.0-0.nightly-ppc64le-2022-05-06-144942/openshift-install-linux-amd64.tar.gz)

Option 3: Extract from the release image(suitable for the testing)

$ oc adm release extract --tools registry.ci.openshift.org/ocp-ppc64le/release-ppc64le:4.12.0-0.nightly-ppc64le-2022-09-02-071559

Set the environment variables

export IBMCLOUD_API_KEY=<API_KEY>
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release-nightly:4.11.0-0.nightly-ppc64le-2022-05-10-222820
 Note: You can find the latest promoted release image here: https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview <br>
 You shouldn't need any special pull secrets to download these images. <br>
 Make sure you can pull the image before you spin up a cluster, e.g. `podman pull quay.io/openshift-release-dev/ocp-release-nightly:4.11.0-0.nightly-ppc64le-2022-04-14-130031`

Create the installation directory and Install config file

If you don't have an install-config.yaml already:

$ mkdir ocp-test
$ ./bin/openshift-install  create install-config --dir ocp-test
  1. domain should be scnl-ibm.com (for the dev teams)
  2. cluster name should begin with rdr-<your_ibm_id
  3. manually enter the ID of your service instance (ibmcloud login -r ; ibmcloud sl | grep )

Note: pullSecret can be found from the following link

Note: When testing in staging, manually update the sysType to sysType: s922-ocp

Create manifests

./bin/openshift-install create manifests --dir ocp-test

Cloud Credentials

Extract cloud credential requests from the build and generate cloud credentials using ccoctl

oc adm release extract --cloud=powervs --credentials-requests quay.io/openshift-release-dev/ocp-release-nightly@sha256:dd6ca3e165aca4e05e80726e23b4f74112afbf5d2ef04c24bc8f9d80f942ef1a --to=new_crs -a ./.hypershift/pull_secret.txt --filter-by-os=linux/ppc64le

You'll need to get four credentials files from someone who has admin authority on your account. Copy them into your manifests/ directory:

cp ~/Downloads/openshift-cloud-controller-manager-ibm-cloud-credentials-credentials.yaml ocp-test/manifests/.
cp ~/Downloads/openshift-ingress-operator-cloud-credentials-credentials.yaml ocp-test/manifests/.
cp ~/Downloads/openshift-machine-api-powervs-credentials-credentials.yaml ocp-test/manifests/.
cp ~/Downloads/openshift-cluster-csi-drivers-ibm-powervs-cloud-credentials-credentials.yaml ocp-test/manifests/.

Note: These will contain API keys that are tied to a service account created under your account. Do not change the API Keys here to your own. (This will work if you have enough access, but that's not technically how users will do this, so please help us find problems early by doing it this way). You may also created these service ids and files yourself if you want. Just remember to delete old service IDs. For more information, see https://github.com/openshift/cloud-credential-operator/blob/master/docs/ccoctl.md#ibm-cloud-power-vs and https://docs.openshift.com/container-platform/4.10/installing/installing_ibm_cloud_public/installing-ibm-cloud-customizations.html#manually-create-iam-ibm-cloud_installing-ibm-cloud-customizations

Trigger the installer

TF_LOG=debug ./bin/openshift-install create cluster --log-level=debug --dir ocp-test

Destroy

We no longer need the cleanup script. openshift-install destroy cluster works, but may have timeouts and need to be run multiple times.

Workarounds:

  • If you accidentally delete your test dir, you can still use the cleanup script. https://github.com/openshift-powervs/hack/blob/main/scripts/cleanup.sh

    Example usage: CLUSTER_NAME=rdr-cln INFRA_ID="$(cat ocp-test/terraform.tfvars.json | jq -r .cluster_id)" ./cleanup.sh

    Your security groups will most likely not be deleted after the number of tries in the script, so wait about 5 minutes and re-run the script if they aren't.

Set the kubeconfig file to access the OCP cluster via oc command

$ export KUBECONFIG=<dir>/auth/kubeconfig

Troubleshooting

IPI is supposedly opinionated, allowing few user customizations, thus solid, works every time, and everything, but still dealing with Cloud which goes through constant changes, and sometimes things do break.

Getting bootstrap and master node IPs:

# cat bootstrap.tfvars.json | jq .bootstrap_private_ip
# cat cluster.tfvars.json | jq .control_plane_ips

You can set up a jumpbox on the same DHCP network. Give it a public IP and add it to the same DHCP network:

Then you can ssh -J core@bootstrap from your system.

If you don't have the -J option, you can use agent forwarding: ssh -o ForwardAgent=yes root@<jumpbox>, then from there, ssh core@<bootstrap>

  • Make sure you can ping the 192.x control plane IPs from the bootstrap node
  • Make sure you can ping the private network gateway (192.168.0.1)
  • If you're having connectivity problems, you can also try creating a VM in the VPC and pinging that. Just remember to allow ICMP in the security group rule there.

Clone this wiki locally