-
Notifications
You must be signed in to change notification settings - Fork 40
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Enabling disconnected cluster installation (#205)
Part of #143 This PR sets the core requirements for disconnected cluster installation. This includes required variables, roles for mirroring (legacy for the platform and `oc-mirror` for everything else), applying `oc-mirror` manifests post cluster up, and corresponding changes to the rest of the playbook and docs. ```bash # update the vars files ❯ ansible-playbook playbooks/0_setup.yaml ❯ ansible-playbook playbooks/3_setup_kvm_host.yaml # If using in in-place registry (for testing) set this up on the plar here and bind it to the interface where the cluster will be setup. Make sure the lpar can resolve to this and then the cluster forwarder IP is set to the gateway IP of the interface Eg 192.168.122.1 and then patch ins reg and pull secret values accordingly. A sample script for this is provided below ❯ ansible-playbook playbooks/4_create_bastion.yaml ❯ ansible-playbook playbooks/disconnected_mirror_artifacts.yaml ❯ ansible-playbook playbooks/5_setup_bastion.yaml ❯ ansible-playbook playbooks/6_create_nodes.yaml ❯ ansible-playbook playbooks/7_ocp_verification.yaml ❯ ansible-playbook playbooks/disconnected_apply_operator_manifests.yaml ``` Sample script for in-place registry on the LPAR itself for NAT-based cluster setup. To be run only after the libvirt interface is created/available on lpar: https://github.com/mohammedzee1000/IBM-Ansible-OpenShift-Provisioning-TestNotes/blob/main/scripts/testing_utils/create_reg.sh --------- Signed-off-by: Mohammed Zeeshan Ahmed <mohammed.ahmed18@ibm.com> Signed-off-by: Mohammed Zeeshan Ahmed <mohammed.zee1000@gmail.com> Co-authored-by: Mohammed Zeeshan Ahmed <mohammed.ahmed18@ibm.com> Co-authored-by: Jacob Emery <jacob.emery@ibm.com>
- Loading branch information
3 people
committed
Oct 30, 2023
1 parent
3168120
commit 1e81c50
Showing
22 changed files
with
444 additions
and
27 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| .DS_Store | ||
| .iso | ||
| .vscode | ||
| site | ||
| site | ||
| .oc-mirror-results |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # Run the Playbooks | ||
| ## Overview | ||
| For installing disconnected clusters, you will mostly be following rhe same process as a standard connected cluster. | ||
|
|
||
| The main additional steps we would be doing is mirroring the OCP images to another registry which is accessible to | ||
| the cluster and post the cluster coming up, we will be applying operator hub manifests such as image content source | ||
| policy and catalog source, generated by `oc-mirror`, to the cluster. | ||
|
|
||
| Disconnected playbook are mentioned below. Please refer the **4 Run the Playbooks** documentation for details of rest of the playbooks: | ||
|
|
||
| * disconnected_mirror_artifacts.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/disconnected_mirror_artifacts.yaml)) - Run before **6_create_nodes.yaml** | ||
| * disconnected_apply_operator_manifests.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/disconnected_apply_operator_manifests.yaml)) - Run after **7_ocp_verification.yaml** | ||
| ## Pre-requisites | ||
| * A running registry where the OCP and operator hub images will be mirrored. If the CA of this registry is not automatically trusted, then keep the | ||
| CA cert content handy to update in inventory file. The CA cert is the file with which, do dont need to skip tls to access the registry. | ||
| * Make sure you have required pull secrets handy. You will need 2 pull secrets, one to apply on the cluster and another which will be used for | ||
| mirroring. The mirroring pull secret MUST have push access to the mirror registry as well as must give you access to Red Hat registries. | ||
| A good way to create this would be take the Red Hat pull secret from **Get Info page** and do a podman login with creds having write access. | ||
|
|
||
| ``` | ||
| cp -avrf /path/to/redhat-pull-secrets.json ./mirror-secret.json | ||
| podman login -u admin -p admin <mirror_registry> --tls-verify=false --authfile=./mirror-secret.json | ||
| cat ./mirror-secret.json | jq -r tostring | ||
| <copy this output> | ||
| ``` | ||
|
|
||
| * A mirror host. This can be any host that can access the internet (mainly the registry being mirrored from) as well as the registry being mirrored to. | ||
| This registries being mirrored from would typically be the Red Hat registries (registry.redhat.io, quay.io etc) | ||
| * The file server, configured mentioned below. | ||
| * Appropriately updated variables in your `all.yaml`. Refer the variables documentation. | ||
| ### File Server | ||
| This configuration will take place on the file server mentioned under **File Server** section in overall pre-requisites documentaion. The additional | ||
| configurations are mentioned over here. | ||
|
|
||
| * Make sure to have a directory housing the clients | ||
|
|
||
| * For FTP: | ||
|
|
||
| ``` | ||
| sudo mkdir /home/<username>/clients | ||
| ``` | ||
|
|
||
| * or HTTP: | ||
|
|
||
| ``` | ||
| sudo mkdir /var/www/html/clients | ||
| ``` | ||
|
|
||
| Make sure this directory contains a pre-downloaded `oc-mirror` binary in `tar.gz` format. Currently the supported binary is available for `x86_64` on Red Hat Customer portal openshift [downloads](https://console.redhat.com/openshift/downloads) page. It can also be found on mirror.openshift.com from `4.14` onwards for other architectures. | ||
| ### NOTE | ||
| * At this stage, only oc-mirror binary is fetched from File Server, so it is expected that the lpar for disconnected cluster can at least reach `mirror.openshift.com` to download the | ||
| other artifacts for cluster installation. | ||
| * The platorm related image content source policy will be baked into the install config as part of **5 Setup Bastion Playbook**. | ||
| * Right now on legacy platform mirroring is supported in this playook during the creation of the cluster. | ||
| * Manifests generated by `oc-mirror` will be applied to the cluster cluster up. So if you add platform details in image set, it | ||
| will be applied on cluster only after the cluster is up. | ||
|
|
||
| ## Disconnected Mirror Artifacts Playbook | ||
| ### Overview | ||
| Mirror the ocp platform and other necessary images to the mirror registry. Please run this playbook before you run **6 Create Nodes Playbook** and after | ||
| **0 Setup Playbook**. | ||
| ### Outcomes | ||
| * Download `oc` and `oc-mirror` to the mirror host. | ||
| * Template the mirror pull secret to the mirror host. | ||
| * Add the ca cert to the mirror host anchors if ca is not trusted. | ||
| * Mirror the platform images using `oc adm release mirror` if legacy mirroring is enabled. | ||
| * Template the image set to mirror host and then mirror it using `oc-mirror` plogin. | ||
| * Copy the results on the `oc-mirror` to ansible controller to apply to cluster in future steps. | ||
| ### Notes | ||
| * Currently, platform can **only** be mirrored the legacy way. While the image set can contain platform mirroring configs, it will **not** be applied to cluster during creation. | ||
| * This playbook can be run at any stage after the **0 Setup** playbook. Make sure to run this before the cluster starts pulling at the images from the registry | ||
| which typically happens where the **Create Nodes Playbook** is run. | ||
|
|
||
| # Disconnected apply oc mirror manifests to cluster Playbook | ||
| ### Overview | ||
| Post cluster creation, `oc-mirror` manifests are applied to the cluster. Please run this playbook after **7 OCP Verification Playbook**. | ||
| ### Outcomes | ||
| * Copy the `oc-mirror` results manifests to the bastion. | ||
| * Apply the copied manifests to the cluster. | ||
| * Disable default content sources. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| --- | ||
| - name: Disconnected apply operator manifests | ||
| hosts: bastion | ||
| become: true | ||
| environment: | ||
| KUBECONFIG: "{{ '/home/' if (env.bastion.access.user != 'root') else '/'}}{{ env.bastion.access.user }}/.kube/config" | ||
| gather_facts: true | ||
| vars_files: | ||
| - "{{ inventory_dir }}/group_vars/all.yaml" | ||
| tasks: | ||
| - name: apply operator manifests when cluster is disconnected | ||
| ansible.builtin.include_role: | ||
| name: '{{ item }}' | ||
| loop: | ||
| - disconnected_check_vars | ||
| - disconnected_apply_operator_manifests_to_cluster | ||
| when: disconnected.enabled |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| --- | ||
| - name: Mirror ocp artifacts | ||
| hosts: mirrorhost | ||
| gather_facts: false | ||
| vars_files: | ||
| - "{{ inventory_dir }}/group_vars/all.yaml" | ||
| tasks: | ||
| - name: mirror ocp artifacts when disconnected install | ||
| ansible.builtin.include_role: | ||
| name: '{{ item }}' | ||
| loop: | ||
| - disconnected_check_vars | ||
| - disconnected_mirror_images | ||
| when: disconnected.enabled |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.