Skip to content

Commit

Permalink
feat: Enabling disconnected cluster installation (#205)
Browse files Browse the repository at this point in the history
Part of #143 

This PR sets the core requirements for disconnected cluster
installation. This includes required variables, roles for mirroring
(legacy for the platform and `oc-mirror` for everything else), applying
`oc-mirror` manifests post cluster up, and corresponding changes to the
rest of the playbook and docs.

```bash
# update the vars files
❯ ansible-playbook playbooks/0_setup.yaml
❯ ansible-playbook playbooks/3_setup_kvm_host.yaml
# If using in in-place registry (for testing) set this up on the plar here and bind it to the interface where the cluster will be setup. Make sure the lpar can resolve to this and then the cluster forwarder IP is set to the gateway IP of the interface Eg 192.168.122.1 and then patch ins reg and pull secret values accordingly. A sample script for this is provided below
❯ ansible-playbook playbooks/4_create_bastion.yaml
❯ ansible-playbook playbooks/disconnected_mirror_artifacts.yaml
❯ ansible-playbook playbooks/5_setup_bastion.yaml
❯ ansible-playbook playbooks/6_create_nodes.yaml
❯ ansible-playbook playbooks/7_ocp_verification.yaml
❯ ansible-playbook playbooks/disconnected_apply_operator_manifests.yaml
```
Sample script for in-place registry on the LPAR itself for NAT-based
cluster setup. To be run only after the libvirt interface is
created/available on lpar:
https://github.com/mohammedzee1000/IBM-Ansible-OpenShift-Provisioning-TestNotes/blob/main/scripts/testing_utils/create_reg.sh

---------

Signed-off-by: Mohammed Zeeshan Ahmed <mohammed.ahmed18@ibm.com>
Signed-off-by: Mohammed Zeeshan Ahmed <mohammed.zee1000@gmail.com>
Co-authored-by: Mohammed Zeeshan Ahmed <mohammed.ahmed18@ibm.com>
Co-authored-by: Jacob Emery <jacob.emery@ibm.com>
  • Loading branch information
3 people committed Oct 30, 2023
1 parent 3168120 commit 1e81c50
Show file tree
Hide file tree
Showing 22 changed files with 444 additions and 27 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
.DS_Store
.iso
.vscode
site
site
.oc-mirror-results
2 changes: 1 addition & 1 deletion docs/prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
```
sudo mkdir /home/<username>/ocp-config
```
or HTTP:
* or HTTP:
```
sudo mkdir /var/www/html/ocp-config
```
Expand Down
80 changes: 80 additions & 0 deletions docs/run-the-playbooks-for-disconnected.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Run the Playbooks
## Overview
For installing disconnected clusters, you will mostly be following rhe same process as a standard connected cluster.

The main additional steps we would be doing is mirroring the OCP images to another registry which is accessible to
the cluster and post the cluster coming up, we will be applying operator hub manifests such as image content source
policy and catalog source, generated by `oc-mirror`, to the cluster.

Disconnected playbook are mentioned below. Please refer the **4 Run the Playbooks** documentation for details of rest of the playbooks:

* disconnected_mirror_artifacts.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/disconnected_mirror_artifacts.yaml)) - Run before **6_create_nodes.yaml**
* disconnected_apply_operator_manifests.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/disconnected_apply_operator_manifests.yaml)) - Run after **7_ocp_verification.yaml**
## Pre-requisites
* A running registry where the OCP and operator hub images will be mirrored. If the CA of this registry is not automatically trusted, then keep the
CA cert content handy to update in inventory file. The CA cert is the file with which, do dont need to skip tls to access the registry.
* Make sure you have required pull secrets handy. You will need 2 pull secrets, one to apply on the cluster and another which will be used for
mirroring. The mirroring pull secret MUST have push access to the mirror registry as well as must give you access to Red Hat registries.
A good way to create this would be take the Red Hat pull secret from **Get Info page** and do a podman login with creds having write access.

```
cp -avrf /path/to/redhat-pull-secrets.json ./mirror-secret.json
podman login -u admin -p admin <mirror_registry> --tls-verify=false --authfile=./mirror-secret.json
cat ./mirror-secret.json | jq -r tostring
<copy this output>
```

* A mirror host. This can be any host that can access the internet (mainly the registry being mirrored from) as well as the registry being mirrored to.
This registries being mirrored from would typically be the Red Hat registries (registry.redhat.io, quay.io etc)
* The file server, configured mentioned below.
* Appropriately updated variables in your `all.yaml`. Refer the variables documentation.
### File Server
This configuration will take place on the file server mentioned under **File Server** section in overall pre-requisites documentaion. The additional
configurations are mentioned over here.

* Make sure to have a directory housing the clients

* For FTP:

```
sudo mkdir /home/<username>/clients
```

* or HTTP:

```
sudo mkdir /var/www/html/clients
```

Make sure this directory contains a pre-downloaded `oc-mirror` binary in `tar.gz` format. Currently the supported binary is available for `x86_64` on Red Hat Customer portal openshift [downloads](https://console.redhat.com/openshift/downloads) page. It can also be found on mirror.openshift.com from `4.14` onwards for other architectures.
### NOTE
* At this stage, only oc-mirror binary is fetched from File Server, so it is expected that the lpar for disconnected cluster can at least reach `mirror.openshift.com` to download the
other artifacts for cluster installation.
* The platorm related image content source policy will be baked into the install config as part of **5 Setup Bastion Playbook**.
* Right now on legacy platform mirroring is supported in this playook during the creation of the cluster.
* Manifests generated by `oc-mirror` will be applied to the cluster cluster up. So if you add platform details in image set, it
will be applied on cluster only after the cluster is up.

## Disconnected Mirror Artifacts Playbook
### Overview
Mirror the ocp platform and other necessary images to the mirror registry. Please run this playbook before you run **6 Create Nodes Playbook** and after
**0 Setup Playbook**.
### Outcomes
* Download `oc` and `oc-mirror` to the mirror host.
* Template the mirror pull secret to the mirror host.
* Add the ca cert to the mirror host anchors if ca is not trusted.
* Mirror the platform images using `oc adm release mirror` if legacy mirroring is enabled.
* Template the image set to mirror host and then mirror it using `oc-mirror` plogin.
* Copy the results on the `oc-mirror` to ansible controller to apply to cluster in future steps.
### Notes
* Currently, platform can **only** be mirrored the legacy way. While the image set can contain platform mirroring configs, it will **not** be applied to cluster during creation.
* This playbook can be run at any stage after the **0 Setup** playbook. Make sure to run this before the cluster starts pulling at the images from the registry
which typically happens where the **Create Nodes Playbook** is run.

# Disconnected apply oc mirror manifests to cluster Playbook
### Overview
Post cluster creation, `oc-mirror` manifests are applied to the cluster. Please run this playbook after **7 OCP Verification Playbook**.
### Outcomes
* Copy the `oc-mirror` results manifests to the bastion.
* Apply the copied manifests to the cluster.
* Disable default content sources.
2 changes: 1 addition & 1 deletion docs/run-the-playbooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Configuration of the bastion to host essential infrastructure services for the c
* CoreOS roofts is pulled to the bastion if not already there.
* OCP client and installer are pulled down if not there already.
* oc, kubectl and openshift-install binaries are installed.
* OCP install-config is templated and backed up.
* OCP install-config is templated and backed up. In disconnected mode, if platform is mirrored (currently only legacy), image content source policy and additionalTrustBundle is also patched.
* Manfifests are created.
* OCP install directory found at /root/ocpinst/ is created and populated with necessary files.
* Ignition files for the bootstrap, control, and compute nodes are transferred to HTTP-accessible directory for booting nodes.
Expand Down
31 changes: 30 additions & 1 deletion docs/set-variables-group-vars.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,36 @@
**hypershift.agents_parms.vcpus** | vCPUs for agents | 4
**hypershift.agents_parms.nameserver** | Nameserver to be used for agents | 192.168.10.1

## 17 - (Optional) Create compute node in a day-2 operation
## 17 - (Optional) Disconnected cluster setup
**Variable Name** | **Description** | **Example**
:--- | :--- | :---
**disconnected.enabled** | True or False, to enable disconnected mode | False
**disconnected.registry.url** | String containing url of disconnected registry with or without port and without protocol | registry.tt.testing:5000
**disconnected.registry.pull_secret** | String containing pull secret of the disconnected registry to be applied on the *cluster*. <br /> Make sure to enclose pull_secret in 'single quotes' and it has appropriate pull access. | '{"auths":{"registry.tt.<br />.testing:5000":{"auth":"b3Blb<br />...<br />4yQQ==","email":"test.<br />user@example.com"}}}'
**disconnected.registry.mirror_pull_ecret** | String containing pull secret to use for mirroring. Contains Red Hat secret and registry pull <br /> secret. Make sure to enclose pull_secret in 'single quotes' and must be able to push to mirror<br /> registry. | '{"auths":{"cloud.openshift<br />.com":{"auth":"b3Blb<br />...<br />4yQQ==","email":"redhat.<br />user@gmail.com", "registry.tt..testing:5000":<br />...<br />user@example.com"}}}'
**disconnected.registry.ca_trusted** | True or False to indicate that mirror registry CA is implicitly trusted or needs to be made<br /> trusted on mirror host and cluster. | False
**disconnected.registry.ca_cert** | Multiline string containing the mirror registry CA bundle | -----BEGIN CERTIFICATE-----<br />MIIDqDCCApCgAwIBAgIULL+d1HTYsiP+8jeWnqBis3N4BskwDQYJKoZIhvcNAQEF<br />...<br />-----END CERTIFICATE-----
**disconnected.mirroring.host.name** | String containing the hostname of the host, which will be used for mirroring | mirror-host-1
**disconnected.mirroring.host.ip** | String containing ip of the host, which will be used for mirroring | 192.168.10.99
**disconnected.mirroring.host.user** | String containing the username of the host, which will be used for mirroring | mirroruser
**disconnected.mirroring.host.pass** | String containing the password of the host, which will be used for mirroring | mirrorpassword
**disconnected.mirroring.file_server.clients_dir** | Directory path relative to the HTTP/FTP accessible directory on **env.file_server**<br /> where client binary tarballs are kept | clients
**disconnected.mirroring.file_server.oc_mirror_tgz** | Name of oc-mirror tarball on **env.file_server** in **disconnected.mirroring.file_server.clients_dir** | oc-mirror.tar.gz
**disconnected.mirroring.legacy.platform** | True or False if the platform should be mirrored using `oc adm release mirror`. | True
**disconnected.mirroring.legacy.ocp_quay_release_image_tag** | The tag of the release image *quay.io/openshift-release-dev/ocp-release* to mirror and use | 4.13.1-s390x
**disconnected.mirroring.legacy.ocp_org** | The org part of the repo on the mirror registry where the release image will be pushed | ocp4
**disconnected.mirroring.legacy.ocp_repo** | The repo part of the repo on the mirror registry where the release image will be pushed | openshift4
**disconnected.mirroring.legacy.ocp_tag** | The tag part of the repo on the mirror registry where the release image will be pushed.<br /> Full image would be as below.:<br /><br /> disconnected.registry.url/disconnected.mirroring.legacy.ocp_org/disconnected...ocp_repo<br />:disconnected..ocp_tag | v4.13.1
**disconnected.mirroring.oc_mirror.oc_mirror_args.continue_on_error** | True or False to give `--continue-on-error` flag to `oc-mirror` | False
**disconnected.mirroring.oc_mirror.oc_mirror_args.source_skip_tls** | True or False to give `--source-skip-tls` flag to `oc-mirror` | False
**disconnected.mirroring.oc_mirror.image_set** | YAML fields containing a standard `oc-mirror` [image set](https://docs.openshift.com/container-platform/latest/installing/disconnected_install/installing-mirroring-disconnected.html#oc-mirror-creating-image-set-config_installing-mirroring-disconnected) with some minor changes to schema. <br /> Differences are documented as needed. Used to generate final image set. | see template
**disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.enabled** | True or False to use registry storage backend for pushing mirrored content directly to the registry. <br /> Currently only this backend is supported.| True
**disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.imageURL.org** | The org part of registry imageURL from standard image set. | mirror
**disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.imageURL.repo** | The repo part of registry imageURL from standard image set. <br /> Final imageURL will be as below:<br /> <br /> disconnected.registry.url/disconnected.mirroring.oc_mirror.image_set.storageConfig<br />.registry.imageURL.org/disconnected...imageURL.repo | oc-mirror-metadata
**disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.skipTLS** | True of False same purpose served as in standard image set i.e. skip the tls for the registry<br /> during mirroring.| false
**disconnected.mirrroing.oc_mirror.image_set.mirror** | YAML containing a list of what needs to be mirrored. See the oc mirror image set documentation. <br /> *WARNING*: Platform mirroring in this way is not supported. Use legacy way of platform mirroring | see oc-mirror [image set](https://docs.openshift.com/container-platform/latest/installing/disconnected_install/installing-mirroring-disconnected.html#oc-mirror-creating-image-set-config_installing-mirroring-disconnected) documentation

## 18 - (Optional) Create compute node in a day-2 operation

**Variable Name** | **Description** | **Example**
:--- | :--- | :---
Expand Down
91 changes: 74 additions & 17 deletions inventories/default/group_vars/all.yaml.template
Original file line number Diff line number Diff line change
Expand Up @@ -246,19 +246,19 @@ hypershift:

create_bastion: true
networking_device: enc1100
gateway:
gateway:

bastion_parms:
interface:
hostname:
base_domain:
os_variant:
interface:
hostname:
base_domain:
os_variant:
nameserver:
gateway:
subnet_mask:
gateway:
subnet_mask:

mgmt_cluster_nameserver:
oc_url:
mgmt_cluster_nameserver:
oc_url:

#Hosted Control Plane Parameters

Expand All @@ -275,7 +275,7 @@ hypershift:

# MultiClusterEngine Parameters
mce:
version:
version:
instance_name: engine
delete: false

Expand All @@ -294,25 +294,82 @@ hypershift:
static_ip_parms:
static_ip: true
ip: # Required only if static_ip is true
#-
#-
#-
#-
interface: eth0
agents_count:
agents_count:
# If you want to use specific mac addresses, provide them here
agent_mac_addr:
#-
agent_mac_addr:
#-
disk_size: 100G
ram: 16384
vcpus: 4
nameserver:

# Section 17 - (Optional) Create additional compute node in a day-2 operation
# Section 17 - (Optional) Setup disconnected clusters
# Warning: currently, the oc-mirror plugin is officially downloadable to amd64 only.
disconnected:
enabled: False
registry:
url: 'registry url'
pull_secret: '' # this is similar to env.redhat.pull_secret but it will only contain secrets to be applied to the cluster in disconnected mode.
mirror_pull_secret: '' # this should contain a pull secret that contains the combination of env.redhat.pull_secret and pull secret with push access to mirror registry for mirroring
ca_trusted: False
ca_cert: |
-----BEGIN CERTIFICATE-----
if ca_trusted is False, then this ca will be added to mirror host anchors as well as to the install config of cluster
-----END CERTIFICATE-----
mirroring:
host: # this is the host that can access the internet as well as the registry
name: hosname
ip: x.x.x.x
user: mirroruser # with become access
pass: mirrorpassword
file_server: # in disconnected mode, the client binaries and rhcos will be put on env.file_server and then downloaded to the final destination from there. For now, its only oc-mirror. Rest of artifacts will be downloaded from urls
clients_dir: 'clients'
oc_mirror_tgz: 'oc-mirror.tar.gz' # name of oc-mirror plugin binary in clients_dir. should be a tar.gz file. You must place this in your ftp server after downloading it yourself from https://console.redhat.com/openshift/downloads for amd64 (or building it yourself if your mirror host is s390x)
client_download: # this will download oc binary to the mirror host for use on the mirror host for mirroring
ocp_download_url: "https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp/4.13.1/amd64/"
ocp_client_tgz: 'openshift-client-linux.tar.gz' # name of the oc binary. Should be a tar.gz file
legacy:
platform: True # if true then platform is mirrored in the old way and install config will be patched with the imagecontentsourcepolicy
ocp_quay_release_image_tag: '4.13.1-s390x'
ocp_org: 'ocp4'
ocp_repo: 'openshift4'
ocp_tag: 'v4.13.1' # platform images will be pushed to {tegistry_url}/{ocp_org}/{ocp_repo}:{ocp_tag}
oc_mirror:
oc_mirror_args:
continue_on_error: False
source_skip_tls: False
image_set: # this field is a standard image set from oc-mirror documentation. The only exception is the storageConfig which is altered to allow substitution of disconnected.registry.url
storageConfig:
registry:
enabled: True # use registry storage backend. Currently only method supproted
imageURL: # the final value will be {imageURL: disconnected.registry.url/org/repo}
org: mirror
repo: oc-mirror-metadata
skipTLS: false # standard field form oc-mirror schema
mirror: # this field is also atandard from the oc-mirror schema. It will be substituted as is into the final image set # WARNING: platform mirroring through oc-mirror is currently not supported, use legacy
operators:
- catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13
full: false
packages:
- name: serverless-operator
channels:
- name: stable
#minVersion: '2.4.1-0'
#maxVersion: '2.4.1-0'
additionalImages:
- name: registry.redhat.io/ubi8/ubi:latest
helm: {}

# Section 18 - (Optional) Create additional compute node in a day-2 operation

day2_compute_node:
vm_name:
vm_hostname:
vm_ip:
vm_ipv6:
vm_interface:
vm_interface:
hostname:
host_arch:
5 changes: 3 additions & 2 deletions mkdocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ repo_url: https://github.com/IBM/Ansible-OpenShift-Provisioning
site_url: https://ibm.github.io/Ansible-OpenShift-Provisioning/
edit_uri: edit/main/docs

nav:
nav:
- Home: 'index.md'
- Read Me:
- Before You Begin: 'before-you-begin.md'
Expand All @@ -13,6 +13,7 @@ nav:
- 2 Set Variables (group_vars): 'set-variables-group-vars.md'
- 3 Set Variables (host_vars): 'set-variables-host-vars.md'
- 4 Run the Playbooks: 'run-the-playbooks.md'
- Run the Playbooks (Disconnected): 'run-the-playbooks-for-disconnected.md'
- Run the Playbooks (HyperShift): 'run-the-playbooks-for-hypershift.md'
- Misc:
- Troubleshooting: 'troubleshooting.md'
Expand All @@ -37,4 +38,4 @@ markdown_extensions:
- toc:
permalink: "#"
- attr_list

7 changes: 7 additions & 0 deletions playbooks/0_setup.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@
roles:
- set_inventory

pre_tasks:
- name: Check disconnected variables if disconnected cluster will be installed.
tags: disconnected
ansible.builtin.include_role:
name: disconnected_check_vars
when: disconnected.enabled

post_tasks:
- name: Ensure Ansible Galaxy collections have been installed.
tags: galaxy
Expand Down
17 changes: 17 additions & 0 deletions playbooks/disconnected_apply_operator_manifests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
- name: Disconnected apply operator manifests
hosts: bastion
become: true
environment:
KUBECONFIG: "{{ '/home/' if (env.bastion.access.user != 'root') else '/'}}{{ env.bastion.access.user }}/.kube/config"
gather_facts: true
vars_files:
- "{{ inventory_dir }}/group_vars/all.yaml"
tasks:
- name: apply operator manifests when cluster is disconnected
ansible.builtin.include_role:
name: '{{ item }}'
loop:
- disconnected_check_vars
- disconnected_apply_operator_manifests_to_cluster
when: disconnected.enabled
14 changes: 14 additions & 0 deletions playbooks/disconnected_mirror_artifacts.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
- name: Mirror ocp artifacts
hosts: mirrorhost
gather_facts: false
vars_files:
- "{{ inventory_dir }}/group_vars/all.yaml"
tasks:
- name: mirror ocp artifacts when disconnected install
ansible.builtin.include_role:
name: '{{ item }}'
loop:
- disconnected_check_vars
- disconnected_mirror_images
when: disconnected.enabled
4 changes: 4 additions & 0 deletions playbooks/pre-existing_site.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@

- import_playbook: 0_setup.yaml
- import_playbook: 4_create_bastion.yaml
- import_playbook: disconnected_mirror_artifacts.yaml
when: disconnected.enabled
- import_playbook: 5_setup_bastion.yaml
- import_playbook: 6_create_nodes.yaml
- import_playbook: 7_ocp_verification.yaml
- import_playbook: disconnected_apply_operator_manifests.yaml
when: disconnected.enabled
Loading

0 comments on commit 1e81c50

Please sign in to comment.