feat: Add NAT-based networking support besides macvtap (#77)

This code change adds NAT support. In case of KVM NAT network
mode and an installation from remote (e.g. from your laptop). In case of
NAT the IP of the bastion and cluster nodes will use 192.168.x.x adr.
which are usually not accessible from remote. To access those addresses
an SSH tunnel is needed where usually the KVM host acts as a jumphost.
These new features are adding new variables to the all.yaml to specify the
network mode (NAT) and the required variables for the jumphost. If 
network mode set to NAT, a section will be added to the SSH config file 
containing the bastion node and the matching jumphost.

In addition, many small other fixes and updates related to enabling NAT-
based networking are included. Please read through these changes,
as they most likely will cause errors if you leave the new variables
undefined, etc.

---------

Signed-off-by: Amadeus Podvratnik <pod@de.ibm.com>
Signed-off-by: Jacob Emery <jacob.emery@ibm.com>
Co-authored-by: Klaus Smolin <88041391+smolin-de@users.noreply.github.com>
Co-authored-by: Jacob Emery <jacob.emery@ibm.com>

docs/prerequisites.md

@@ -55,7 +55,16 @@ ansible-galaxy collection install ibm.ibm_zhmc
 ```
 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
 ```
-  * and [Xcode](https://apps.apple.com/us/app/xcode/id497799835?mt=12):
+  * [Xcode](https://apps.apple.com/us/app/xcode/id497799835?mt=12):
 ```
 xcode-select --install
-```
+```
+## Jumphost for NAT network
+* If for KVM network NAT is used, instead of macvtap, a ssh tunnel using a jumphost is required to access the OCP cluster. To configure the ssh tunnel expect is required on the jumphost. Expect will be installed during the setup of the bastion (4_setup_bastion.yaml playbook). In case of missing access to install additional packages, install it manually on the jumphost by executing following command:
+```
+yum install expect 
+```
+In addition make sure that python3 is installed on the jumphost otherwise ansible might fail to run the tasks. You can install python3 manually by executing the following command:
+```
+yum install python3 
+```

docs/set-variables-group-vars.md

@@ -16,6 +16,7 @@
 **Variable Name** | **Description** | **Example**
 :--- | :--- | :---
 **env.z.high_availability** | Is this cluster spread across three LPARs? If yes, mark True. If not (just in<br /> one LPAR), mark False | True
+**env.z.ip_forward** | This variable specifies if ip forwarding is enabled or not if NAT network is selected. If ip_forwarding is set to 0, the installed OCP cluster will not be able to access external services. This setting will be configured during 3_setup_kvm playbook. If NAT will be configured after 3_setup_kvm playbook, the setup needs to be done manually before bastion is being created, configured or reconfigured by running the 3_setup_kvm playbook with parameter: --tags cfg_ip_forward | 1
 **env.z.lpar1.create** | To have Ansible create an LPAR and install RHEL on it for the KVM<br /> host, mark True. If using a pre-existing LPAR with RHEL already<br /> installed, mark False. | True
 **env.z.lpar1.hostname** | The hostname of the KVM host. | kvm-host-01
 **env.z.lpar1.ip** | The IPv4 address of the KVM host. | 192.168.10.1
@@ -59,12 +60,13 @@
 **env.bastion.resources.vcpu** | How many virtual CPUs would you like to allocate to the bastion? Recommended 4 or more. | 4
 **env.bastion.networking.ip** | IPv4 address for the bastion. | 192.168.10.3
 **env.bastion.networking.hostname** | Hostname of the bastion. Will be combined with<br /> env.bastion.networking.base_domain to create a Fully Qualified Domain Name (FQDN). | ocpz-bastion
+**env.bastion.networking.base_<br />domain** | Base domain that, when combined with the hostname, creates a fully-qualified<br /> domain name (FQDN) for the bastion? | ihost.com
 **env.bastion.networking.<br />subnetmask** | Subnet of the bastion. | 255.255.255.0
 **env.bastion.networking.gateway** | IPv4 of he bastion's gateway server. | 192.168.10.0
 **env.bastion.networking.name<br />server1** | IPv4 address of the server that resolves the bastion's hostname. | 192.168.10.200
 **env.bastion.networking.name<br />server2** | <b>(Optional)</b> A second IPv4 address that resolves the bastion's hostname. | 192.168.10.201
+**env.bastion.networking.forwarder** | What IPv4 address will be used to make external DNS calls for the bastion? Can use 1.1.1.1 or 8.8.8.8 as defaults. | 8.8.8.8
 **env.bastion.networking.interface** | Name of the networking interface on the bastion from Linux's perspective. Most likely enc1. | enc1
-**env.bastion.networking.base_<br />domain** | Base domain that, when combined with the hostname, creates a fully-qualified<br /> domain name (FQDN) for the bastion? | ihost.com
 **env.bastion.access.user** | What would you like the admin's username to be on the bastion?<br /> If root, make pass and root_pass vars the same. | admin
 **env.bastion.access.pass** | The password to the bastion's admin user. If using root, make<br /> pass and root_pass vars the same. | cH4ngeM3!
 **env.bastion.access.root_pass** | The root password for the bastion. If using root, make<br /> pass and root_pass vars the same. | R0OtPa$s!
@@ -80,7 +82,7 @@
 **env.cluster.networking.base_domain** | The site name, where is the cluster being hosted? This will be combined with the metadata_name<br /> and hostnames to create FQDNs.  | ihost.com
 **env.cluster.networking.nameserver1** | IPv4 address that the cluster get its hostname resolution from. If env.bastion.options.dns<br /> is True, this should be the IP address of the bastion. | 192.168.10.200
 **env.cluster.networking.nameserver2** | <b>(Optional)</b> A second IPv4 address will the cluster get its hostname resolution from? If env.bastion.options.dns<br /> is True, this should be left commented out. | 192.168.10.201
-**env.cluster.networking.forwarder** | What IPv4 address will be used to make external DNS calls? Can use 1.1.1.1 or 8.8.8.8 as defaults. | 8.8.8.8
+**env.cluster.networking.forwarder** | What IPv4 address will be used to make external DNS calls for the cluster? Can use 1.1.1.1 or 8.8.8.8 as defaults. | 8.8.8.8
 
 ## 7 - Bootstrap Node
 **Variable Name** | **Description** | **Example**
@@ -159,4 +161,10 @@
 **env.timezone** | Which timezone would you like Red Hat Enterprise Linux to use? A list of available timezone<br /> options can be found [here](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). | America/New_York
 **env.ansible_key_name** | (Optional) Name of the SSH key that Ansible will use to connect to hosts. | ansible-ocpz
 **env.ocp_key_name** | Comment to describe the SSH key used for OCP. Arbitrary value. | OCPZ-01 key
-**env.bridge_name** | (Optional) Name of the macvtap bridge that will be created on the KVM host. | macvtap-net
+**env.bridge_name** | (Optional) Name of the macvtap bridge that will be created on the KVM host or in case of NAT the name of the NAT network defenition (usually it is 'default'). If NAT is being used and a jumphost is needed, the parameters network_mode, jumphost.name, jumphost.user and jumphost.pass must be specified, too. In case of default (NAT) network verify that the configured IP ranges does not interfere with the IPs defined for the controle and compute nodes. Modify the default network (dhcp range setting) to prevent issues with VMs using dhcp and OCP nodes having fixed IPs.| macvtap-net
+**env.network_mode** | (Optional) In case the network mode will be NAT and the installation will be executed from remote (e.g. your laptop), a jumphost needs to be defined to let the installation access the bastion host. If macvtap for networking is being used this variable should be empty. | NAT
+**env.jumphost.name** | (Optional) If env.network.mode is set to 'NAT' the name of the jumphost (e.g. the name of KVM host if used as jumphost) should be specified. | kvm-host-01
+**env.jumphost.ip** | (Optional) The ip of the jumphost. | 192.168.10.1
+**env.jumphost.user** | (Optional) The user name to login to the jumphost. | admin
+**env.jumphost.pass** | (Optional) The password for user to login to the jumphost. | ch4ngeMe!
+**env.jumphost.path_to_keypair** | (Optional) The absolute path to the public key file on the jumphost to be copied to the bastion. | /home/admin/.ssh/id_rsa.pub

inventories/default/group_vars/all.yaml.template

@@ -11,6 +11,7 @@ env:
 # Section 2 - LPAR(s)
   z:
     high_availability: False
+    ip_forward: #X
     lpar1:
       create: True
       hostname: #X
@@ -57,12 +58,13 @@ env:
     networking:
       ip: #X
       hostname: #X
+      base_domain: #X
       subnetmask: #X
       gateway: #X
       nameserver1: #X
 #      nameserver2:
+      forwarder: 1.1.1.1
       interface: #X
-      base_domain: #X
     access:
       user: #X
       pass: #X
@@ -79,6 +81,8 @@ env:
     networking:
       metadata_name: #X
       base_domain: #X
+      subnetmask: #X
+      gateway: #X
       nameserver1: #X
 #      nameserver2:
       forwarder: 1.1.1.1
@@ -148,7 +152,7 @@ env:
 # Section 11 - (Optional) Packages
   pkgs:
     galaxy: [ ibm.ibm_zhmc, community.general, community.crypto, ansible.posix, community.libvirt ]
-    controller: [ openssh, expect ]
+    controller: [ openssh, expect, sshuttle ]
     kvm: [ libguestfs, libvirt-client, libvirt-daemon-config-network, libvirt-daemon-kvm, cockpit-machines, libvirt-devel, virt-top, qemu-kvm, python3-lxml, cockpit, lvm2 ]
     bastion: [ haproxy, httpd, bind, bind-utils, expect, firewalld, mod_ssl, python3-policycoreutils, rsync ]
 
@@ -183,4 +187,12 @@ env:
   ansible_key_name: ansible-ocpz
   ocp_ssh_key_comment: OpenShift key
   bridge_name: macvtap
-
+  network_mode: 
+
+#jumphost if network mode is NAT
+  jumphost:     
+    name: 
+    ip:
+    user: 
+    pass:  
+    path_to_keypair:

playbooks/0_setup.yaml

@@ -59,7 +59,7 @@
     - "{{ inventory_dir }}/group_vars/all.yaml"
   vars:
     packages: "{{ env.pkgs.controller }}"
-    ssh_target: [ "{{ env.ftp.ip }}", "{{ env.ftp.user }}", "{{ env.ftp.pass }}" ]
+    ssh_target: [ "{{ env.ftp.ip }}", "{{ env.ftp.user }}", "{{ env.ftp.pass }}", "{{ path_to_key_pair }}" ]
   roles:
     - install_packages
     - ssh_key_gen

playbooks/3_setup_kvm_host.yaml

@@ -9,7 +9,7 @@
   vars_files: 
     - "{{ inventory_dir }}/group_vars/all.yaml"
   vars:
-    ssh_target: ["{{ env.z.lpar1.ip }}","{{ env.z.lpar1.user }}","{{ env.z.lpar1.pass }}"]
+    ssh_target: ["{{ env.z.lpar1.ip }}","{{ env.z.lpar1.user }}","{{ env.z.lpar1.pass }}","{{ path_to_key_pair }}"]
   tasks:
     - name: Include vars for the KVM host.
       include_vars:
@@ -27,7 +27,7 @@
   vars_files: 
     - "{{ inventory_dir }}/group_vars/all.yaml"
   vars:
-    ssh_target: ["{{ env.z.lpar2.ip }}","{{ env.z.lpar2.user }}","{{ env.z.lpar2.pass }}"]
+    ssh_target: ["{{ env.z.lpar2.ip }}","{{ env.z.lpar2.user }}","{{ env.z.lpar2.pass }}","{{ path_to_key_pair }}"]
   tasks:
     - name: Include vars for second KVM host.
       include_vars:
@@ -48,7 +48,7 @@
   vars_files: 
     - "{{ inventory_dir }}/group_vars/all.yaml"
   vars:
-    ssh_target: ["{{ env.z.lpar3.ip }}","{{ env.z.lpar3.user }}","{{ env.z.lpar3.pass }}"]
+    ssh_target: ["{{ env.z.lpar3.ip }}","{{ env.z.lpar3.user }}","{{ env.z.lpar3.pass }}","{{ path_to_key_pair }}"]
   tasks:
     - name: Include vars for third KVM host.
       include_vars:
@@ -67,23 +67,59 @@
   gather_facts: true
   become: true
   vars: 
-    packages: "{{ env.pkgs.kvm}}"
+    packages: "{{ env.pkgs.kvm }}"
   roles:
     - { role: attach_subscription, when: env.redhat.username is defined and env.redhat.password is defined }
     - install_packages
+    - httpd
   post_tasks:
-    - name: Enable cockpit console
-      command: systemctl enable --now cockpit.socket
-
+    - name: Add ports to firewall
+      tags: firewall-libvirt
+      ansible.posix.firewalld:
+        port: 80/tcp
+        permanent: yes
+        state: enabled
+
     - name: Start and enable libvirt
-      service:
+      tags: firewall-libvirt
+      ansible.builtin.service:
         name: libvirtd
         enabled: yes
         state: started
 
+    - name: Permit traffic in libvirt zone
+      tags: firewall-libvirt
+      ansible.posix.firewalld:
+        service: http
+        permanent: yes
+        state: enabled
+        zone: libvirt
+        immediate: true
+
+    - name: Enable cockpit console
+      ansible.builtin.command: systemctl enable --now cockpit.socket
+
+- name: Configure ip_forward in case of NAT
+  hosts: kvm_host
+  tags: cfg_ip_forward, section_3
+  gather_facts: true
+  become: true
+  vars_files:
+    - "{{ inventory_dir }}/group_vars/all.yaml"
+  tasks:
+    - name: Configure ip_forward in case of network "NAT"
+      tags: cfg_ip_forward
+      ansible.posix.sysctl:
+        name: net.ipv4.ip_forward
+        value: "{{ env.z.ip_forward }}"
+        sysctl_set: true
+        state: present
+        reload: true
+      when: env.network_mode | upper == 'NAT' 
+
 - hosts: kvm_host
   tags: setup, section_3
   become: true
   roles:
     - configure_storage
-    - macvtap
+    - { role: macvtap, when: env.network_mode | upper != 'NAT' }

playbooks/5_setup_bastion.yaml

@@ -1,6 +1,6 @@
 ---
 
-- name: 5 setup bastion - copy SSH key to access bastion
+- name: Copy ssh key to jumphost if network is NAT and jumphost defined, and add jumphost section to ssh config.
   hosts: localhost
   tags: ssh, ssh_copy_id, section_1
   connection: local
@@ -9,7 +9,54 @@
   vars_files:
     - "{{ inventory_dir }}/group_vars/all.yaml"
   vars:
-    ssh_target: ["{{ env.bastion.networking.ip }}", "{{ env.bastion.access.user }}", "{{ env.bastion.access.pass }}"]
+    ssh_target: ["{{ env.jumphost.ip }}", "{{ env.jumphost.user }}", "{{ env.jumphost.pass }}", "{{ path_to_key_pair }}"]
+  roles:
+    - { role: ssh_copy_id, tags: ssh_copy_id, ssh, when: (env.network_mode | upper == "NAT") and ( env.jumphost.ip is not none ) }
+    - { role: ssh_add_config, tags: ssh_copy_id, ssh, when: (env.network_mode | upper == "NAT") and ( env.jumphost.ip is not none ) }
+
+- name: Configure jumphost if network mode == 'NAT'
+  hosts: jumphost
+  tags: ssh, ssh_copy_id, section_1
+  become: false
+  gather_facts: true
+  vars_files:
+    - "{{ inventory_dir }}/group_vars/all.yaml"
+  vars:
+    ssh_target: ["{{ env.bastion.networking.ip }}", "{{ env.bastion.access.user }}", "{{ env.bastion.access.pass }}","{{ env.jumphost.path_to_keypair }}"]
+  pre_tasks:
+    - name: Generate an OpenSSH keypair with the default values (4096 bits, RSA), if using jumphost for NAT.
+      become: false
+      tags: ssh_key_gen, ssh, section_1
+      community.crypto.openssh_keypair:
+        path: "{{ env.jumphost.path_to_keypair.split('.')[:-1] | join('.') }}"
+        passphrase: ""
+        regenerate: never
+      when: (env.network_mode | upper == "NAT") and ( env.jumphost.ip is not none )
+    - block:
+        - name: Check if 'expect' is installed on jumphost, for use in ssh-copy-id role for NAT.
+          package_facts:
+          failed_when: "'expect' not in ansible_facts.packages"
+          when: (env.network_mode | upper == "NAT") and ( env.jumphost.ip is not none )
+      rescue:
+        - name: Package 'expect' must be installed on the jumphost, attempting to install it. #Using 'block' and 'rescue' to avoid running the 'package' module (which requires 'sudo') unless necessary.
+          become: true
+          package:
+            name: expect
+          when: (env.network_mode | upper == "NAT") and ( env.jumphost.ip is not none )
+  roles:
+    - { role: ssh_copy_id, ssh, when: (env.network_mode | upper == "NAT") and ( env.jumphost.ip is not none ) }
+  post_tasks:
+    - meta: clear_facts
+
+- name: 5 setup bastion - copy SSH key from localhost to access bastion.
+  hosts: localhost
+  tags: ssh, ssh_copy_id, section_1
+  become: false
+  gather_facts: true
+  vars_files:
+    - "{{ inventory_dir }}/group_vars/all.yaml"
+  vars:
+    ssh_target: ["{{ env.bastion.networking.ip }}", "{{ env.bastion.access.user }}", "{{ env.bastion.access.pass }}","{{ path_to_key_pair }}"]
   roles:
     - ssh_copy_id
 
@@ -21,6 +68,10 @@
     packages: "{{ env.pkgs.bastion }}"
   vars_files:
     - "{{ inventory_dir }}/group_vars/all.yaml"
+  pre_tasks:
+    - import_role:
+        name: dns
+        tasks_from: initial-resolv.yaml
   roles:
     - { role: attach_subscription, when: env.redhat.username is defined and env.redhat.password is defined }
     - install_packages
@@ -85,6 +136,7 @@
       loop:
         - issued
         - private
+      when: env.z.high_availability == True
 
     - name: Copy certificates and keys from controller to KVM hosts.
       tags: openvpn
@@ -112,6 +164,7 @@
       file:
         state: absent
         path: tmp
+      when: env.z.high_availability == True
 
 - hosts: bastion
   tags: get_ocp, section_3

roles/create_bastion/tasks/main.yaml

@@ -53,7 +53,7 @@
   ansible.builtin.command: pwd
   register: kvm_host_home
 
-- name: Boot and kickstart bastion (up to 3 min). To monitor, login to your KVM host and run 'virsh console <bastion VM name>'
+- name: Boot and kickstart bastion. To monitor, login to your KVM host and run 'virsh console <bastion VM name>'
   tags: create_bastion, virt-install
   ansible.builtin.shell: |
     set -o pipefail
@@ -72,8 +72,3 @@
     --initrd-inject "/{{ kvm_host_home.stdout }}/{{ env.ftp.cfgs_dir }}/{{ env.bastion.networking.hostname }}/bastion-ks.cfg" \
     --extra-args "inst.ks=file:/bastion-ks.cfg ip={{ env.bastion.networking.ip }}::{{ env.bastion.networking.gateway }}\
     :{{ env.bastion.networking.subnetmask }}:{{ env.bastion.networking.hostname }}:enc1:none console=ttysclp0"
-
-- name: Waiting 1 minute for automated bastion installation and configuration to complete
-  tags: create_bastion, virt-install
-  ansible.builtin.pause:
-    minutes: 1

roles/create_bastion/templates/bastion-ks.cfg.j2

@@ -34,7 +34,7 @@ eula --agreed
 
 # Network information
 network --bootproto=static --device={{ env.bastion.networking.interface }} --ip={{ env.bastion.networking.ip }} --gateway={{ env.bastion.networking.gateway }} --netmask={{ env.bastion.networking.subnetmask }} --noipv6 --nameserver={{ env.bastion.networking.nameserver1 }}{{ (',' + env.bastion.networking.nameserver2) if env.bastion.networking.nameserver2 is defined else '' }} --activate
-network --hostname={{ env.bastion.networking.hostname }}.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}
+network --hostname={{ env.bastion.networking.hostname }}.{{ env.cluster.networking.base_domain }}
 
 # Firewall and SELinux
 firewall --enabled --http --ftp --smtp --ssh --port=443,9090
@@ -74,6 +74,7 @@ python3-pip
 rsync
 vim
 wget
+network-scripts
 %end
 
 %addon com_redhat_kdump --disable

roles/create_bootstrap/tasks/main.yaml

@@ -18,7 +18,7 @@
     --location /var/lib/libvirt/images,kernel=rhcos-live-kernel-{{ env.openshift.version }}-{{ env.install_config.control.architecture }},initrd=rhcos-live-initramfs-{{ env.openshift.version }}-{{ env.install_config.control.architecture }}.img \
     --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda \
     coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/rhcos-live-rootfs-{{ env.openshift.version }}-{{ env.install_config.control.architecture }}.img \
-    ip={{ env.cluster.nodes.bootstrap.ip }}::{{ networking.gateway }}:{{ networking.subnetmask }}:{{ env.cluster.nodes.bootstrap.hostname }}::none:1500 \
+    ip={{ env.cluster.nodes.bootstrap.ip }}::{{ env.cluster.networking.gateway }}:{{ env.cluster.networking.subnetmask }}:{{ env.cluster.nodes.bootstrap.hostname }}.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}::none:1500 \
     nameserver={{ env.cluster.networking.nameserver1 }} {{ ('--nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }} \
     coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/bootstrap.ign" \
     --graphics none \
@@ -28,9 +28,9 @@
 - name: Set bootstrap qcow2 permissions
   become: true
   tags: create_bootstrap
-  command: chmod 600 /var/lib/libvirt/images/{{env.cluster.nodes.bootstrap.vm_name}}.qcow2
+  command: chmod 600 /var/lib/libvirt/images/{{ env.cluster.nodes.bootstrap.vm_name }}.qcow2
 
 - name: Set bootstrap qcow2 ownership to qemu
   become: true
   tags: create_bootstrap
-  command: chown qemu:qemu /var/lib/libvirt/images/{{env.cluster.nodes.bootstrap.vm_name}}.qcow2
+  command: chown qemu:qemu /var/lib/libvirt/images/{{ env.cluster.nodes.bootstrap.vm_name }}.qcow2