Add support for manila in multinode

doc/source/configuration/cephadm.rst

@@ -242,19 +242,26 @@ for Cinder, Cinder backup, Glance, and Nova in Kolla Ansible.
 Ceph Commands
 ~~~~~~~~~~~~~
 
-It is possible to run an arbitrary list of commands against the cluster after deployment
-by setting the ``cephadm_commands`` variable. ``cephadm_commands`` should be a list of commands
-to pass to ``cephadm shell -- ceph``. For example:
+It is possible to run an arbitrary list of commands against the cluster after
+deployment by setting the ``cephadm_commands_pre`` and ``cephadm_commands_post``
+variables. Each should be a list of commands to pass to ``cephadm shell --
+ceph``. For example:
 
 .. code:: yaml
 
    # A list of commands to pass to cephadm shell -- ceph. See stackhpc.cephadm.commands
    # for format.
-   cephadm_commands:
+   cephadm_commands_pre:
     # Configure Prometheus exporter to listen on a specific interface. The default
     # is to listen on all interfaces.
     - "config set mgr mgr/prometheus/server_addr 10.0.0.1"
 
+Both variables have the same format, however commands in the
+``cephadm_commands_pre`` list are executed before the rest of the Ceph
+post-deployment configuration is applied. Commands in the
+``cephadm_commands_post`` list are executed after the rest of the Ceph
+post-deployment configuration is applied.
+
 Deployment
 ==========
 

doc/source/contributor/environments/ci-multinode.rst

@@ -53,3 +53,153 @@ Final steps
 1. ``source kayobe-env --environment ci-aio``
 2. Run ``kayobe overcloud host configure``
 3. Run ``kayobe overcloud service deploy``
+
+
+Manila
+======
+The Multinode environment supports Manila with the CephFS native backend, but it
+is not enabled by default. To enable it, set the following in
+``/etc/kayobe/environments/ci-multinode/kolla.yml``:
+
+.. code-block:: yaml
+
+      kolla_enable_manila: true
+      kolla_enable_manila_backend_cephfs: true
+
+And re-run ``kayobe overcloud service deploy`` if you are working on an existing
+deployment.
+
+To test it, you will need two virtual machines. Cirros does not support the Ceph
+kernel client, so you will need to use a different image. Any regular Linux
+distribution should work. As an example, we will use Ubuntu 20.04.
+
+Download the image locally:
+
+.. code-block:: bash
+
+      wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
+
+Upload the image to Glance:
+
+.. code-block:: bash
+
+      openstack image create --container-format bare --disk-format qcow2 --file focal-server-cloudimg-amd64.img Ubuntu-20.04 --progress
+
+Create a keypair:
+
+.. code-block:: bash
+
+      openstack keypair create --private-key ~/.ssh/id_rsa id_rsa
+
+Create two virtual machines from the image:
+
+.. code-block:: bash
+
+      openstack server create --flavor m1.small --image Ubuntu-20.04 --key-name id_rsa --network admin-tenant ubuntu-client-1
+      openstack server create --flavor m1.small --image Ubuntu-20.04 --key-name id_rsa --network admin-tenant ubuntu-client-2
+
+Wait until the instances are active. It is worth noting that this process can
+take a while, especially if the overcloud is deployed to virtual machines. You
+can monitor the progress with the following command:
+
+.. code-block:: bash
+
+      watch openstack server list
+
+Once they are active, create two floating IPs:
+
+.. code-block:: bash
+
+      openstack floating ip create external
+      openstack floating ip create external
+
+Associate the floating IPs to the instances:
+
+.. code-block:: bash
+
+      openstack server add floating ip ubuntu-client-1 <floating-ip-1>
+      openstack server add floating ip ubuntu-client-2 <floating-ip-2>
+
+
+Then SSH into each instance and install the Ceph client:
+
+.. code-block:: bash
+
+      sudo apt update
+      sudo apt install -y ceph-common
+
+
+Back on the host, install the Manila client:
+
+.. code-block:: bash
+
+      sudo dnf install -y python-manilaclient
+
+Then create a share type and share:
+
+.. code-block:: bash
+
+      manila type-create cephfs-type false --is_public true
+      manila type-key cephfs-type set vendor_name=Ceph storage_protocol=CEPHFS
+      manila create --name test-share --share-type cephfs-type CephFS 2
+
+Wait until the share is available:
+
+.. code-block:: bash
+
+      manila list
+
+Then allow access to the shares to two users:
+
+.. code-block:: bash
+
+      manila access-allow test-share cephx alice
+      manila access-allow test-share cephx bob
+
+Show the access list to make sure the state of both entries is ``active`` and
+take note of the access keys:
+
+.. code-block:: bash
+
+      manila access-list test-share
+
+And take note of the path to the share:
+
+.. code-block:: bash
+
+      manila share-export-location-list test-share
+
+SSH into the first instance, create a directory for the share, and mount it:
+
+.. code-block:: bash
+
+      mkdir testdir
+      sudo mount -t ceph {path} -o name=alice,secret='{access_key}' testdir
+
+Where the path is the path to the share from the previous step, and the secret
+is the access key for the user alice.
+
+Then create a file in the share:
+
+.. code-block:: bash
+
+      sudo touch testdir/testfile
+
+SSH into the second instance, create a directory for the share, and mount it:
+
+.. code-block:: bash
+
+      mkdir testdir
+      sudo mount -t ceph {path} -o name=bob,secret='{access_key}' testdir
+
+Where the path is the same as before, and the secret is the access key for the
+user bob.
+
+Then check that the file created in the first instance is visible in the second
+instance:
+
+.. code-block:: bash
+
+      ls testdir
+
+If it shows the test file then the share is working correctly.

etc/kayobe/ansible/cephadm-commands.yml → etc/kayobe/ansible/cephadm-commands-post.yml

@@ -1,5 +1,5 @@
 ---
-- name: Ensure additional Ceph commands are run
+- name: Ensure additional Ceph commands are run after post-deployment configuration
   gather_facts: false
   hosts: mons
   become: true
@@ -9,3 +9,5 @@
   tasks:
     - import_role:
         name: stackhpc.cephadm.commands
+      vars:
+        cephadm_commands: "{{ cephadm_commands_post }}"

etc/kayobe/ansible/cephadm-commands-pre.yml

@@ -0,0 +1,13 @@
+---
+- name: Ensure additional Ceph commands are run before post-deployment configuration
+  gather_facts: false
+  hosts: mons
+  become: true
+  tags:
+    - cephadm
+    - cephadm-commands
+  tasks:
+    - import_role:
+        name: stackhpc.cephadm.commands
+      vars:
+        cephadm_commands: "{{ cephadm_commands_pre }}"

etc/kayobe/ansible/cephadm.yml

@@ -1,8 +1,9 @@
 ---
 # Deploy Ceph via Cephadm. Create EC profiles, CRUSH rules, pools and keys.
 - import_playbook: cephadm-deploy.yml
-- import_playbook: cephadm-commands.yml
+- import_playbook: cephadm-commands-pre.yml
 - import_playbook: cephadm-ec-profiles.yml
 - import_playbook: cephadm-crush-rules.yml
 - import_playbook: cephadm-pools.yml
 - import_playbook: cephadm-keys.yml
+- import_playbook: cephadm-commands-post.yml

etc/kayobe/cephadm.yml

@@ -60,9 +60,12 @@ cephadm_cluster_network: "{{ storage_mgmt_net_name | net_cidr }}"
 # List of Cephx keys. See stackhpc.cephadm.keys role for format.
 #cephadm_keys:
 
-# A list of commands to pass to cephadm shell -- ceph. See stackhpc.cephadm.commands
-# for format.
-#cephadm_commands:
+# A list of commands to pass to cephadm shell -- ceph. See
+# stackhpc.cephadm.commands for format. Pre commands run before the rest of the
+# post-deployment configuration, post commands run after the rest of the
+# post-deployment configuration.
+#cephadm_commands_pre:
+#cephadm_commands_post:
 
 ###############################################################################
 # Kolla Ceph auto-configuration.

etc/kayobe/environments/ci-multinode/cephadm.yml

@@ -35,6 +35,12 @@ cephadm_pools:
   - name: vms
     application: rbd
     state: present
+  - name: cephfs_data
+    application: cephfs
+    state: "{{ 'present' if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else 'absent' }}"
+  - name: cephfs_metadata
+    application: cephfs
+    state: "{{ 'present' if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else 'absent' }}"
 
 # List of Cephx keys. See stackhpc.cephadm.keys role for format.
 cephadm_keys:
@@ -56,3 +62,16 @@ cephadm_keys:
       osd: "profile rbd pool=images"
       mgr: "profile rbd pool=images"
     state: present
+  - name: client.manila
+    caps:
+      mon: "allow r"
+      mgr: "allow rw"
+    state: "{{ 'present' if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else 'absent' }}"
+
+# List of Cephadm commands to run. See stackhpc.cephadm.commands role for format.
+cephadm_commands_pre: []
+
+cephadm_commands_post: "{{ cephadm_commands_manila_cephfs_native if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else [] }}"
+cephadm_commands_manila_cephfs_native:
+  - "fs new manila-cephfs cephfs_metadata cephfs_data"
+  - "orch apply mds manila-cephfs"

etc/kayobe/environments/ci-multinode/inventory/group_vars/seed/network-interfaces

@@ -10,6 +10,10 @@ external_interface: "{{ vxlan_interfaces[0].device }}.{{ external_vlan }}"
 
 public_interface: "{{ vxlan_interfaces[0].device }}.{{ public_vlan }}"
 
+# The storage interface is required for routing manila traffic between VMs and
+# the Storage nodes.
+storage_interface: "{{ vxlan_interfaces[0].device }}.{{ storage_vlan }}"
+
 ###############################################################################
 # Dummy variable to allow Ansible to accept this file.
 workaround_ansible_issue_8743: yes

etc/kayobe/environments/ci-multinode/kolla.yml

@@ -4,3 +4,7 @@ kolla_enable_cinder_backup: true
 kolla_enable_neutron_provider_networks: true
 kolla_enable_ovn: true
 kolla_enable_octavia: true
+
+# The multinode environment supports Manila but it is not enabled by default.
+# kolla_enable_manila: true
+# kolla_enable_manila_backend_cephfs_native: true

etc/kayobe/environments/ci-multinode/kolla/globals.yml

@@ -22,3 +22,6 @@ es_heap_size: 1g
 octavia_auto_configure: "no"
 octavia_provider_drivers: "ovn:OVN provider"
 octavia_provider_agents: "ovn"
+
+# Manila CephFS configuration
+manila_cephfs_filesystem_name: manila-cephfs

etc/kayobe/environments/ci-multinode/seed.yml

@@ -3,12 +3,29 @@ seed_bootstrap_user: "{{ os_distribution if os_distribution == 'ubuntu' else 'cl
 seed_lvm_groups:
   - "{{ stackhpc_lvm_group_rootvg }}"
 
+seed_extra_network_interfaces: >
+  "{{ seed_extra_network_interfaces_external +
+  (seed_extra_network_interfaces_manila if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else []) }}"
+
 # Seed has been provided an external interface
 # for tempest tests and SSH access to machines.
-seed_extra_network_interfaces:
+seed_extra_network_interfaces_external:
   - "external"
   - "public"
 
+# Seed requires access to the storage network for manila-cephfs.
+seed_extra_network_interfaces_manila:
+  - "storage"
+
 # Enable IP routing and source NAT on the seed, allowing it to be used as the
 # external subnet gateway and provide internet access for VMs in the deployment.
 seed_enable_snat: true
+
+snat_rules_default:
+  - interface: "{{ ansible_facts.default_ipv4.interface }}"
+    source_ip: "{{ ansible_facts.default_ipv4.address }}"
+snat_rules_manila:
+  - interface: "{{ storage_interface }}"
+    source_ip: "{{ ansible_facts[storage_interface].ipv4.address }}"
+# Only add the storage snat rule if we are using manila-cephfs.
+snat_rules: "{{ snat_rules_default + snat_rules_manila if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else snat_rules_default }}"

releasenotes/notes/manila-for-multinode-df2186276c6b434a.yaml

@@ -0,0 +1,8 @@
+---
+features:
+  - |
+    Adds support for Manila in the ci-multinode environment using the CephFS
+    native backend. This is disabled by default, but can be enabled by setting
+    the following variables in the kayobe configuration:
+    `kolla_enable_manila: true`
+    `kolla_enable_manila_backend_cephfs_native: true`

releasenotes/notes/split-cephadm-commands-pre-post-bddd25b8bd171ab0.yaml

@@ -0,0 +1,8 @@
+---
+features:
+  - |
+    Split `cephadm_commands` into `cephadm_commands_pre` and
+    `cephadm_commands_post` commands. This allows the user to run commands that
+    must be run before the rest of the post-deployment configuration, as well
+    as commands that rely on resources created by the post-deployment config.
+