diff --git a/doc/source/configuration/cephadm.rst b/doc/source/configuration/cephadm.rst index 0322482c0..dfab2b748 100644 --- a/doc/source/configuration/cephadm.rst +++ b/doc/source/configuration/cephadm.rst @@ -242,19 +242,26 @@ for Cinder, Cinder backup, Glance, and Nova in Kolla Ansible. Ceph Commands ~~~~~~~~~~~~~ -It is possible to run an arbitrary list of commands against the cluster after deployment -by setting the ``cephadm_commands`` variable. ``cephadm_commands`` should be a list of commands -to pass to ``cephadm shell -- ceph``. For example: +It is possible to run an arbitrary list of commands against the cluster after +deployment by setting the ``cephadm_commands_pre`` and ``cephadm_commands_post`` +variables. Each should be a list of commands to pass to ``cephadm shell -- +ceph``. For example: .. code:: yaml # A list of commands to pass to cephadm shell -- ceph. See stackhpc.cephadm.commands # for format. - cephadm_commands: + cephadm_commands_pre: # Configure Prometheus exporter to listen on a specific interface. The default # is to listen on all interfaces. - "config set mgr mgr/prometheus/server_addr 10.0.0.1" +Both variables have the same format, however commands in the +``cephadm_commands_pre`` list are executed before the rest of the Ceph +post-deployment configuration is applied. Commands in the +``cephadm_commands_post`` list are executed after the rest of the Ceph +post-deployment configuration is applied. + Deployment ========== diff --git a/doc/source/contributor/environments/ci-multinode.rst b/doc/source/contributor/environments/ci-multinode.rst index 0dd787140..e42d61807 100644 --- a/doc/source/contributor/environments/ci-multinode.rst +++ b/doc/source/contributor/environments/ci-multinode.rst @@ -53,3 +53,153 @@ Final steps 1. ``source kayobe-env --environment ci-aio`` 2. Run ``kayobe overcloud host configure`` 3. Run ``kayobe overcloud service deploy`` + + +Manila +====== +The Multinode environment supports Manila with the CephFS native backend, but it +is not enabled by default. To enable it, set the following in +``/etc/kayobe/environments/ci-multinode/kolla.yml``: + +.. code-block:: yaml + + kolla_enable_manila: true + kolla_enable_manila_backend_cephfs: true + +And re-run ``kayobe overcloud service deploy`` if you are working on an existing +deployment. + +To test it, you will need two virtual machines. Cirros does not support the Ceph +kernel client, so you will need to use a different image. Any regular Linux +distribution should work. As an example, we will use Ubuntu 20.04. + +Download the image locally: + +.. code-block:: bash + + wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img + +Upload the image to Glance: + +.. code-block:: bash + + openstack image create --container-format bare --disk-format qcow2 --file focal-server-cloudimg-amd64.img Ubuntu-20.04 --progress + +Create a keypair: + +.. code-block:: bash + + openstack keypair create --private-key ~/.ssh/id_rsa id_rsa + +Create two virtual machines from the image: + +.. code-block:: bash + + openstack server create --flavor m1.small --image Ubuntu-20.04 --key-name id_rsa --network admin-tenant ubuntu-client-1 + openstack server create --flavor m1.small --image Ubuntu-20.04 --key-name id_rsa --network admin-tenant ubuntu-client-2 + +Wait until the instances are active. It is worth noting that this process can +take a while, especially if the overcloud is deployed to virtual machines. You +can monitor the progress with the following command: + +.. code-block:: bash + + watch openstack server list + +Once they are active, create two floating IPs: + +.. code-block:: bash + + openstack floating ip create external + openstack floating ip create external + +Associate the floating IPs to the instances: + +.. code-block:: bash + + openstack server add floating ip ubuntu-client-1 + openstack server add floating ip ubuntu-client-2 + + +Then SSH into each instance and install the Ceph client: + +.. code-block:: bash + + sudo apt update + sudo apt install -y ceph-common + + +Back on the host, install the Manila client: + +.. code-block:: bash + + sudo dnf install -y python-manilaclient + +Then create a share type and share: + +.. code-block:: bash + + manila type-create cephfs-type false --is_public true + manila type-key cephfs-type set vendor_name=Ceph storage_protocol=CEPHFS + manila create --name test-share --share-type cephfs-type CephFS 2 + +Wait until the share is available: + +.. code-block:: bash + + manila list + +Then allow access to the shares to two users: + +.. code-block:: bash + + manila access-allow test-share cephx alice + manila access-allow test-share cephx bob + +Show the access list to make sure the state of both entries is ``active`` and +take note of the access keys: + +.. code-block:: bash + + manila access-list test-share + +And take note of the path to the share: + +.. code-block:: bash + + manila share-export-location-list test-share + +SSH into the first instance, create a directory for the share, and mount it: + +.. code-block:: bash + + mkdir testdir + sudo mount -t ceph {path} -o name=alice,secret='{access_key}' testdir + +Where the path is the path to the share from the previous step, and the secret +is the access key for the user alice. + +Then create a file in the share: + +.. code-block:: bash + + sudo touch testdir/testfile + +SSH into the second instance, create a directory for the share, and mount it: + +.. code-block:: bash + + mkdir testdir + sudo mount -t ceph {path} -o name=bob,secret='{access_key}' testdir + +Where the path is the same as before, and the secret is the access key for the +user bob. + +Then check that the file created in the first instance is visible in the second +instance: + +.. code-block:: bash + + ls testdir + +If it shows the test file then the share is working correctly. diff --git a/etc/kayobe/ansible/cephadm-commands.yml b/etc/kayobe/ansible/cephadm-commands-post.yml similarity index 52% rename from etc/kayobe/ansible/cephadm-commands.yml rename to etc/kayobe/ansible/cephadm-commands-post.yml index 32742a343..fbca30dfd 100644 --- a/etc/kayobe/ansible/cephadm-commands.yml +++ b/etc/kayobe/ansible/cephadm-commands-post.yml @@ -1,5 +1,5 @@ --- -- name: Ensure additional Ceph commands are run +- name: Ensure additional Ceph commands are run after post-deployment configuration gather_facts: false hosts: mons become: true @@ -9,3 +9,5 @@ tasks: - import_role: name: stackhpc.cephadm.commands + vars: + cephadm_commands: "{{ cephadm_commands_post }}" diff --git a/etc/kayobe/ansible/cephadm-commands-pre.yml b/etc/kayobe/ansible/cephadm-commands-pre.yml new file mode 100644 index 000000000..08047cc51 --- /dev/null +++ b/etc/kayobe/ansible/cephadm-commands-pre.yml @@ -0,0 +1,13 @@ +--- +- name: Ensure additional Ceph commands are run before post-deployment configuration + gather_facts: false + hosts: mons + become: true + tags: + - cephadm + - cephadm-commands + tasks: + - import_role: + name: stackhpc.cephadm.commands + vars: + cephadm_commands: "{{ cephadm_commands_pre }}" diff --git a/etc/kayobe/ansible/cephadm.yml b/etc/kayobe/ansible/cephadm.yml index 1cb4d56e9..b699df153 100644 --- a/etc/kayobe/ansible/cephadm.yml +++ b/etc/kayobe/ansible/cephadm.yml @@ -1,8 +1,9 @@ --- # Deploy Ceph via Cephadm. Create EC profiles, CRUSH rules, pools and keys. - import_playbook: cephadm-deploy.yml -- import_playbook: cephadm-commands.yml +- import_playbook: cephadm-commands-pre.yml - import_playbook: cephadm-ec-profiles.yml - import_playbook: cephadm-crush-rules.yml - import_playbook: cephadm-pools.yml - import_playbook: cephadm-keys.yml +- import_playbook: cephadm-commands-post.yml diff --git a/etc/kayobe/cephadm.yml b/etc/kayobe/cephadm.yml index 4f5f0ed7c..ae06880ca 100644 --- a/etc/kayobe/cephadm.yml +++ b/etc/kayobe/cephadm.yml @@ -60,9 +60,12 @@ cephadm_cluster_network: "{{ storage_mgmt_net_name | net_cidr }}" # List of Cephx keys. See stackhpc.cephadm.keys role for format. #cephadm_keys: -# A list of commands to pass to cephadm shell -- ceph. See stackhpc.cephadm.commands -# for format. -#cephadm_commands: +# A list of commands to pass to cephadm shell -- ceph. See +# stackhpc.cephadm.commands for format. Pre commands run before the rest of the +# post-deployment configuration, post commands run after the rest of the +# post-deployment configuration. +#cephadm_commands_pre: +#cephadm_commands_post: ############################################################################### # Kolla Ceph auto-configuration. diff --git a/etc/kayobe/environments/ci-multinode/cephadm.yml b/etc/kayobe/environments/ci-multinode/cephadm.yml index addb37ddf..7885a5735 100644 --- a/etc/kayobe/environments/ci-multinode/cephadm.yml +++ b/etc/kayobe/environments/ci-multinode/cephadm.yml @@ -35,6 +35,12 @@ cephadm_pools: - name: vms application: rbd state: present + - name: cephfs_data + application: cephfs + state: "{{ 'present' if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else 'absent' }}" + - name: cephfs_metadata + application: cephfs + state: "{{ 'present' if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else 'absent' }}" # List of Cephx keys. See stackhpc.cephadm.keys role for format. cephadm_keys: @@ -56,3 +62,16 @@ cephadm_keys: osd: "profile rbd pool=images" mgr: "profile rbd pool=images" state: present + - name: client.manila + caps: + mon: "allow r" + mgr: "allow rw" + state: "{{ 'present' if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else 'absent' }}" + +# List of Cephadm commands to run. See stackhpc.cephadm.commands role for format. +cephadm_commands_pre: [] + +cephadm_commands_post: "{{ cephadm_commands_manila_cephfs_native if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else [] }}" +cephadm_commands_manila_cephfs_native: + - "fs new manila-cephfs cephfs_metadata cephfs_data" + - "orch apply mds manila-cephfs" diff --git a/etc/kayobe/environments/ci-multinode/inventory/group_vars/seed/network-interfaces b/etc/kayobe/environments/ci-multinode/inventory/group_vars/seed/network-interfaces index 0c868b110..a918946eb 100644 --- a/etc/kayobe/environments/ci-multinode/inventory/group_vars/seed/network-interfaces +++ b/etc/kayobe/environments/ci-multinode/inventory/group_vars/seed/network-interfaces @@ -10,6 +10,10 @@ external_interface: "{{ vxlan_interfaces[0].device }}.{{ external_vlan }}" public_interface: "{{ vxlan_interfaces[0].device }}.{{ public_vlan }}" +# The storage interface is required for routing manila traffic between VMs and +# the Storage nodes. +storage_interface: "{{ vxlan_interfaces[0].device }}.{{ storage_vlan }}" + ############################################################################### # Dummy variable to allow Ansible to accept this file. workaround_ansible_issue_8743: yes diff --git a/etc/kayobe/environments/ci-multinode/kolla.yml b/etc/kayobe/environments/ci-multinode/kolla.yml index 5c6c80dae..a087ad66f 100644 --- a/etc/kayobe/environments/ci-multinode/kolla.yml +++ b/etc/kayobe/environments/ci-multinode/kolla.yml @@ -4,3 +4,7 @@ kolla_enable_cinder_backup: true kolla_enable_neutron_provider_networks: true kolla_enable_ovn: true kolla_enable_octavia: true + +# The multinode environment supports Manila but it is not enabled by default. +# kolla_enable_manila: true +# kolla_enable_manila_backend_cephfs_native: true diff --git a/etc/kayobe/environments/ci-multinode/kolla/globals.yml b/etc/kayobe/environments/ci-multinode/kolla/globals.yml index bd726ddc5..e0766e7fe 100644 --- a/etc/kayobe/environments/ci-multinode/kolla/globals.yml +++ b/etc/kayobe/environments/ci-multinode/kolla/globals.yml @@ -22,3 +22,6 @@ es_heap_size: 1g octavia_auto_configure: "no" octavia_provider_drivers: "ovn:OVN provider" octavia_provider_agents: "ovn" + +# Manila CephFS configuration +manila_cephfs_filesystem_name: manila-cephfs diff --git a/etc/kayobe/environments/ci-multinode/seed.yml b/etc/kayobe/environments/ci-multinode/seed.yml index 3eb7afa90..749c398d1 100644 --- a/etc/kayobe/environments/ci-multinode/seed.yml +++ b/etc/kayobe/environments/ci-multinode/seed.yml @@ -3,12 +3,29 @@ seed_bootstrap_user: "{{ os_distribution if os_distribution == 'ubuntu' else 'cl seed_lvm_groups: - "{{ stackhpc_lvm_group_rootvg }}" +seed_extra_network_interfaces: > + "{{ seed_extra_network_interfaces_external + + (seed_extra_network_interfaces_manila if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else []) }}" + # Seed has been provided an external interface # for tempest tests and SSH access to machines. -seed_extra_network_interfaces: +seed_extra_network_interfaces_external: - "external" - "public" +# Seed requires access to the storage network for manila-cephfs. +seed_extra_network_interfaces_manila: + - "storage" + # Enable IP routing and source NAT on the seed, allowing it to be used as the # external subnet gateway and provide internet access for VMs in the deployment. seed_enable_snat: true + +snat_rules_default: + - interface: "{{ ansible_facts.default_ipv4.interface }}" + source_ip: "{{ ansible_facts.default_ipv4.address }}" +snat_rules_manila: + - interface: "{{ storage_interface }}" + source_ip: "{{ ansible_facts[storage_interface].ipv4.address }}" +# Only add the storage snat rule if we are using manila-cephfs. +snat_rules: "{{ snat_rules_default + snat_rules_manila if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else snat_rules_default }}" diff --git a/releasenotes/notes/manila-for-multinode-df2186276c6b434a.yaml b/releasenotes/notes/manila-for-multinode-df2186276c6b434a.yaml new file mode 100644 index 000000000..6952fe9bc --- /dev/null +++ b/releasenotes/notes/manila-for-multinode-df2186276c6b434a.yaml @@ -0,0 +1,8 @@ +--- +features: + - | + Adds support for Manila in the ci-multinode environment using the CephFS + native backend. This is disabled by default, but can be enabled by setting + the following variables in the kayobe configuration: + `kolla_enable_manila: true` + `kolla_enable_manila_backend_cephfs_native: true` diff --git a/releasenotes/notes/split-cephadm-commands-pre-post-bddd25b8bd171ab0.yaml b/releasenotes/notes/split-cephadm-commands-pre-post-bddd25b8bd171ab0.yaml new file mode 100644 index 000000000..c39ffa43a --- /dev/null +++ b/releasenotes/notes/split-cephadm-commands-pre-post-bddd25b8bd171ab0.yaml @@ -0,0 +1,8 @@ +--- +features: + - | + Split `cephadm_commands` into `cephadm_commands_pre` and + `cephadm_commands_post` commands. This allows the user to run commands that + must be run before the rest of the post-deployment configuration, as well + as commands that rely on resources created by the post-deployment config. +