ceph-volume dmcrypt support for simple

doc/ceph-volume/index.rst

@@ -35,14 +35,8 @@ Existing OSDs
 If the cluster has OSDs that were provisioned with ``ceph-disk``, then
 ``ceph-volume`` can take over the management of these with
 :ref:`ceph-volume-simple`. A scan is done on the data device or OSD directory,
-and ``ceph-disk`` is fully disabled.
+and ``ceph-disk`` is fully disabled. Encryption is fully supported.
 
-Encrypted OSDs
-^^^^^^^^^^^^^^
-If using encryption with OSDs, there is currently no support in ``ceph-volume``
-for this scenario (although support for this is coming soon). In this case, it
-is OK to continue to use ``ceph-disk`` until ``ceph-volume`` fully supports it.
-This page will be updated when that happens.
 
 .. toctree::
    :hidden:

doc/ceph-volume/lvm/prepare.rst

@@ -149,7 +149,6 @@ already running there are a few things to take into account:
 
 * Preferably, no other mechanisms to mount the volume should exist, and should
   be removed (like fstab mount points)
-* There is currently no support for encrypted volumes
 
 The one time process for an existing OSD, with an ID of 0 and using
 a ``"ceph"`` cluster name would look like (the following command will **destroy

doc/ceph-volume/simple/scan.rst

@@ -4,7 +4,8 @@
 ========
 Scanning allows to capture any important details from an already-deployed OSD
 so that ``ceph-volume`` can manage it without the need of any other startup
-workflows or tools (like ``udev`` or ``ceph-disk``).
+workflows or tools (like ``udev`` or ``ceph-disk``). Encryption with LUKS or
+PLAIN formats is fully supported.
 
 The command has the ability to inspect a running OSD, by inspecting the
 directory where the OSD data is stored, or by consuming the data partition.
@@ -42,6 +43,12 @@ are a few files that must exist in order to have a successful scan:
 * ``type``
 * ``whoami``
 
+If the OSD is encrypted, it will additionally add the following keys:
+
+* ``encrypted``
+* ``encryption_type``
+* ``lockbox_keyring``
+
 In the case of any other file, as long as it is not a binary or a directory, it
 will also get captured and persisted as part of the JSON object.
 
@@ -53,20 +60,23 @@ would look like::
 
     "whoami": "1",
 
-For files that may have more than one line, the contents are left as-is, for
-example, a ``keyring`` could look like this::
+For files that may have more than one line, the contents are left as-is, except
+for keyrings which are treated specially and parsed to extract the keyring. For
+example, a ``keyring`` that gets read as::
+
+    [osd.1]\n\tkey = AQBBJ/dZp57NIBAAtnuQS9WOS0hnLVe0rZnE6Q==\n
+
+Would get stored as::
+
+    "keyring": "AQBBJ/dZp57NIBAAtnuQS9WOS0hnLVe0rZnE6Q==",
 
-    "keyring": "[osd.1]\n\tkey = AQBBJ/dZp57NIBAAtnuQS9WOS0hnLVe0rZnE6Q==\n",
 
 For a directory like ``/var/lib/ceph/osd/ceph-1``, the command could look
 like::
 
     ceph-volume simple scan /var/lib/ceph/osd/ceph1
 
 
-.. note:: There is no support for encrypted OSDs
-
-
 .. _ceph-volume-simple-scan-device:
 
 Device scan
@@ -93,9 +103,6 @@ could look like::
     ceph-volume simple scan /dev/sda1
 
 
-.. note:: There is no support for encrypted OSDs
-
-
 .. _ceph-volume-simple-scan-json:
 
 ``JSON`` contents
@@ -147,7 +154,7 @@ This is a sample ``JSON`` metadata, from an OSD that is using ``bluestore``::
             "uuid": "86ebd829-1405-43d3-8fd6-4cbc9b6ecf96"
         },
         "fsid": "86ebd829-1405-43d3-8fd6-4cbc9b6ecf96",
-        "keyring": "[osd.3]\n\tkey = AQBBJ/dZp57NIBAAtnuQS9WOS0hnLVe0rZnE6Q==\n",
+        "keyring": "AQBBJ/dZp57NIBAAtnuQS9WOS0hnLVe0rZnE6Q==",
         "kv_backend": "rocksdb",
         "magic": "ceph osd volume v026",
         "mkfs_done": "yes",

doc/man/8/ceph-volume.rst

@@ -15,6 +15,9 @@ Synopsis
 | **ceph-volume** **lvm** [ *trigger* | *create* | *activate* | *prepare*
 | *zap* | *list*]
 
+| **ceph-volume** **simple** [ *trigger* | *scan* | *activate* ]
+
+
 Description
 ===========
 
@@ -157,6 +160,84 @@ Positional arguments:
 * <DEVICE>  Either in the form of ``vg/lv`` for logical volumes or
   ``/path/to/sda1`` for regular devices.
 
+
+simple
+------
+
+Scan legacy OSD directories or data devices that may have been created by
+ceph-disk, or manually.
+
+Subcommands:
+
+**activate**
+Enables a systemd unit that persists the OSD ID and its UUID (also called
+``fsid`` in Ceph CLI tools), so that at boot time it can understand what OSD is
+enabled and needs to be mounted, while reading information that was previously
+created and persisted at ``/etc/ceph/osd/`` in JSON format.
+
+Usage::
+
+    ceph-volume simple activate --bluestore <osd id> <osd fsid>
+
+Optional Arguments:
+
+* [-h, --help]  show the help message and exit
+* [--bluestore] bluestore objectstore (default)
+* [--filestore] filestore objectstore
+
+Note: It requires a matching JSON file with the following format::
+
+    /etc/ceph/osd/<osd id>-<osd fsid>.json
+
+
+**scan**
+Scan a running OSD or data device for an OSD for metadata that can later be
+used to activate and manage the OSD with ceph-volume. The scan method will
+create a JSON file with the required information plus anything found in the OSD
+directory as well.
+
+Optionally, the JSON blob can be sent to stdout for further inspection.
+
+Usage on data devices::
+
+    ceph-volume simple scan <data device>
+
+Running OSD directories::
+
+    ceph-volume simple scan <path to osd dir>
+
+
+Optional arguments:
+
+* [-h, --help]          show the help message and exit
+* [--stdout]            Send the JSON blob to stdout
+* [--force]             If the JSON file exists at destination, overwrite it
+
+Required Positional arguments:
+
+* <DATA DEVICE or OSD DIR>  Actual data partition or a path to the running OSD
+
+**trigger**
+This subcommand is not meant to be used directly, and it is used by systemd so
+that it proxies input to ``ceph-volume simple activate`` by parsing the
+input from systemd, detecting the UUID and ID associated with an OSD.
+
+Usage::
+
+    ceph-volume simple trigger <SYSTEMD-DATA>
+
+The systemd "data" is expected to be in the format of::
+
+    <OSD ID>-<OSD UUID>
+
+The JSON file associated with the OSD need to have been persisted previously by
+a scan (or manually), so that all needed metadata can be used.
+
+Positional arguments:
+
+* <SYSTEMD_DATA>  Data from a systemd unit containing ID and UUID of the OSD.
+
+
 Availability
 ============
 

src/ceph-volume/ceph_volume/devices/simple/activate.py

@@ -1,15 +1,18 @@
 from __future__ import print_function
 import argparse
+import base64
 import json
 import logging
 import os
 from textwrap import dedent
 from ceph_volume import process, decorators, terminal
 from ceph_volume.util import system, disk
+from ceph_volume.util import encryption as encryption_utils
 from ceph_volume.systemd import systemctl
 
 
 logger = logging.getLogger(__name__)
+mlogger = terminal.MultiLogger(__name__)
 
 
 class Activate(object):
@@ -20,28 +23,108 @@ def __init__(self, argv, systemd=False):
         self.argv = argv
         self.systemd = systemd
 
+    def validate_devices(self, json_config):
+        """
+        ``json_config`` is the loaded dictionary coming from the JSON file. It is usually mixed with
+        other non-device items, but for sakes of comparison it doesn't really matter. This method is
+        just making sure that the keys needed exist
+        """
+        devices = json_config.keys()
+        try:
+            objectstore = json_config['type']
+        except KeyError:
+            logger.warning('"type" was not defined, will assume "bluestore"')
+            objectstore = 'bluestore'
+
+        # Go through all the device combinations that are absolutely required,
+        # raise an error describing what was expected and what was found
+        # otherwise.
+        if objectstore == 'filestore':
+            if {'data', 'journal'}.issubset(set(devices)):
+                return True
+            else:
+                found = [i for i in devices if i in ['data', 'journal']]
+                mlogger.error("Required devices (data, and journal) not present for filestore")
+                mlogger.error('filestore devices found: %s', found)
+                raise RuntimeError('Unable to activate filestore OSD due to missing devices')
+        else:
+            # This is a bit tricky, with newer bluestore we don't need data, older implementations
+            # do (e.g. with ceph-disk). ceph-volume just uses a tmpfs that doesn't require data.
+            if {'block', 'data'}.issubset(set(devices)):
+                return True
+            else:
+                bluestore_devices = ['block.db', 'block.wal', 'block', 'data']
+                found = [i for i in devices if i in bluestore_devices]
+                mlogger.error("Required devices (block and data) not present for bluestore")
+                mlogger.error('bluestore devices found: %s', found)
+                raise RuntimeError('Unable to activate bluestore OSD due to missing devices')
+
+    def get_device(self, uuid):
+        """
+        If a device is encrypted, it will decrypt/open and return the mapper
+        path, if it isn't encrypted it will just return the device found that
+        is mapped to the uuid.  This will make it easier for the caller to
+        avoid if/else to check if devices need decrypting
+
+        :param uuid: The partition uuid of the device (PARTUUID)
+        """
+        device = disk.get_device_from_partuuid(uuid)
+
+        # If device is not found, it is fine to return an empty string from the
+        # helper that finds `device`. If it finds anything and it is not
+        # encrypted, just return what was found
+        if not self.is_encrypted or not device:
+            return device
+
+        if self.encryption_type == 'luks':
+            encryption_utils.luks_open(self.dmcrypt_secret, device, uuid)
+        else:
+            encryption_utils.plain_open(self.dmcrypt_secret, device, uuid)
+
+        return '/dev/mapper/%s' % uuid
+
     @decorators.needs_root
     def activate(self, args):
         with open(args.json_config, 'r') as fp:
             osd_metadata = json.load(fp)
 
+        # Make sure that required devices are configured
+        self.validate_devices(osd_metadata)
+
         osd_id = osd_metadata.get('whoami', args.osd_id)
         osd_fsid = osd_metadata.get('fsid', args.osd_fsid)
-
-        cluster_name = osd_metadata.get('cluster_name', 'ceph')
-        osd_dir = '/var/lib/ceph/osd/%s-%s' % (cluster_name, osd_id)
         data_uuid = osd_metadata.get('data', {}).get('uuid')
         if not data_uuid:
             raise RuntimeError(
                 'Unable to activate OSD %s - no "uuid" key found for data' % args.osd_id
             )
-        data_device = disk.get_device_from_partuuid(data_uuid)
-        journal_device = disk.get_device_from_partuuid(osd_metadata.get('journal', {}).get('uuid'))
-        block_device = disk.get_device_from_partuuid(osd_metadata.get('block', {}).get('uuid'))
-        block_db_device = disk.get_device_from_partuuid(osd_metadata.get('block.db', {}).get('uuid'))
-        block_wal_device = disk.get_device_from_partuuid(
-            osd_metadata.get('block.wal', {}).get('uuid')
-        )
+
+        # Encryption detection, and capturing of the keys to decrypt
+        self.is_encrypted = osd_metadata.get('encrypted', False)
+        self.encryption_type = osd_metadata.get('encryption_type')
+        if self.is_encrypted:
+            lockbox_secret = osd_metadata.get('lockbox.keyring')
+            # write the keyring always so that we can unlock
+            encryption_utils.write_lockbox_keyring(osd_id, osd_fsid, lockbox_secret)
+            # Store the secret around so that the decrypt method can reuse
+            raw_dmcrypt_secret = encryption_utils.get_dmcrypt_key(osd_id, osd_fsid)
+            # Note how both these calls need b64decode. For some reason, the
+            # way ceph-disk creates these keys, it stores them in the monitor
+            # *undecoded*, requiring this decode call again. The lvm side of
+            # encryption doesn't need it, so we are assuming here that anything
+            # that `simple` scans, will come from ceph-disk and will need this
+            # extra decode call here
+            self.dmcrypt_secret = base64.b64decode(raw_dmcrypt_secret)
+
+        cluster_name = osd_metadata.get('cluster_name', 'ceph')
+        osd_dir = '/var/lib/ceph/osd/%s-%s' % (cluster_name, osd_id)
+
+        # XXX there is no support for LVM here
+        data_device = self.get_device(data_uuid)
+        journal_device = self.get_device(osd_metadata.get('journal', {}).get('uuid'))
+        block_device = self.get_device(osd_metadata.get('block', {}).get('uuid'))
+        block_db_device = self.get_device(osd_metadata.get('block.db', {}).get('uuid'))
+        block_wal_device = self.get_device(osd_metadata.get('block.wal', {}).get('uuid'))
 
         if not system.device_is_mounted(data_device, destination=osd_dir):
             process.run(['mount', '-v', data_device, osd_dir])