Add support for multiple root encryption secrets

For some use cases operators would like to periodically introduce a
new encryption root secret that would be used when new object data is
written. However, existing encrypted data does not need to be
re-encrypted with keys derived from the new root secret. Older root
secret(s) would still be used as necessary to decrypt older object
data.

This patch modifies the KeyMaster class to support multiple root
secrets indexed via unique secret_id's, and to store the id of the
root secret used for an encryption operation in the crypto meta. The
decrypter is modified to fetch appropriate keys based on the secret id
in retrieved crypto meta.

The changes are backwards compatible with previous crypto middleware
configurations and existing encrypted object data.

Change-Id: I40307acf39b6c1cc9921f711a8da55d03924d232

doc/source/overview_encryption.rst

@@ -97,12 +97,6 @@ the `proxy-server.conf` file, for example::
 Root secret values MUST be at least 44 valid base-64 characters and
 should be consistent across all proxy servers. The minimum length of 44 has
 been chosen because it is the length of a base-64 encoded 32 byte value.
-Alternatives to specifying the encryption root secret directly in the
-`proxy-server.conf` file are storing it in a separate file, or storing it in
-an :ref:`external key management system
-<encryption_root_secret_in_external_kms>` such as `Barbican
-<https://docs.openstack.org/barbican>`_ or a
-`KMIP <https://www.oasis-open.org/committees/kmip/>`_ service.
 
 .. note::
 
@@ -169,6 +163,62 @@ into GET and PUT requests by the :ref:`copy` middleware before reaching the
 encryption middleware and as a result object data and metadata is decrypted and
 re-encrypted when copied.
 
+Changing the encryption root secret
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+From time to time it may be desirable to change the root secret that is used to
+derive encryption keys for new data written to the cluster. The `keymaster`
+middleware allows alternative root secrets to be specified in its configuration
+using options of the form::
+
+    encryption_root_secret_<secret_id> = <secret value>
+
+where ``secret_id`` is a unique identifier for the root secret and ``secret
+value`` is a value that meets the requirements for a root secret described
+above.
+
+Only one root secret is used to encrypt new data at any moment in time. This
+root secret is specified using the ``active_root_secret_id`` option. If
+specified, the value of this option should be one of the configured root secret
+``secret_id`` values; otherwise the value of ``encryption_root_secret`` will be
+taken as the default active root secret.
+
+.. note::
+
+    The active root secret is only used to derive keys for new data written to
+    the cluster. Changing the active root secret does not cause any existing
+    data to be re-encrypted.
+
+Existing encrypted data will be decrypted using the root secret that was active
+when that data was written. All previous active root secrets must therefore
+remain in the middleware configuration in order for decryption of existing data
+to succeed.  Existing encrypted data will reference previous root secret by
+the ``secret_id`` so it must be kept consistent in the configuration.
+
+.. note::
+
+    Do not remove or change any previously active ``<secret value>`` or ``<secret_id>``.
+
+For example, the following keymaster configuration file specifies three root
+secrets, with the value of ``encryption_root_secret_2`` being the current
+active root secret::
+
+    [keymaster]
+    active_root_secret_id = 2
+    encryption_root_secret = your_secret
+    encryption_root_secret_1 = your_secret_1
+    encryption_root_secret_2 = your_secret_2
+
+.. note::
+
+    To ensure there is no loss of data availability, deploying a new key to
+    your cluster requires a two-stage config change. First, add the new key
+    to the ``key_id_<secret_id>`` option and restart the proxy-server. Do this
+    for all proxies. Next, set the ``active_root_secret_id`` option to the
+    new secret id and restart the proxy. Again, do this for all proxies. This
+    process ensures that all proxies will have the new key available for
+    *decryption* before any proxy uses it for *encryption*.
+
 Encryption middleware
 ---------------------
 

etc/proxy-server.conf-sample

@@ -1047,6 +1047,19 @@ use = egg:swift#keymaster
 # likely to result in data loss.
 encryption_root_secret = changeme
 
+# Multiple root secrets may be configured using options named
+# 'encryption_root_secret_<secret_id>' where 'secret_id' is a unique
+# identifier. This enables the root secret to be changed from time to time.
+# Only one root secret is used for object PUTs or POSTs at any moment in time.
+# This is specified by the 'active_root_secret_id' option. If
+# 'active_root_secret_id' is not specified then the root secret specified by
+# 'encryption_root_secret' is considered to be the default. Once a root secret
+# has been used as the default root secret it must remain in the config file in
+# order that any objects that were encrypted with it may be subsequently
+# decrypted. The secret_id used to identify the key cannot change.
+# encryption_root_secret_myid = changeme
+# active_root_secret_id = myid
+
 # Sets the path from which the keymaster config options should be read. This
 # allows multiple processes which need to be encryption-aware (for example,
 # proxy-server and container-sync) to share the same config file, ensuring

swift/common/exceptions.py

@@ -223,6 +223,10 @@ class EncryptionException(SwiftException):
     pass
 
 
+class UnknownSecretIdError(EncryptionException):
+    pass
+
+
 class ClientException(Exception):
 
     def __init__(self, msg, http_scheme='', http_host='', http_port='',

swift/common/middleware/crypto/crypto_utils.py

@@ -24,7 +24,7 @@
 from six.moves.urllib import parse as urlparse
 
 from swift import gettext_ as _
-from swift.common.exceptions import EncryptionException
+from swift.common.exceptions import EncryptionException, UnknownSecretIdError
 from swift.common.swob import HTTPInternalServerError
 from swift.common.utils import get_logger
 from swift.common.wsgi import WSGIContext
@@ -155,7 +155,7 @@ def __init__(self, crypto_app, server_type, logger):
         self.logger = logger
         self.server_type = server_type
 
-    def get_keys(self, env, required=None):
+    def get_keys(self, env, required=None, key_id=None):
         # Get the key(s) from the keymaster
         required = required if required is not None else [self.server_type]
         try:
@@ -165,11 +165,14 @@ def get_keys(self, env, required=None):
             raise HTTPInternalServerError(
                 "Unable to retrieve encryption keys.")
 
+        err = None
         try:
-            keys = fetch_crypto_keys()
+            keys = fetch_crypto_keys(key_id=key_id)
+        except UnknownSecretIdError as err:
+            self.logger.error('get_keys(): unknown key id: %s', err)
+            raise
         except Exception as err:  # noqa
-            self.logger.exception(_(
-                'ERROR get_keys(): from callback: %s') % err)
+            self.logger.exception('get_keys(): from callback: %s', err)
             raise HTTPInternalServerError(
                 "Unable to retrieve encryption keys.")
 
@@ -191,6 +194,17 @@ def get_keys(self, env, required=None):
 
         return keys
 
+    def get_multiple_keys(self, env):
+        # get a list of keys from the keymaster containing one dict of keys for
+        # each of the keymaster root secret ids
+        keys = [self.get_keys(env)]
+        active_key_id = keys[0]['id']
+        for other_key_id in keys[0].get('all_ids', []):
+            if other_key_id == active_key_id:
+                continue
+            keys.append(self.get_keys(env, key_id=other_key_id))
+        return keys
+
 
 def dump_crypto_meta(crypto_meta):
     """