chore(MPL): Update README and primary pydocs (#658)

Author: lucasmcdonald3

README.rst

@@ -39,6 +39,12 @@ Required Prerequisites
 * boto3 >= 1.10.0
 * attrs
 
+Recommended Prerequisites
+=========================
+
+* aws-cryptographic-material-providers: >= TODO.TODO.TODO (TODO-MPL: versionme)
+    * Requires Python 3.11+.
+
 Installation
 ============
 
@@ -49,42 +55,71 @@ Installation
 
    .. code::
 
-       $ pip install aws-encryption-sdk
+       $ pip install "aws-encryption-sdk[MPL]"
 
+The `[MPL]` suffix also installs the `AWS Cryptographic Material Providers Library (MPL)`_.
+This is a library that contains constructs for encrypting and decrypting your data.
+We highly recommend installing the MPL.
+However, if you do not wish to install the MPL, omit the `[MPL]` suffix.
 
 Concepts
 ========
-There are four main concepts that you need to understand to use this library:
+There are three main concepts that you need to understand to use this library:
+
+Data Keys
+---------
+Data keys are the encryption keys that are used to encrypt your data. If your algorithm suite
+uses a key derivation function, the data key is used to generate the key that directly encrypts the data.
+
+Keyrings
+--------
+Keyrings are resources that generate, encrypt, and decrypt data keys.
+You specify a keyring when encrypting and the same or a different keyring when decrypting.
+
+Note: You must also install the `AWS Cryptographic Material Providers Library (MPL)`_ to create and use keyrings.
+
+For more information, see the `AWS Documentation for Keyrings`_.
 
 Cryptographic Materials Managers
 --------------------------------
 Cryptographic materials managers (CMMs) are resources that collect cryptographic materials and prepare them for
 use by the Encryption SDK core logic.
 
-An example of a CMM is the default CMM, which is automatically generated anywhere a caller provides a master
-key provider. The default CMM collects encrypted data keys from all master keys referenced by the master key
-provider.
+An example of a CMM is the default CMM,
+which is automatically generated anywhere a caller provides a keyring.
+
+Note: You must also install the `AWS Cryptographic Material Providers Library (MPL)`_
+to create and use CMMs that use keyrings.
+CMMs that use master key providers have been marked as legacy since v4 of this library.
 
-An example of a more advanced CMM is the caching CMM, which caches cryptographic materials provided by another CMM.
+Legacy Concepts
+===============
+This section describes legacy concepts introduced in earlier versions of this library.
+These components have been superseded by new components in the `AWS Cryptographic Material Providers Library (MPL)`_.
+Please avoid using these components, and instead use components in the MPL.
 
 Master Key Providers
 --------------------
 Master key providers are resources that provide master keys.
-An example of a master key provider is `AWS KMS`_.
 
 To encrypt data in this client, a ``MasterKeyProvider`` object must contain at least one ``MasterKey`` object.
 
 ``MasterKeyProvider`` objects can also contain other ``MasterKeyProvider`` objects.
 
+NOTE: Master key providers are legacy components
+and have been superseded by keyrings
+provided by the `AWS Cryptographic Material Providers Library (MPL)`_.
+Please install this library and migrate master key providers to keyring interfaces.
+
 Master Keys
 -----------
 Master keys generate, encrypt, and decrypt data keys.
-An example of a master key is a `KMS customer master key (CMK)`_.
+An example of a master key is an `AWS KMS key`_.
 
-Data Keys
----------
-Data keys are the encryption keys that are used to encrypt your data. If your algorithm suite
-uses a key derivation function, the data key is used to generate the key that directly encrypts the data.
+NOTE: Master keys are legacy constructs
+and have been superseded by keyrings
+provided by the `AWS Cryptographic Material Providers Library (MPL)`_.
+Please install this library and migrate master key providers to keyring interfaces.
 
 *****
 Usage
@@ -110,147 +145,71 @@ version of the AWS Encryption SDK, we recommend using the default value.
     )
 
 
-You must then create an instance of either a master key provider or a CMM. The examples in this
-readme use the ``StrictAwsKmsMasterKeyProvider`` class.
+You must then create an instance of either a keyring (with the MPL installed) or a CMM.
+Note: You must also install the `AWS Cryptographic Material Providers Library (MPL)`_ to use keyrings.
+(You may also provide an instance of a legacy master key provider, but this is not recommended.)
+
 
+AwsKmsMultiKeyring
+==================
 
-StrictAwsKmsMasterKeyProvider
-=============================
-A ``StrictAwsKmsMasterKeyProvider`` is configured with an explicit list of AWS KMS
-CMKs with which to encrypt and decrypt data. On encryption, it encrypts the plaintext with all
-configured CMKs. On decryption, it only attempts to decrypt ciphertexts that have been wrapped
-with a CMK that matches one of the configured CMK ARNs.
+An ``AwsKmsMultiKeyring`` is configured with a generator keyring and a list of
+child keyrings of type ``AwsKmsKeyring``. The effect is like using several keyrings
+in a series. When you use a multi-keyring to encrypt data, any of the wrapping keys
+in any of its keyrings can decrypt that data.
 
-To create a ``StrictAwsKmsMasterKeyProvider`` you must provide one or more CMKs. For providers that will only
-be used for encryption, you can use any valid `KMS key identifier`_. For providers that will be used for decryption, you
-must use the key ARN; key ids, alias names, and alias ARNs are not supported.
+On encryption, the generator keyring generates and encrypts the plaintext data key.
+Then, all of the wrapping keys in all of the child keyrings encrypt the same plaintext data key.
+The final `encrypted message`_ will include a copy of the data key encrypted by each configured key.
+On decryption, the AWS Encryption SDK uses the keyrings to try to decrypt one of the encrypted data keys.
+The keyrings are called in the order that they are specified in the multi-keyring.
+Processing stops as soon as any key in any keyring can decrypt an encrypted data key.
 
-Because the ``StrictAwsKmsMasterKeyProvider`` uses the `boto3 SDK`_ to interact with `AWS KMS`_,
+An individual ``AwsKmsKeyring`` in an ``AwsKmsMultiKeyring`` is configured with an
+AWS KMS key ARN.
+For keyrings that will only be used for encryption,
+you can use any valid `KMS key identifier`_.
+For providers that will be used for decryption,
+you must use the key ARN.
+Key ids, alias names, and alias ARNs are not supported for decryption.
+
+Because the ``AwsKmsMultiKeyring`` uses the `boto3 SDK`_ to interact with `AWS KMS`_,
 it requires AWS Credentials.
 To provide these credentials, use the `standard means by which boto3 locates credentials`_ or provide a
-pre-existing instance of a ``botocore session`` to the ``StrictAwsKmsMasterKeyProvider``.
+pre-existing instance of a ``botocore session`` to the ``AwsKmsMultiKeyring``.
 This latter option can be useful if you have an alternate way to store your AWS credentials or
 you want to reuse an existing instance of a botocore session in order to decrease startup costs.
+You can also add KMS keys from multiple regions to the ``AwsKmsMultiKeyring``.
 
-If you configure the the ``StrictAwsKmsMasterKeyProvider`` with multiple CMKs, the `final message`_
-will include a copy of the data key encrypted by each configured CMK.
+See `examples/src/aws_kms_multi_keyring_example.py`_ for a code example configuring and using
+a ``AwsKmsMultiKeyring`` with the ``EncryptionSDKClient``.
 
-.. code:: python
-
-    import aws_encryption_sdk
-
-    kms_key_provider = aws_encryption_sdk.StrictAwsKmsMasterKeyProvider(key_ids=[
-        'arn:aws:kms:us-east-1:2222222222222:key/22222222-2222-2222-2222-222222222222',
-        'arn:aws:kms:us-east-1:3333333333333:key/33333333-3333-3333-3333-333333333333'
-    ])
-
-You can add CMKs from multiple regions to the ``StrictAwsKmsMasterKeyProvider``.
-
-.. code:: python
-
-    import aws_encryption_sdk
-
-    kms_key_provider = aws_encryption_sdk.StrictAwsKmsMasterKeyProvider(key_ids=[
-        'arn:aws:kms:us-east-1:2222222222222:key/22222222-2222-2222-2222-222222222222',
-        'arn:aws:kms:us-west-2:3333333333333:key/33333333-3333-3333-3333-333333333333',
-        'arn:aws:kms:ap-northeast-1:4444444444444:key/44444444-4444-4444-4444-444444444444'
-    ])
-
-
-DiscoveryAwsKmsMasterKeyProvider
-================================
-We recommend using a ``StrictAwsKmsMasterKeyProvider`` in order to ensure that you can only
-encrypt and decrypt data using the AWS KMS CMKs you expect. However, if you are unable to
-explicitly identify the AWS KMS CMKs that should be used for decryption, you can instead
-use a ``DiscoveryAwsKmsMasterKeyProvider`` for decryption operations. This provider
+AwsKmsDiscoveryKeyring
+======================
+We recommend using an ``AwsKmsMultiKeyring`` in order to ensure that you can only
+encrypt and decrypt data using the AWS KMS key ARN you expect. However, if you are unable to
+explicitly identify the AWS KMS key ARNs that should be used for decryption, you can instead
+use an ``AwsKmsDiscoveryKeyring`` for decryption operations. This provider
 attempts decryption of any ciphertexts as long as they match a ``DiscoveryFilter`` that
 you configure. A ``DiscoveryFilter`` consists of a list of AWS account ids and an AWS
 partition.
+If you do not want to filter the set of allowed accounts, you can also omit the ``discovery_filter`` argument.
 
-.. code:: python
+Note that an ``AwsKmsDiscoveryKeyring`` cannot be used for encryption operations.
 
-    import aws_encryption_sdk
-    from aws_encryption_sdk.key_providers.kms import DiscoveryFilter
+See `examples/src/aws_kms_discovery_keyring_example.py`_ for a code example configuring and using
+an ``AwsKmsDiscoveryKeyring`` with the ``EncryptionSDKClient``.
 
-    discovery_filter = DiscoveryFilter(
-        account_ids=['222222222222', '333333333333'],
-        partition='aws'
-    )
-    kms_key_provider = aws_encryption_sdk.DiscoveryAwsKmsMasterKeyProvider(
-        discovery_filter=discovery_filter
-    )
-
-If you do not want to filter the set of allowed accounts, you can also omit the ``discovery_filter`` argument.
-
-Note that a ``DiscoveryAwsKmsMasterKeyProvider`` cannot be used for encryption operations.
 
 Encryption and Decryption
 =========================
-After you create an instance of an ``EncryptionSDKClient`` and a ``MasterKeyProvider``, you can use either of
-the client's two ``encrypt``/``decrypt`` functions to encrypt and decrypt your data.
-
-.. code:: python
-
-    import aws_encryption_sdk
-    from aws_encryption_sdk.identifiers import CommitmentPolicy
-
-    client = aws_encryption_sdk.EncryptionSDKClient(
-        commitment_policy=CommitmentPolicy.FORBID_ENCRYPT_ALLOW_DECRYPT
-    )
-
-    kms_key_provider = aws_encryption_sdk.StrictAwsKmsMasterKeyProvider(key_ids=[
-        'arn:aws:kms:us-east-1:2222222222222:key/22222222-2222-2222-2222-222222222222',
-        'arn:aws:kms:us-east-1:3333333333333:key/33333333-3333-3333-3333-333333333333'
-    ])
-    my_plaintext = b'This is some super secret data!  Yup, sure is!'
-
-    my_ciphertext, encryptor_header = client.encrypt(
-        source=my_plaintext,
-        key_provider=kms_key_provider
-    )
-
-    decrypted_plaintext, decryptor_header = client.decrypt(
-        source=my_ciphertext,
-        key_provider=kms_key_provider
-    )
-
-    assert my_plaintext == decrypted_plaintext
-    assert encryptor_header.encryption_context == decryptor_header.encryption_context
-
-You can provide an `encryption context`_: a form of additional authenticating information.
-
-.. code:: python
-
-    import aws_encryption_sdk
-    from aws_encryption_sdk.identifiers import CommitmentPolicy
-
-    client = aws_encryption_sdk.EncryptionSDKClient(
-        commitment_policy=CommitmentPolicy.FORBID_ENCRYPT_ALLOW_DECRYPT
-    )
+After you create an instance of an ``EncryptionSDKClient`` and a ``Keyring``, you can use
+the client's ``encrypt`` and ``decrypt`` functions to encrypt and decrypt your data.
 
-    kms_key_provider = aws_encryption_sdk.StrictAwsKmsMasterKeyProvider(key_ids=[
-        'arn:aws:kms:us-east-1:2222222222222:key/22222222-2222-2222-2222-222222222222',
-        'arn:aws:kms:us-east-1:3333333333333:key/33333333-3333-3333-3333-333333333333'
-    ])
-    my_plaintext = b'This is some super secret data!  Yup, sure is!'
-
-    my_ciphertext, encryptor_header = client.encrypt(
-        source=my_plaintext,
-        key_provider=kms_key_provider,
-        encryption_context={
-            'not really': 'a secret',
-            'but adds': 'some authentication'
-        }
-    )
-
-    decrypted_plaintext, decryptor_header = client.decrypt(
-        source=my_ciphertext,
-        key_provider=kms_key_provider
-    )
-
-    assert my_plaintext == decrypted_plaintext
-    assert encryptor_header.encryption_context == decryptor_header.encryption_context
+You can also provide an `encryption context`_: a form of additional authenticating information.
 
+See code in the `examples/src/`_ directory for code examples configuring and using
+keyrings and encryption context with the ``EncryptionSDKClient``.
 
 Streaming
 =========
@@ -259,57 +218,19 @@ memory at once, you can use this library's streaming clients directly. The strea
 file-like objects, and behave exactly as you would expect a Python file object to behave,
 offering context manager and iteration support.
 
-.. code:: python
-
-    import aws_encryption_sdk
-    from aws_encryption_sdk.identifiers import CommitmentPolicy
-    import filecmp
-
-    client = aws_encryption_sdk.EncryptionSDKClient(
-        commitment_policy=CommitmentPolicy.FORBID_ENCRYPT_ALLOW_DECRYPT
-    )
-
-    kms_key_provider = aws_encryption_sdk.StrictAwsKmsMasterKeyProvider(key_ids=[
-        'arn:aws:kms:us-east-1:2222222222222:key/22222222-2222-2222-2222-222222222222',
-        'arn:aws:kms:us-east-1:3333333333333:key/33333333-3333-3333-3333-333333333333'
-    ])
-    plaintext_filename = 'my-secret-data.dat'
-    ciphertext_filename = 'my-encrypted-data.ct'
-
-    with open(plaintext_filename, 'rb') as pt_file, open(ciphertext_filename, 'wb') as ct_file:
-        with client.stream(
-            mode='e',
-            source=pt_file,
-            key_provider=kms_key_provider
-        ) as encryptor:
-            for chunk in encryptor:
-                ct_file.write(chunk)
-
-    new_plaintext_filename = 'my-decrypted-data.dat'
-
-    with open(ciphertext_filename, 'rb') as ct_file, open(new_plaintext_filename, 'wb') as pt_file:
-        with client.stream(
-            mode='d',
-            source=ct_file,
-            key_provider=kms_key_provider
-        ) as decryptor:
-            for chunk in decryptor:
-                pt_file.write(chunk)
-
-    assert filecmp.cmp(plaintext_filename, new_plaintext_filename)
-    assert encryptor.header.encryption_context == decryptor.header.encryption_context
+See `examples/src/file_streaming_example.py`_ for a code example streaming data to and from files.
 
 Performance Considerations
 ==========================
 Adjusting the frame size can significantly improve the performance of encrypt/decrypt operations with this library.
 
-Processing each frame in a framed message involves a certain amount of overhead.  If you are encrypting a large file,
-increasing the frame size can offer potentially significant performance gains.  We recommend that you tune these values
+Processing each frame in a framed message involves a certain amount of overhead. If you are encrypting a large file,
+increasing the frame size can offer potentially significant performance gains. We recommend that you tune these values
 to your use-case in order to obtain peak performance.
 
 Thread safety
 ==========================
-The ``EncryptionSDKClient`` and all provided ``CryptoMaterialsManager`` are thread safe.
+The ``EncryptionSDKClient`` and all provided ``CryptoMaterialsManager`` in this library are thread safe.
 But instances of ``BaseKMSMasterKeyProvider`` MUST not be shared between threads,
 for the reasons outlined in `the boto3 docs <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-or-multiprocessing-with-resources>`_.
 
@@ -323,17 +244,28 @@ Finally, while the ``CryptoMaterialsCache`` is thread safe,
 sharing entries in that cache across threads needs to be done carefully
 (see the !Note about partition name `in the API Docs <https://aws-encryption-sdk-python.readthedocs.io/en/latest/generated/aws_encryption_sdk.materials_managers.caching.html#aws_encryption_sdk.materials_managers.caching.CachingCryptoMaterialsManager>`_).
 
+**Important:** Components from the `AWS Cryptographic Material Providers Library (MPL)`_
+have separate thread safety considerations.
+For more information, see the note on thread safety in that project's README (TODO-MPL: link)
+
+
 .. _AWS Encryption SDK: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html
 .. _cryptography: https://cryptography.io/en/latest/
 .. _cryptography installation guide: https://cryptography.io/en/latest/installation/
 .. _Read the Docs: http://aws-encryption-sdk-python.readthedocs.io/en/latest/
 .. _GitHub: https://github.com/aws/aws-encryption-sdk-python/
 .. _AWS KMS: https://docs.aws.amazon.com/kms/latest/developerguide/overview.html
-.. _KMS customer master key (CMK): https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#master_keys
+.. _AWS KMS key: https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#master_keys
 .. _KMS key identifier: https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id
 .. _boto3 SDK: https://boto3.readthedocs.io/en/latest/
 .. _standard means by which boto3 locates credentials: https://boto3.readthedocs.io/en/latest/guide/configuration.html
-.. _final message: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/message-format.html
+.. _encrypted message: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/message-format.html
 .. _encryption context: https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context
 .. _Security issue notifications: ./CONTRIBUTING.md#security-issue-notifications
 .. _Support Policy: ./SUPPORT_POLICY.rst
+.. _AWS Cryptographic Material Providers Library (MPL): https://github.com/aws/aws-cryptographic-material-providers-library
+.. _AWS Documentation for Keyrings: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/choose-keyring.html
+.. _examples/src/aws_kms_multi_keyring_example.py: https://github.com/aws/aws-encryption-sdk-python/blob/master/examples/src/aws_kms_multi_keyring_example.py
+.. _examples/src/aws_kms_discovery_keyring_example.py: https://github.com/aws/aws-encryption-sdk-python/blob/master/examples/src/aws_kms_discovery_keyring_example.py
+.. _examples/src/: https://github.com/aws/aws-encryption-sdk-python/tree/master/examples/src/
+.. _examples/src/file_streaming_example.py: https://github.com/aws/aws-encryption-sdk-python/blob/master/examples/src/file_streaming_example.py