Skip to content

Commit

Permalink
WIP feat: Cache across Hierarchy Keyrings
Browse files Browse the repository at this point in the history
  • Loading branch information
@texastony
texastony committed Jun 9, 2024
1 parent c7c7e08 commit d08ba88
Show file tree
Hide file tree
Showing 3 changed files with 111 additions and 14 deletions.
95 changes: 93 additions & 2 deletions changes/2024-TBD_cache-across-hierarchy-keyrings/change.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,11 @@ This serves as a reference of all specification documents that this change affec

## Definitions

TODO
An "MPL Consumer" is a library, service, or other application
that uses the AWS Cryptographic Material Providers Library (MPL)
to manage cryptographic materials.
An "MPL Consumer" MAY be using an AWS Crypto Tools product,
such as the AWS Encryption SDK or AWS Database Encryption SDK.

### Conventions used in this document

Expand All @@ -43,7 +47,94 @@ in this document are to be interpreted as described in

## Summary

TODO
The Hierarchy Keyring,
and it's component the (Branch) Keystore,
allow MPL Consumers to reduce their KMS Call volume
by persiting KMS protected cryptographic materials into
an avabile medium
(currently, only a DynamoDB table is avabile as persitance medium).

We call these cryptographic materials Branch Keys.

However, an instance of the Hierarchy Keyring
can only ever call KMS with one KMS Relationship,
which is, at least partly,
configured on the KMS Client determined
at the Hierarchy Keyring's construction.

By KMS Relationship, we mean any or all of the following:

- KMS Configuration
- Credentials used when creating the KMS Client, and thus used when calling KMS
- Other properties of the KMS Client, such as the region, or request headers

The Local Cryptographic Material Cache of
the Hierarchy Keyring instance is then only
populated with Branch Keys that correspond with
that KMS relationship.

Which is appropriate,
as it is clear under what KMS relationship
a Branch Key is accessed.

However,
the Hierarchy Keyring,
and it's Keystore,
have a runtime cost,
exceting memory pressure
and, without manual optimization,
requiring at least 2 TLS handshakes
when first serving a request
(TLS to KMS & TLS to DDB).

Additionally,
the local Cryptographic Materials Cache
excerts some runtime cost,
particualry in a multi-threaded enviorment,
when a background worker thread MAY be refreshing
or pruning entries of the cache.

For MPL Consumers that MUST work with Branch Keys
under different KMS Relationships,
this runtime cost adds up.

These MPL Consumers MAY end up establishing
a LRU Cache of Hierarchy Keyrings.

Which, while workable, is sub-optimal,
and clearly makes the Hierarchy Keyring,
in these conditions,
"Hard to Use".

The objective, with these changes,
is to make the Hierarchy Keyring
"Easy to Use" in a multiple KMS Relationship
enviorment.

To facilitate Caching across Keystores/KMS Clients/KMS Keys,
we MUST break the Cryptographic Materials Cache (CMC)
out of the Hierarchy Keyring.

By allowing MPL Consumers to provide an already intialized CMC
to the Hierarchy Keyring at construction,
the CMC MAY cache Branch Keys protected by different
KMS Relationships.

This simplifies Mutliple KMS Relationship MPL Consumers,
as they do not need to stand up LRU Cache of Hierarchy Keyrings.

Instead, they may maintain one CMC.
They still create a Hierarchy Keyring instance per KMS Relationship,
and they MUST use the correct Keyring to retrieve material
from the Cache.

But they need not maintain many Keyrings;
only the common cache.

Cache misses will populate the cache via
the Hierarchy Keyring that requested the material.

Again, cache entries MUST ONLY be

## Out of Scope

Expand Down
15 changes: 9 additions & 6 deletions framework/cryptographic-materials-cache.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,15 @@

## Implementations

| Language | Confirmed Compatible with Spec Version | Minimum Version Confirmed | Implementation |
| ---------- | -------------------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| C | 0.1.0-preview | 0.1.0 | [cache.h](https://github.com/aws/aws-encryption-sdk-c/blob/master/include/aws/cryptosdk/cache.h) |
| Javascript | 0.1.0-preview | 0.1.0 | [cryptographic_materials_cache.ts](https://github.com/awslabs/aws-encryption-sdk-javascript/blob/master/modules/cache-material/src/cryptographic_materials_cache.ts) |
| Python | 0.1.0-preview | 1.3.0 | [caches/base.py](https://github.com/aws/aws-encryption-sdk-python/blob/master/src/aws_encryption_sdk/caches/base.py) |
| Java | 0.1.0-preview | 1.3.0 | [CryptoMaterialsCache.java](https://github.com/aws/aws-encryption-sdk-java/blob/master/src/main/java/com/amazonaws/encryptionsdk/caching/CryptoMaterialsCache.java) |
| Language | Confirmed Compatible with Spec Version | Minimum Version Confirmed | Implementation |
| ------------ | -------------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| C | 0.1.0-preview | 0.1.0 | [cache.h](https://github.com/aws/aws-encryption-sdk-c/blob/master/include/aws/cryptosdk/cache.h) |
| Javascript | 0.1.0-preview | 0.1.0 | [cryptographic_materials_cache.ts](https://github.com/awslabs/aws-encryption-sdk-javascript/blob/master/modules/cache-material/src/cryptographic_materials_cache.ts) |
| Python | 0.1.0-preview | 1.3.0 | [caches/base.py](https://github.com/aws/aws-encryption-sdk-python/blob/master/src/aws_encryption_sdk/caches/base.py) |
| Java | 0.1.0-preview | 1.3.0 | [CryptoMaterialsCache.java](https://github.com/aws/aws-encryption-sdk-java/blob/master/src/main/java/com/amazonaws/encryptionsdk/caching/CryptoMaterialsCache.java) |
| Dafny | 0.5.2 | 1.0.0 | [ICryptographicMaterialsCache in AwsCryptographyMaterialProvidersTypes.dfy](https://github.com/aws/aws-cryptographic-material-providers-library/blob/main/AwsCryptographicMaterialProviders/dafny/AwsCryptographicMaterialProviders/Model/AwsCryptographyMaterialProvidersTypes.dfy#L858-L1022) |
| Java (Dafny) | 0.5.2 | 1.0.0 | [ICryptographicMaterialsCache.java](https://github.com/aws/aws-cryptographic-material-providers-library/blob/main/AwsCryptographicMaterialProviders/runtimes/java/src/main/smithy-generated/software/amazon/cryptography/materialproviders/ICryptographicMaterialsCache.java) |
| .NET | 0.5.2 | 1.0.0 | [CryptographicMaterialsCacheBase.cs](https://github.com/aws/aws-cryptographic-material-providers-library/blob/main/AwsCryptographicMaterialProviders/runtimes/net/Generated/AwsCryptographicMaterialProviders/CryptographicMaterialsCacheBase.cs) |

## Overview

Expand Down
15 changes: 9 additions & 6 deletions framework/local-cryptographic-materials-cache.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,15 @@

## Implementations

| Language | Confirmed Compatible with Spec Version | Minimum Version Confirmed | Implementation |
| ---------- | -------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| C | n/a | n/a | [local_cache.c](https://github.com/aws/aws-encryption-sdk-c/blob/master/source/local_cache.c) |
| Javascript | 0.1.0 | 0.1.0 | [get_local_cryptographic_materials_cache.ts](https://github.com/aws/aws-encryption-sdk-javascript/blob/master/modules/cache-material/src/get_local_cryptographic_materials_cache.ts) |
| Python | n/a | n/a | [caches/local.py](https://github.com/aws/aws-encryption-sdk-python/blob/master/src/aws_encryption_sdk/caches/local.py) |
| Java | 0.1.0 | 1.3.0 | [LocalCryptoMaterialsCache.java](https://github.com/aws/aws-encryption-sdk-java/blob/master/src/main/java/com/amazonaws/encryptionsdk/caching/LocalCryptoMaterialsCache.java) |
| Language | Confirmed Compatible with Spec Version | Minimum Version Confirmed | Implementation |
| ------------ | -------------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| C | n/a | n/a | [local_cache.c](https://github.com/aws/aws-encryption-sdk-c/blob/master/source/local_cache.c) |
| Javascript | 0.1.0 | 0.1.0 | [get_local_cryptographic_materials_cache.ts](https://github.com/aws/aws-encryption-sdk-javascript/blob/master/modules/cache-material/src/get_local_cryptographic_materials_cache.ts) |
| Python | n/a | n/a | [caches/local.py](https://github.com/aws/aws-encryption-sdk-python/blob/master/src/aws_encryption_sdk/caches/local.py) |
| Java | 0.1.0 | 1.3.0 | [LocalCryptoMaterialsCache.java](https://github.com/aws/aws-encryption-sdk-java/blob/master/src/main/java/com/amazonaws/encryptionsdk/caching/LocalCryptoMaterialsCache.java) |
| Dafny | 0.2.0 | 1.0.0 | [LocalCMC.dfy](https://github.com/aws/aws-cryptographic-material-providers-library/blob/main/AwsCryptographicMaterialProviders/dafny/AwsCryptographicMaterialProviders/src/CMCs/LocalCMC.dfy) |
| Java (Dafny) | 0.2.0 | 1.0.0 | [CryptographicMaterialsCache.java](https://github.com/aws/aws-cryptographic-material-providers-library/blob/main/AwsCryptographicMaterialProviders/runtimes/java/src/main/smithy-generated/software/amazon/cryptography/materialproviders/CryptographicMaterialsCache.java) |
| .NET | 0.2.0 | 1.0.0 | [CryptographicMaterialsCache.cs](https://github.com/aws/aws-cryptographic-material-providers-library/blob/main/AwsCryptographicMaterialProviders/runtimes/net/Generated/AwsCryptographicMaterialProviders/CryptographicMaterialsCache.cs) |

## Overview

Expand Down

0 comments on commit d08ba88

Please sign in to comment.