Skip to content

Commit

Permalink
DRIVERS-2524 do not create or drop eccCollection (#1396)
Browse files Browse the repository at this point in the history
* do not create or drop the eccCollection

* DRIVERS-2586 do not document `escCollection` and `ecocCollection` options

* remove eccCollection from fle2v2-CreateCollection

* remove `eccCollection` from `encryptedFields` data files

* regenerate fle2v2 tests

* remove eccCollection from fle2v2-EncryptedFields-vs-EncryptedFieldsMap

* remove eccCollection from fle2v2-Range-WrongType

* assert eccCollection is not created

* remove incorrect comment

* add wire version check and test

* remove `escCollection` and `ecocCollection` from test data

* regenerate tests

* add $$exists to legacy format

* add $$exists to tests

Assert that state collections names are not sent to server

* test encryptedFields is consulted for metadata collection names

* change SHOULD to MUST

* use YAML anchors for collection names

* add comment that ecc collection is no longer created for QEv2

* remove `encryptedFieldsMap with cyclic entries does not loop`

* use YAML anchors for encryptedFields

* use `null`, not $$exists

* Revert "add $$exists to legacy format"

This reverts commit 72280f9.

* swap order of `base64` and `subType`

* remove unnecessary anchor

* regenerate fle2v2-CreateCollection-OldServer
  • Loading branch information
kevinAlbs committed Apr 12, 2023
1 parent 364464f commit aa28f78
Show file tree
Hide file tree
Showing 125 changed files with 481 additions and 1,724 deletions.
37 changes: 27 additions & 10 deletions source/client-side-encryption/client-side-encryption.rst
Expand Up @@ -115,7 +115,6 @@ encryptedFields
{
"escCollection": "enxcol_.CollectionName.esc",
"eccCollection": "enxcol_.CollectionName.ecc",
"ecocCollection": "enxcol_.CollectionName.ecoc",
"fields": [
{
Expand All @@ -134,7 +133,6 @@ encryptedFields
The acronyms within ``encryptedFields`` are defined as follows:

* ECC: Encrypted Cache Collection
* ECOC: Encrypted Compaction Collection
* ESC: Encrypted State Collection

Expand Down Expand Up @@ -914,6 +912,17 @@ Drivers MUST support a BSON document option named ``encryptedFields`` for any
will be interpreted by the helper method and MUST be passed to the `create`_
command.

.. note::
Users are not expected to set the ``escCollection`` and ``ecocCollection`` in
``encryptedFields``. SERVER-74069 added server-side validation for those fields
and no longer allows names to deviate from the following:

- ``enxcol_.<collectionName>.esc``
- ``enxcol_.<collectionName>.ecoc`
Drivers MUST NOT document the ``escCollection`` and ``ecocCollection``
options.

For a helper function, ``CreateCollection(collectionName, collectionOptions)``
with the name of the database associated as `dbName`, look up the encrypted
fields ``encryptedFields`` for the collection as
Expand All @@ -924,17 +933,16 @@ If a set of ``encryptedFields`` was found, then do the following operations. If
any of the following operations error, the remaining operations are not
attempted:

- Check the wire version of the writable server. If the wire version is less
than 21 (for server 7.0.0), return an error containing the error message:
"Driver support of Queryable Encryption is incompatible with server. Upgrade
server to use Queryable Encryption."
- Create the collection with name ``encryptedFields["escCollection"]`` as a
clustered collection using the options
``{clusteredIndex: {key: {_id: 1}, unique: true}}``. If
``encryptedFields["escCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.esc``. Creating this collection MUST NOT check if
the collection namespace is in the ``AutoEncryptionOpts.encryptedFieldsMap``.
- Create the collection with name ``encryptedFields["eccCollection"]`` as a
clustered collection using the options
``{clusteredIndex: {key: {_id: 1}, unique: true}}``. If
``encryptedFields["eccCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.ecc``. Creating this collection MUST NOT check if
the collection namespace is in the ``AutoEncryptionOpts.encryptedFieldsMap``.
- Create the collection with name ``encryptedFields["ecocCollection"]`` as a
clustered collection using the options
Expand Down Expand Up @@ -1002,6 +1010,17 @@ Drivers MUST support a BSON document option named ``encryptedFields`` for any
``Collection.drop()``). This option will only be interpreted by the helper
method and MUST NOT be passed to the `drop`_ command.

.. note::
Users are not expected to set the ``escCollection`` and ``ecocCollection`` in
``encryptedFields``. SERVER-74069 added server-side validation for those fields
and no longer allows names to deviate from the following:

- ``enxcol_.<collectionName>.esc``
- ``enxcol_.<collectionName>.ecoc`
Drivers SHOULD NOT document the ``escCollection`` and ``ecocCollection``
options.

For a helper function ``DropCollection(dropOptions)`` with associated collection
named `collName` and database named `dbName`, look up the encrypted fields
``encryptedFields`` as `GetEncryptedFields(dropOptions, collName, dbname, true)`
Expand All @@ -1015,9 +1034,6 @@ are not attempted. A ``namespace not found`` error returned from the server
- Drop the collection with name ``encryptedFields["escCollection"]``. If
``encryptedFields["escCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.esc``.
- Drop the collection with name ``encryptedFields["eccCollection"]``. If
``encryptedFields["eccCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.ecc``.
- Drop the collection with name ``encryptedFields["ecocCollection"]``. If
``encryptedFields["ecocCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.ecoc``.
Expand Down Expand Up @@ -2714,6 +2730,7 @@ explicit session parameter as described in the
Changelog
=========
:2023-03-30: Remove ECC collection
:2023-02-01: Replace ``DataKeyOpts`` with ``masterKey`` in ``createEncryptedCollection``.
:2023-01-31: ``createEncryptedCollection`` does not check AutoEncryptionOptions for the encryptedFieldsMap.
:2023-01-30: Return ``encryptedFields`` on ``CreateCollection`` error.
Expand Down
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -33,4 +30,4 @@
}
}
]
}
}
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand All @@ -23,4 +20,4 @@
}
}
]
}
}
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -32,4 +29,4 @@
}
}
]
}
}
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand All @@ -23,4 +20,4 @@
}
}
]
}
}
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -32,4 +29,4 @@
}
}
]
}
}
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -29,4 +26,4 @@
}
}
]
}
}
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -29,4 +26,4 @@
}
}
]
}
}
5 changes: 1 addition & 4 deletions source/client-side-encryption/etc/data/encryptedFields.json
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -30,4 +27,4 @@
"bsonType": "string"
}
]
}
}
Expand Up @@ -18,9 +18,6 @@ tests:
local: {{ local_provider() }}
encryptedFieldsMap: {
"default.default": {
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": []
}
}
Expand Down
Expand Up @@ -45,9 +45,7 @@ tests:
},
"schema": {
"default.default": {
"fields": [],
"escCollection": "enxcol_.default.esc",
"ecocCollection": "enxcol_.default.ecoc"
"fields": []
}
}
}
Expand Down
Expand Up @@ -14,9 +14,6 @@
"collection_name": "default",
"data": [],
"encrypted_fields": {
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down
Expand Up @@ -8,7 +8,7 @@ runOn:
database_name: &database_name "default"
collection_name: &collection_name "default"
data: []
encrypted_fields: &encrypted_fields {'escCollection': 'enxcol_.default.esc', 'eccCollection': 'enxcol_.default.ecc', 'ecocCollection': 'enxcol_.default.ecoc', 'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
encrypted_fields: &encrypted_fields {'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
key_vault_data: [{'_id': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'sHe0kz57YW7v8g9VP9sf/+K1ex4JqKc5rf/URX3n3p8XdZ6+15uXPaSayC6adWbNxkFskuMCOifDoTT+rkqMtFkDclOy884RuGGtUysq3X7zkAWYTKi8QAfKkajvVbZl2y23UqgVasdQu3OVBQCrH/xY00nNAs/52e958nVjBuzQkSb1T8pKJAyjZsHJ60+FtnfafDZSTAIBJYn7UWBCwQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}} ]

tests:
Expand Down
Expand Up @@ -14,9 +14,6 @@
"collection_name": "default",
"data": [],
"encrypted_fields": {
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down
Expand Up @@ -8,7 +8,7 @@ runOn:
database_name: &database_name "default"
collection_name: &collection_name "default"
data: []
encrypted_fields: &encrypted_fields {'escCollection': 'enxcol_.default.esc', 'eccCollection': 'enxcol_.default.ecc', 'ecocCollection': 'enxcol_.default.ecoc', 'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
encrypted_fields: &encrypted_fields {'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
key_vault_data: [ {'_id': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'sHe0kz57YW7v8g9VP9sf/+K1ex4JqKc5rf/URX3n3p8XdZ6+15uXPaSayC6adWbNxkFskuMCOifDoTT+rkqMtFkDclOy884RuGGtUysq3X7zkAWYTKi8QAfKkajvVbZl2y23UqgVasdQu3OVBQCrH/xY00nNAs/52e958nVjBuzQkSb1T8pKJAyjZsHJ60+FtnfafDZSTAIBJYn7UWBCwQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}}, {'_id': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'HBk9BWihXExNDvTp1lUxOuxuZK2Pe2ZdVdlsxPEBkiO1bS4mG5NNDsQ7zVxJAH8BtdOYp72Ku4Y3nwc0BUpIKsvAKX4eYXtlhv5zUQxWdeNFhg9qK7qb8nqhnnLeT0f25jFSqzWJoT379hfwDeu0bebJHr35QrJ8myZdPMTEDYF08QYQ48ShRBli0S+QzBHHAQiM2iJNr4svg2WR8JSeWQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}} ]
tests:
- description: "Compact works"
Expand Down
@@ -0,0 +1,62 @@
{
"runOn": [
{
"minServerVersion": "6.0.0",
"maxServerVersion": "6.3.99",
"topology": [
"replicaset",
"sharded",
"load-balanced"
]
}
],
"database_name": "default",
"collection_name": "default",
"tests": [
{
"description": "driver returns an error if creating a QEv2 collection on unsupported server",
"clientOptions": {
"autoEncryptOpts": {
"kmsProviders": {
"aws": {}
},
"encryptedFieldsMap": {
"default.encryptedCollection": {
"fields": [
{
"path": "firstName",
"bsonType": "string",
"keyId": {
"$binary": {
"base64": "AAAAAAAAAAAAAAAAAAAAAA==",
"subType": "04"
}
}
}
]
}
}
}
},
"operations": [
{
"name": "dropCollection",
"object": "database",
"arguments": {
"collection": "encryptedCollection"
}
},
{
"name": "createCollection",
"object": "database",
"arguments": {
"collection": "encryptedCollection"
},
"result": {
"errorContains": "Driver support of Queryable Encryption is incompatible with server. Upgrade server to use Queryable Encryption."
}
}
]
}
]
}
@@ -0,0 +1,39 @@
# Requires libmongocrypt 1.8.0.
runOn:
- minServerVersion: "6.0.0"
maxServerVersion: "6.3.99"
# FLE 2 Encrypted collections are not supported on standalone.
topology: [ "replicaset", "sharded", "load-balanced" ]

database_name: &database_name "default"
collection_name: &collection_name "default"

tests:
- description: "driver returns an error if creating a QEv2 collection on unsupported server"
clientOptions:
autoEncryptOpts:
kmsProviders:
aws: {} # Credentials filled in from environment.
encryptedFieldsMap:
default.encryptedCollection: {
"fields": [
{
"path": "firstName",
"bsonType": "string",
"keyId": { "$binary": { "base64": "AAAAAAAAAAAAAAAAAAAAAA==", "subType": "04" }}
}
]
}

operations:
# Do an initial drop to remove collections that may exist from previous test runs.
- name: dropCollection
object: database
arguments:
collection: "encryptedCollection"
- name: createCollection
object: database
arguments:
collection: "encryptedCollection"
result:
errorContains: "Driver support of Queryable Encryption is incompatible with server. Upgrade server to use Queryable Encryption."

0 comments on commit aa28f78

Please sign in to comment.