diff --git a/source/client-side-encryption/client-side-encryption.rst b/source/client-side-encryption/client-side-encryption.rst index 4cecd38ec3..6052ade74b 100644 --- a/source/client-side-encryption/client-side-encryption.rst +++ b/source/client-side-encryption/client-side-encryption.rst @@ -4,8 +4,8 @@ Client Side Encryption :Status: Accepted :Minimum Server Version: 4.2 (CSFLE), 6.0 (Queryable Encryption) -:Last Modified: 2022-11-28 -:Version: 1.11.1 +:Last Modified: 2022-11-30 +:Version: 1.12.0 .. _lmc-c-api: https://github.com/mongodb/libmongocrypt/blob/master/src/mongocrypt.h.in @@ -1050,6 +1050,17 @@ ClientEncryption // Returns an encrypted value (BSON binary of subtype 6). The underlying implementation MAY return an error for prohibited BSON values. encrypt(value: BsonValue, opts: EncryptOpts): Binary; + // encryptExpression encrypts a Match Expression or Aggregate Expression to query a range index. + // `expr` is expected to be a BSON document of one of the following forms: + // 1. A Match Expression of this form: + // {$and: [{: {$gt: }}, {: {$lt: }}]} + // 2. An Aggregate Expression of this form: + // {$and: [{$gt: [, ]}, {$lt: [, ]}] + // $gt may also be $gte. $lt may also be $lte. + // Only supported when queryType is "rangePreview" and algorithm is "RangePreview". + // NOTE: The Range algorithm is experimental only. It is not intended for public use. It is subject to breaking changes. + encryptExpression(expr: Document, opts: EncryptOpts): Document; + // Decrypts an encrypted value (BSON binary of subtype 6). // Returns the original BSON value. decrypt(value: Binary): BsonValue; @@ -1238,6 +1249,21 @@ EncryptOpts algorithm: String, contentionFactor: Optional, queryType: Optional + rangeOpts: Optional + } + + // NOTE: The Range algorithm is experimental only. It is not intended for public use. It is subject to breaking changes. + // RangeOpts specifies index options for a Queryable Encryption field supporting "rangePreview" queries. + // min, max, sparsity, and range must match the values set in the encryptedFields of the destination collection. + // For double and decimal128, min/max/precision must all be set, or all be unset. + class RangeOpts { + // min is required if precision is set. + min: Optional, + // max is required if precision is set. + max: Optional, + sparsity: Int64, + // precision may only be set for double or decimal128. + precision: Optional } Explicit encryption requires a key and algorithm. Keys are either @@ -1258,24 +1284,39 @@ One of the strings: - "AEAD_AES_256_CBC_HMAC_SHA_512-Random" - "Indexed" - "Unindexed" +- "RangePreview" -The result of explicit encryption with the "Indexed" algorithm must be processed by the server to insert or query. Drivers MUST document the following behavior: +The result of explicit encryption with the "Indexed" or "RangePreview" algorithm must be processed by the server to insert or query. Drivers MUST document the following behavior: - To insert or query with an "Indexed" encrypted payload, use a ``MongoClient`` configured with ``AutoEncryptionOpts``. + To insert or query with an "Indexed" or "RangePreview" encrypted payload, use a ``MongoClient`` configured with ``AutoEncryptionOpts``. ``AutoEncryptionOpts.bypassQueryAnalysis`` may be true. ``AutoEncryptionOpts.bypassAutoEncryption`` must be false. +NOTE: The Range algorithm is experimental only. It is not intended for public use. It is subject to breaking changes. + contentionFactor ^^^^^^^^^^^^^^^^ -contentionFactor only applies when algorithm is "Indexed". +contentionFactor only applies when algorithm is "Indexed" or "RangePreview". It is an error to set contentionFactor when algorithm is not "Indexed". +NOTE: The Range algorithm is experimental only. It is not intended for public use. It is subject to breaking changes. + queryType ^^^^^^^^^ One of the strings: - "equality" +- "rangePreview" + +queryType only applies when algorithm is "Indexed" or "RangePreview". +It is an error to set queryType when algorithm is not "Indexed" or "RangePreview". -queryType only applies when algorithm is "Indexed". -It is an error to set queryType when algorithm is not "Indexed". +NOTE: The Range algorithm is experimental only. It is not intended for public use. It is subject to breaking changes. + +rangeOpts +^^^^^^^^^ +rangeOpts only applies when algorithm is "rangePreview". +It is an error to set rangeOpts when algorithm is not "rangePreview". + +NOTE: The Range algorithm is experimental only. It is not intended for public use. It is subject to breaking changes. User facing API: When Auto Encryption Fails =========================================== @@ -2531,6 +2572,48 @@ A string value helps with future compatibility. When new values of QueryType and IndexType are added in libmongocrypt, users would only need to upgrade libmongocrypt, and not the driver, to use the new values. +Why is there an encryptExpression helper? +----------------------------------------- + +Querying a range index requires encrypting a lower bound (value for ``$gt`` or ``$gte``) and upper bound (value for ``$lt`` or ``$lte``) payload. +A rejected alternative API is to encrypt the lower and upper bound payloads separately. +The lower and upper bound payloads must have a unique matching UUID. The lower and upper bound payloads are unique. +This API requires handling the UUID and distinguishing the upper and lower bounds. Here are examples showing possible errors: + +.. code:: + + uuid = UUID() + lOpts = EncryptOpts( + keyId=keyId, algorithm="range", queryType="range", uuid=uuid, bound="lower") + lower = clientEncryption.encrypt (value=30, lOpts) + uOpts = EncryptOpts( + keyId=keyId, algorithm="range", queryType="range", uuid=uuid, bound="upper") + upper = clientEncryption.encrypt (value=40, uOpts) + + # Both bounds match UUID ... OK + db.coll.find_one ({"age": {"$gt": lower, "$lt": upper }}) + + # Upper bound is used as a lower bound ... ERROR! + db.coll.find_one ({"age": {"$gt": upper }}) + + lower2 = clientEncryption.encrypt (value=35, lOpts) + + # UUID is re-used ... ERROR! + db.coll.find_one ({ "$or": [ + {"age": {"$gt": lower, "$lt": upper }}, + {"age": {"$gt": lower2 }} + ]}) + + # UUID does not match between lower and upper bound ... ERROR! + db.coll.find_one ({ "age": {"$gt": lower2, "$lt": upper }}) + +Requiring an Aggregate Expression or Match Expression hides the UUID and handles both bounds. + +Returning an Aggregate Expression or Match Expression as a BSON document motivated adding a new ``ClientEncryption.encryptExpression()`` helper. ``ClientEncryption.encrypt()`` cannot be reused since it returns a Binary. + +To limit scope, only $and is supported. Support for other operators ($eq, $in) can be added in the future if desired. + + Future work =========== @@ -2604,6 +2687,7 @@ explicit session parameter as described in the Changelog ========= +:2022-11-30: Add ``Range``. :2022-11-28: Permit `tlsDisableOCSPEndpointCheck` in KMS TLS options. :2022-11-27: Fix typo for references to ``cryptSharedLibRequired`` option. :2022-11-10: Defined a ``CreateEncryptedCollection`` helper for creating new diff --git a/source/client-side-encryption/etc/data/range-encryptedFields-Date.json b/source/client-side-encryption/etc/data/range-encryptedFields-Date.json new file mode 100644 index 0000000000..e19fc1e182 --- /dev/null +++ b/source/client-side-encryption/etc/data/range-encryptedFields-Date.json @@ -0,0 +1,30 @@ +{ + "fields": [ + { + "keyId": { + "$binary": { + "base64": "EjRWeBI0mHYSNBI0VniQEg==", + "subType": "04" + } + }, + "path": "encryptedDate", + "bsonType": "date", + "queries": { + "queryType": "rangePreview", + "sparsity": { + "$numberLong": "1" + }, + "min": { + "$date": { + "$numberLong": "0" + } + }, + "max": { + "$date": { + "$numberLong": "200" + } + } + } + } + ] +} diff --git a/source/client-side-encryption/etc/data/range-encryptedFields-DoubleNoPrecision.json b/source/client-side-encryption/etc/data/range-encryptedFields-DoubleNoPrecision.json new file mode 100644 index 0000000000..4af6422714 --- /dev/null +++ b/source/client-side-encryption/etc/data/range-encryptedFields-DoubleNoPrecision.json @@ -0,0 +1,21 @@ +{ + "fields": [ + { + "keyId": { + "$binary": { + "base64": "EjRWeBI0mHYSNBI0VniQEg==", + "subType": "04" + } + }, + "path": "encryptedDoubleNoPrecision", + "bsonType": "double", + "queries": { + "queryType": "rangePreview", + "sparsity": { + "$numberLong": "1" + } + } + } + ] + } + \ No newline at end of file diff --git a/source/client-side-encryption/etc/data/range-encryptedFields-DoublePrecision.json b/source/client-side-encryption/etc/data/range-encryptedFields-DoublePrecision.json new file mode 100644 index 0000000000..c1f388219d --- /dev/null +++ b/source/client-side-encryption/etc/data/range-encryptedFields-DoublePrecision.json @@ -0,0 +1,30 @@ +{ + "fields": [ + { + "keyId": { + "$binary": { + "base64": "EjRWeBI0mHYSNBI0VniQEg==", + "subType": "04" + } + }, + "path": "encryptedDoublePrecision", + "bsonType": "double", + "queries": { + "queryType": "rangePreview", + "sparsity": { + "$numberLong": "1" + }, + "min": { + "$numberDouble": "0.0" + }, + "max": { + "$numberDouble": "200.0" + }, + "precision": { + "$numberInt": "2" + } + } + } + ] + } + \ No newline at end of file diff --git a/source/client-side-encryption/etc/data/range-encryptedFields-Int.json b/source/client-side-encryption/etc/data/range-encryptedFields-Int.json new file mode 100644 index 0000000000..217bf6743c --- /dev/null +++ b/source/client-side-encryption/etc/data/range-encryptedFields-Int.json @@ -0,0 +1,27 @@ +{ + "fields": [ + { + "keyId": { + "$binary": { + "base64": "EjRWeBI0mHYSNBI0VniQEg==", + "subType": "04" + } + }, + "path": "encryptedInt", + "bsonType": "int", + "queries": { + "queryType": "rangePreview", + "sparsity": { + "$numberLong": "1" + }, + "min": { + "$numberInt": "0" + }, + "max": { + "$numberInt": "200" + } + } + } + ] + } + \ No newline at end of file diff --git a/source/client-side-encryption/etc/data/range-encryptedFields-Long.json b/source/client-side-encryption/etc/data/range-encryptedFields-Long.json new file mode 100644 index 0000000000..0fb87edaef --- /dev/null +++ b/source/client-side-encryption/etc/data/range-encryptedFields-Long.json @@ -0,0 +1,27 @@ +{ + "fields": [ + { + "keyId": { + "$binary": { + "base64": "EjRWeBI0mHYSNBI0VniQEg==", + "subType": "04" + } + }, + "path": "encryptedLong", + "bsonType": "long", + "queries": { + "queryType": "rangePreview", + "sparsity": { + "$numberLong": "1" + }, + "min": { + "$numberLong": "0" + }, + "max": { + "$numberLong": "200" + } + } + } + ] + } + \ No newline at end of file diff --git a/source/client-side-encryption/tests/README.rst b/source/client-side-encryption/tests/README.rst index 474d63245c..803e771c05 100644 --- a/source/client-side-encryption/tests/README.rst +++ b/source/client-side-encryption/tests/README.rst @@ -2615,4 +2615,308 @@ with encrypted value. ssn: } - Expect success. \ No newline at end of file + Expect success. + +22. Range Explicit Encryption +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The Range Explicit Encryption tests require MongoDB server 6.2+. The tests must not run against a standalone. + +Each of the following test cases must pass for each of the supported types (``DoublePrecision``, ``DoubleNoPrecision``, ``Date``, ``Int``, and ``Long``), unless it is stated the type should be skipped. + +Before running each of the following test cases, perform the following Test Setup. + +Test Setup +`````````` +Load the file for the specific data type being tested ``encryptedFields-.json``. For example, for ``Int`` load `range-encryptedFields-Int.json `_ as ``encryptedFields``. + +Load the file `key1-document.json `_ as ``key1Document``. + +Read the ``"_id"`` field of ``key1Document`` as ``key1ID``. + +Drop and create the collection ``db.explicit_encryption`` using ``encryptedFields`` as an option. See `FLE 2 CreateCollection() and Collection.Drop() `_. + +Drop and create the collection ``keyvault.datakeys``. + +Insert ``key1Document`` in ``keyvault.datakeys`` with majority write concern. + +Create a MongoClient named ``keyVaultClient``. + +Create a ClientEncryption object named ``clientEncryption`` with these options: + +.. code:: typescript + + ClientEncryptionOpts { + keyVaultClient: ; + keyVaultNamespace: "keyvault.datakeys"; + kmsProviders: { "local": { "key": } } + } + +Create a MongoClient named ``encryptedClient`` with these ``AutoEncryptionOpts``: + +.. code:: typescript + + AutoEncryptionOpts { + keyVaultNamespace: "keyvault.datakeys"; + kmsProviders: { "local": { "key": } } + bypassQueryAnalysis: true + } + +The remaining tasks require setting ``RangeOpts``. `Test Setup: RangeOpts`_ lists the values to use for ``RangeOpts`` for each of the supported data types. + +Use ``clientEncryption`` to encrypt these values: 0, 6, 30, and 200. Ensure the type matches with the type of the encrypted field. For example, if the encrypted field is ``encryptedDoubleNoPrecision`` encrypt the value 6.0. + +Encrypt these values with the matching ``RangeOpts`` listed in `Test Setup: RangeOpts`_ and these ``EncryptOpts``: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + contentionFactor: 0 + } + +Use ``encryptedClient`` to insert these documents into ``db.explicit_encryption``: + +- ``{ "encrypted": , _id: 0 }`` +- ``{ "encrypted": , _id: 1 }`` +- ``{ "encrypted": , _id: 2 }`` +- ``{ "encrypted": , _id: 3 }`` + + +Test Setup: RangeOpts +````````````````````` +This section lists the values to use for ``RangeOpts`` for each of the supported data types, since each data type requires a different ``RangeOpts``. + +Each test listed in the cases below must pass for all supported data types unless it is stated the type should be skipped. + +#. DoubleNoPrecision + + .. code:: typescript + + class RangeOpts { + sparsity: 1 + } + +#. DoublePrecision + + .. code:: typescript + + class RangeOpts { + min: { "$numberDouble": "0" }, + max: { "$numberDouble": "200" }, + sparsity: 1, + precision: 2 + } + +#. Date + + .. code:: typescript + + class RangeOpts { + min: {"$date": { "$numberLong": "0" } } , + max: {"$date": { "$numberLong": "200" } }, + sparsity: 1 + } + +#. Int + + .. code:: typescript + + class RangeOpts { + min: {"$numberInt": "0" } , + max: {"$numberInt": "200" }, + sparsity: 1 + } + +#. Long + + .. code:: typescript + + class RangeOpts { + min: {"$numberLong": "0" } , + max: {"$numberLong": "200" }, + sparsity: 1 + } + +Case 1: can decrypt a payload +````````````````````````````` +Use ``clientEncryption.encrypt()`` to encrypt the value 6. Ensure the type matches with the type of the encrypted field. For example, if the encrypted field is ``encryptedDoubleNoPrecision`` encrypt the double value 6.0. + +Store the result in ``insertPayload``. + +Encrypt with the matching ``RangeOpts`` listed in `Test Setup: RangeOpts`_ and these ``EncryptOpts``: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + contentionFactor: 0 + } + +Use ``clientEncryption`` to decrypt ``insertPayload``. Assert the returned value equals 6. + +Case 2: can find encrypted range and return the maximum +``````````````````````````````````````````````````````` +Use ``clientEncryption.encryptExpression()`` to encrypt this query: + +.. code:: javascript + + //convert 6 and 200 to match the type of the encrypted field. + {"$and": [{"encrypted": {"$gte": 6}}, {"encrypted": {"$lte": 200}}]} + +Use the matching ``RangeOpts`` listed in `Test Setup: RangeOpts`_ and these ``EncryptOpts`` to encrypt the query: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + queryType: "rangePreview", + contentionFactor: 0 + } + +Store the result in ``findPayload``. + +Use ``encryptedClient`` to run a "find" operation on the ``db.explicit_encryption`` collection with the filter ``findPayload`` and sort the results by ``_id``. + +Assert these three documents ``{ "encrypted": 6 }, { "encrypted": 30 }, { "encrypted": 200}`` are returned. + + +Case 3: can find encrypted range and return the minimum +``````````````````````````````````````````````````````` +Use ``clientEncryption.encryptExpression()`` to encrypt this query: + + +.. code:: javascript + + //convert 0 and 6 to match the type of the encrypted field. + {"$and": [{"encrypted": {"$gte": 0}}, {"encrypted": {"$lte": 6}}]} + +Use the matching ``RangeOpts`` listed in `Test Setup: RangeOpts`_ and these ``EncryptOpts`` to encrypt the query: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + queryType: "rangePreview", + contentionFactor: 0 + } + +Store the result in ``findPayload``. + +Use ``encryptedClient`` to run a "find" operation on the ``db.explicit_encryption`` collection with the filter ``findPayload`` and sort the results by ``_id``. + +Assert these two documents ``{ "encrypted": 0 }, { "encrypted": 6 }`` are returned. + +Case 4: can find encrypted range with an open range query +````````````````````````````````````````````````````````` +Use ``clientEncryption.encryptExpression()`` to encrypt this query: + +.. code:: javascript + + //convert 30 to match the type of the encrypted field. + {"$and": [{"encrypted": {"$gt": 30}}]} + +Use the matching ``RangeOpts`` listed in `Test Setup: RangeOpts`_ and these ``EncryptOpts`` to encrypt the query: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + queryType: "rangePreview", + contentionFactor: 0 + } + +Store the result in ``findPayload``. + +Use ``encryptedClient`` to run a "find" operation on the ``db.explicit_encryption`` collection with the filter ``findPayload`` and sort the results by ``_id``. + +Assert that only this document ``{ "encrypted": 200 }`` is returned. + +Case 5: can run an aggregation expression inside $expr +`````````````````````````````````````````````````````` +Use ``clientEncryption.encryptExpression()`` to encrypt this query: + +.. code:: javascript + + {'$and': [ { '$lt': [ '$encrypted', 30 ] } ] } } + +Use the matching ``RangeOpts`` listed in `Test Setup: RangeOpts`_ and these ``EncryptOpts`` to encrypt the query: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + queryType: "rangePreview", + contentionFactor: 0 + } + +Store the result in ``findPayload``. + +Use ``encryptedClient`` to run a "find" operation on the ``db.explicit_encryption`` collection with the filter ``{'$expr' : }`` and sort the results by ``_id``. + +Assert that these two documents ``{ "encrypted": 0 }, { "encrypted": 6 }`` are returned. + +Case 6: encrypting a document greater than the maximum errors +````````````````````````````````````````````````````````````` +This test case should be skipped if the encrypted field is ``encryptedDoubleNoPrecision``. + +Use ``clientEncryption.encrypt()`` to try to encrypt the value 201 with the matching ``RangeOpts`` listed in `Test Setup: RangeOpts`_ and these ``EncryptOpts``: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + contentionFactor: 0 + } + +Ensure 201 matches the type of the encrypted field. The error should be raised because 201 is greater than the maximum value in ``RangeOpts``. + +Assert that an error was raised. + +Case 7: encrypting a document of a different type errors +```````````````````````````````````````````````````````` +This test case should be skipped if the encrypted field is ``encryptedDoubleNoPrecision``. + +For all the tests below use these ``EncryptOpts``: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + contentionFactor: 0 + } + +If the encrypted field is ``encryptedInt`` insert ``{ "encryptedInt": { "$numberDouble": "6" } }``. +Otherwise, insert ``{ "encrypted": { "$numberInt": "6" }``. +Assert an error was raised. + + +Case 8: setting precision errors if the type is not a double +```````````````````````````````````````````````````````````` +This test case should be skipped if the encrypted field is ``encryptedDoublePrecision`` or ``encryptedDoubleNoPrecision``. + +Use ``clientEncryption.encrypt()`` to try to encrypt the value 6 with these ``EncryptOpts`` and these ``RangeOpts``: + +.. code:: typescript + + class EncryptOpts { + keyId : + algorithm: "RangePreview", + contentionFactor: 0 + } + + class RangeOpts { + min: 0, + max: 200, + sparsity: 1, + precision: 2, + } + +Assert an error was raised.