PYTHON-4181 Use libmongocrypt native crypto when available for 10-50x better performance

[ShaneHarvey]

PYTHON-4181 Use libmongocrypt native crypto when available for better performance

TODO:

• Bundle crypto-enabled libmongocrypt on Mac and Windows.
• Add tests for both crypto and nocrypto builds.
• Add documentation.

Before:

[2024/02/22 11:54:56.554] Running benchmark with pymongocrypt: 1.9.0.dev0 libmongocrypt: 1.10.0-20240222+gitae4c445a80
[2024/02/22 11:54:56.554] Finished TestBulkDecryption, threads=1, median ops_per_second=13.17
[2024/02/22 11:54:56.554] Finished TestBulkDecryption, threads=2, median ops_per_second=5.24
[2024/02/22 11:54:56.554] Finished TestBulkDecryption, threads=8, median ops_per_second=4.41
[2024/02/22 11:54:56.554] Finished TestBulkDecryption, threads=64, median ops_per_second=4.01

After (10-50x higher throughput):

[2024/02/26 13:50:13.106] Running benchmark with pymongocrypt: 1.9.0.dev0 libmongocrypt: 1.8.1-20240226+gita6d8858092
[2024/02/26 13:50:13.106] Finished TestBulkDecryption, threads=1, median ops_per_second=132.25
[2024/02/26 13:50:13.106] Finished TestBulkDecryption, threads=2, median ops_per_second=171.99
[2024/02/26 13:50:13.106] Finished TestBulkDecryption, threads=8, median ops_per_second=204.54
[2024/02/26 13:50:13.106] Finished TestBulkDecryption, threads=64, median ops_per_second=224.22

[ShaneHarvey]

Need to address this problem on macOS <=10.14:

[2024/02/26 17:14:28.366] =================================== FAILURES ===================================
[2024/02/26 17:14:28.366] _________________ TestExplicitEncryption.test_encrypt_indexed __________________
[2024/02/26 17:14:28.366] self = <test.test_mongocrypt.TestExplicitEncryption testMethod=test_encrypt_indexed>
[2024/02/26 17:14:28.366]     def test_encrypt_indexed(self):
[2024/02/26 17:14:28.366]         key_path = 'keys/ABCDEFAB123498761234123456789012-local-document.json'
[2024/02/26 17:14:28.366]         key_id = json_data(key_path)['_id']
[2024/02/26 17:14:28.366]         encrypter = ExplicitEncrypter(MockCallback(
[2024/02/26 17:14:28.366]             key_docs=[bson_data(key_path)],
[2024/02/26 17:14:28.366]             kms_reply=http_data('kms-reply.txt')), self.mongo_crypt_opts())
[2024/02/26 17:14:28.366]         self.addCleanup(encrypter.close)
[2024/02/26 17:14:28.366]         val = {'v': 'value123'}
[2024/02/26 17:14:28.366]         encoded_val = bson.encode(val)
[2024/02/26 17:14:28.366]         for kwargs in [
[2024/02/26 17:14:28.366]             dict(algorithm='Indexed', contention_factor=0),
[2024/02/26 17:14:28.366]             dict(algorithm='Indexed', query_type='equality', contention_factor=0),
[2024/02/26 17:14:28.366]             dict(algorithm='Indexed', contention_factor=100),
[2024/02/26 17:14:28.366]             dict(algorithm='Unindexed'),
[2024/02/26 17:14:28.366]         ]:
[2024/02/26 17:14:28.366]             kwargs['key_id'] = key_id
[2024/02/26 17:14:28.366] >           encrypted = encrypter.encrypt(encoded_val, **kwargs)
[2024/02/26 17:14:28.366] test/test_mongocrypt.py:722:
[2024/02/26 17:14:28.366] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2024/02/26 17:14:28.366] pymongocrypt/explicit_encrypter.py:232: in encrypt
[2024/02/26 17:14:28.366]     return run_state_machine(ctx, self.callback)
[2024/02/26 17:14:28.366] pymongocrypt/state_machine.py:126: in run_state_machine
[2024/02/26 17:14:28.366]     return ctx.finish()
[2024/02/26 17:14:28.366] pymongocrypt/mongocrypt.py:457: in finish
[2024/02/26 17:14:28.366]     self._raise_from_status()
[2024/02/26 17:14:28.366] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2024/02/26 17:14:28.366] self = <pymongocrypt.mongocrypt.ExplicitEncryptionContext object at 0x108cac2d0>
[2024/02/26 17:14:28.366]     def _raise_from_status(self):
[2024/02/26 17:14:28.366]         status = lib.mongocrypt_status_new()
[2024/02/26 17:14:28.366]         try:
[2024/02/26 17:14:28.366]             lib.mongocrypt_ctx_status(self.__ctx, status)
[2024/02/26 17:14:28.366]             exc = MongoCryptError.from_status(status)
[2024/02/26 17:14:28.366]         finally:
[2024/02/26 17:14:28.366]             lib.mongocrypt_status_destroy(status)
[2024/02/26 17:14:28.366] >       raise exc
[2024/02/26 17:14:28.366] E       pymongocrypt.errors.MongoCryptError: error initializing cipher: Unimplemented (-4305). CTR mode is only supported on macOS 10.15+
[2024/02/26 17:14:28.366] pymongocrypt/mongocrypt.py:407: MongoCryptError
[2024/02/26 17:14:28.366] =========================== short test summary info ============================
[2024/02/26 17:14:28.366] FAILED test/test_mongocrypt.py::TestExplicitEncryption::test_encrypt_indexed - pymongocrypt.errors.MongoCryptError: error initializing cipher: Unimplemented (-4305). CTR mode is only supported on macOS 10.15+

https://spruce.mongodb.com/task/libmongocrypt_macos_x86_64_test_python_patch_ae4c445a80e8523752edbc8d3c6452c772d52dab_65dd36d17742ae0ce4523957_24_02_27_01_11_46/logs?execution=0

@kevinAlbs more generally, how can we be sure all crypto functions are supported when mongocrypt_is_crypto_available() returns True?

[kevinAlbs]

The error CTR mode is only supported on macOS 10.15+ is returned by libmongocrypt. Supporting CTR mode on macOS 10.14 and earlier was deliberately not done: MONGOCRYPT-440. This is the only crypto primitive with such a restriction.

@kevinAlbs more generally, how can we be sure all crypto functions are supported when mongocrypt_is_crypto_available() returns True?

libmongocrypt tests check the version of macOS. Perhaps do a similar check in Python?

[ShaneHarvey]

Thanks, since this isn't just a test issue I've opted to always register the CTR crypto callbacks on macOS < 10.15 (mongocrypt_setopt_aes_256_ctr) even when mongocrypt_is_crypto_available is True. This seems to work fine. Could you confirm it's supported to selectively register crypto callbacks like this?

[kevinAlbs]

Could you confirm it's supported to selectively register crypto callbacks like this?

I expect "yes". Filed MONGOCRYPT-647 to add tests within libmongocrypt to guarantee this behavior.

[ShaneHarvey]

Thanks all!