PYTHON-3466 Test crypt_shared with older server versions #1133

juliusgeo · 2023-01-11T01:17:38Z

I opened https://jira.mongodb.org/browse/PYTHON-3559 to address the intermittent test failures.

blink1073

LGTM!

ShaneHarvey

Any idea why test_via_loading_shared_library is failing on various platforms?

 [2023/01/11 01:24:21.646] ERROR: test_via_loading_shared_library (test.test_encryption.TestBypassSpawningMongocryptdProse)
 [2023/01/11 01:24:21.646] ----------------------------------------------------------------------
 [2023/01/11 01:24:21.646] Traceback (most recent call last):
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/encryption.py", line 77, in _wrap_encryption_errors
 [2023/01/11 01:24:21.646]     yield
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/encryption.py", line 365, in encrypt
 [2023/01/11 01:24:21.646]     encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/venv-encryption/lib/python3.9/site-packages/pymongocrypt/auto_encrypter.py", line 44, in encrypt
 [2023/01/11 01:24:21.646]     return run_state_machine(ctx, self.callback)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/venv-encryption/lib/python3.9/site-packages/pymongocrypt/state_machine.py", line 146, in run_state_machine
 [2023/01/11 01:24:21.646]     ctx.complete_mongo_operation()
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/venv-encryption/lib/python3.9/site-packages/pymongocrypt/mongocrypt.py", line 450, in complete_mongo_operation
 [2023/01/11 01:24:21.646]     self._raise_from_status()
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/venv-encryption/lib/python3.9/site-packages/pymongocrypt/mongocrypt.py", line 428, in _raise_from_status
 [2023/01/11 01:24:21.646]     raise exc
 [2023/01/11 01:24:21.646] pymongocrypt.errors.MongoCryptError: not all keys requested were satisfied
 [2023/01/11 01:24:21.646] During handling of the above exception, another exception occurred:
 [2023/01/11 01:24:21.646] Traceback (most recent call last):
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/test/test_encryption.py", line 1936, in test_via_loading_shared_library
 [2023/01/11 01:24:21.646]     client_encrypted.db.coll.insert_one({"encrypted": "test"})
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/collection.py", line 626, in insert_one
 [2023/01/11 01:24:21.646]     self._insert_one(
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/collection.py", line 567, in _insert_one
 [2023/01/11 01:24:21.646]     self.__database.client._retryable_write(acknowledged, _insert_command, session)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/mongo_client.py", line 1482, in _retryable_write
 [2023/01/11 01:24:21.646]     return self._retry_with_session(retryable, func, s, None)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/mongo_client.py", line 1349, in _retry_with_session
 [2023/01/11 01:24:21.646]     return self._retry_internal(retryable, func, session, bulk)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/_csot.py", line 105, in csot_wrapper
 [2023/01/11 01:24:21.646]     return func(self, *args, **kwargs)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/mongo_client.py", line 1390, in _retry_internal
 [2023/01/11 01:24:21.646]     return func(session, sock_info, retryable)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/collection.py", line 555, in _insert_command
 [2023/01/11 01:24:21.646]     result = sock_info.command(
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/pool.py", line 795, in command
 [2023/01/11 01:24:21.646]     self._raise_connection_failure(error)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/pool.py", line 767, in command
 [2023/01/11 01:24:21.646]     return command(
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/network.py", line 117, in command
 [2023/01/11 01:24:21.646]     spec = orig = client._encrypter.encrypt(dbname, spec, codec_options)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/encryption.py", line 368, in encrypt
 [2023/01/11 01:24:21.646]     return encrypt_cmd
 [2023/01/11 01:24:21.646]   File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
 [2023/01/11 01:24:21.646]     self.gen.throw(typ, value, traceback)
 [2023/01/11 01:24:21.646]   File "/System/Volumes/Data/data/mci/61a81ae5a14ebf93687fcf6d919fec61/src/pymongo/encryption.py", line 83, in _wrap_encryption_errors
 [2023/01/11 01:24:21.646]     raise EncryptionError(exc)
 [2023/01/11 01:24:21.646] pymongo.errors.EncryptionError: not all keys requested were satisfied

https://spruce.mongodb.com/task/mongo_python_driver_test_macos_encryption__platform~macos_1100_auth~auth_ssl~nossl_encryption~encryption_crypt_shared_old_server_test_5.0_replica_set_patch_eaea70bf08189d567838b3408bdc9fd5ae7cecf6_63be0e3357e85a19595c060f_23_01_11_01_17_40/logs?execution=0&sortBy=STATUS&sortDir=ASC

ShaneHarvey · 2023-01-11T20:18:08Z

Ah I see you already opened https://jira.mongodb.org/browse/PYTHON-3559

ShaneHarvey · 2023-01-11T20:25:21Z

.evergreen/config.yml

+          test_encryption: true
+          test_crypt_shared: true
+          test_crypt_shared_latest: true
+        batchtime: 10080  # 7 days


What do you think about testing this combination (test_crypt_shared_latest with <6.0 servers) in the encryption_crypt_shared axis instead of adding new variants? This would reduce the visual noise in the EVG UI. It might be as simple as removing the "rules" for removing these server versions for encryption: [ "encryption_crypt_shared" ]:

mongo-python-driver/.evergreen/config.yml

Lines 2856 to 2868 in a43f320

rules:

- if:

platform: "*"

python-version-windows: "*"

auth-ssl: "*"

encryption: [ "encryption_crypt_shared" ]

then:

remove_tasks:

- ".rapid"

- ".5.0"

- ".4.4"

- ".4.2"

- ".4.0"

I initially was going to do that, but the spec states that we must use the latest version of crypt shared for older server versions. Simply removing those excluded server versions attempts to download the version of crypt shared corresponding to that server version, which fails when <6.0. I thought this would be more clear than adding specific logic in run-tests.sh to use latest with older server versions.

I think it's still worthwhile in the long run so that we can avoid duplicating the "Encryption shared lib" variants.

That is fair. I will add a note in config.yml explaining that it has this new behavior.

juliusgeo · 2023-01-11T21:36:50Z

The new behavior can be seen here (4.4 server version):

 [2023/01/11 21:31:18.054] Testing CSFLE with crypt_shared lib version latest
 [2023/01/11 21:31:18.054] Download crypt_shared latest-enterprise for macos-x86_64

blink1073 · 2023-01-11T21:57:21Z

.evergreen/config.yml

@@ -2500,21 +2497,15 @@ axes:
        variables:
          test_encryption: true
        batchtime: 10080  # 7 days
+      # For server >6.0, use corresponding version shared lib.


Good catch, it should be >=. I will change the phrasing to make it more sensical.

ShaneHarvey · 2023-01-11T22:03:38Z

.evergreen/run-tests.sh

@@ -159,10 +159,10 @@ if [ -n "$TEST_ENCRYPTION" ]; then

    if [ -n "$TEST_CRYPT_SHARED" ]; then
        REAL_VERSION=$(mongod --version | head -n1 | cut -d v -f3 | tr -d "\r")
-        if [ "$MONGODB_VERSION" = "latest" ]; then
+        if [ "$MONGODB_VERSION" = "latest" ] || [ "$(echo "$MONGODB_VERSION" | cut -c1-1)" -lt  "6" ]; then


The drivers ticket says:

For server versions < 6.0, drivers MUST test with the latest major release of crypt_shared_ (currently 6.0).

Looking at the test logs, the drivers-evergreen-tools (DET) script already downloads the correct version:

[2023/01/11 21:30:50.692] Downloading crypt_shared package from http://downloads.10gen.com/osx/mongo_crypt_shared_v1-macos-x86_64-enterprise-6.0.2.tgz [2023/01/11 21:30:50.987] CRYPT_SHARED_LIB_PATH: /System/Volumes/Data/data/mci/3aa8d8033082dd76d7134b5b89612189/mongo_crypt_v1.dylib

Then we manually go and download the wrong version:

[2023/01/11 21:31:18.054] Testing CSFLE with crypt_shared lib version latest [2023/01/11 21:31:18.054] Download crypt_shared latest-enterprise for macos-x86_64 [2023/01/11 21:31:18.054] Extract from: [mongo_crypt_shared_v1-macos-x86_64-enterprise-6.2.0-rc4.tgz]

Is there any way to use the version downloaded by DET?

blink1073 · 2023-01-11T23:27:25Z

Lovely cleanup:

ShaneHarvey

LGTM nice!

juliusgeo added 2 commits January 10, 2023 17:10

initial commit

78a5b54

get rid of empty line

8508c63

juliusgeo requested review from blink1073 and ShaneHarvey as code owners January 11, 2023 01:17

blink1073 approved these changes Jan 11, 2023

View reviewed changes

ShaneHarvey reviewed Jan 11, 2023

View reviewed changes

ShaneHarvey requested changes Jan 11, 2023

View reviewed changes

juliusgeo requested a review from ShaneHarvey January 11, 2023 21:04

juliusgeo added 3 commits January 11, 2023 13:23

move the logic to run-tests.sh

4bc0b22

clean up white space

cc653e2

forgot about other variants

ec7c383

blink1073 reviewed Jan 11, 2023

View reviewed changes

make comment more clear

75cc59a

juliusgeo requested a review from blink1073 January 11, 2023 22:03

ShaneHarvey requested changes Jan 11, 2023

View reviewed changes

switch to use CRYPT_SHARED_PATH expansion

3596e81

blink1073 approved these changes Jan 11, 2023

View reviewed changes

juliusgeo added 2 commits January 11, 2023 15:43

didn't have the right name for expansion

88b09c9

use dir instead

02313f5

juliusgeo requested a review from blink1073 January 11, 2023 23:57

blink1073 approved these changes Jan 11, 2023

View reviewed changes

ShaneHarvey approved these changes Jan 12, 2023

View reviewed changes

juliusgeo merged commit a4c90ae into mongodb:master Jan 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PYTHON-3466 Test crypt_shared with older server versions #1133

PYTHON-3466 Test crypt_shared with older server versions #1133

Uh oh!

juliusgeo commented Jan 11, 2023

Uh oh!

blink1073 left a comment

Uh oh!

ShaneHarvey left a comment

Uh oh!

ShaneHarvey commented Jan 11, 2023

Uh oh!

ShaneHarvey Jan 11, 2023 •

edited

Loading

Uh oh!

juliusgeo Jan 11, 2023

Uh oh!

ShaneHarvey Jan 11, 2023

Uh oh!

juliusgeo Jan 11, 2023

Uh oh!

juliusgeo commented Jan 11, 2023 •

edited

Loading

Uh oh!

blink1073 Jan 11, 2023

Uh oh!

juliusgeo Jan 11, 2023

Uh oh!

ShaneHarvey Jan 11, 2023

Uh oh!

blink1073 commented Jan 11, 2023

Uh oh!

ShaneHarvey left a comment

Uh oh!

Uh oh!

	rules:
	- if:
	platform: "*"
	python-version-windows: "*"
	auth-ssl: "*"
	encryption: [ "encryption_crypt_shared" ]
	then:
	remove_tasks:
	- ".rapid"
	- ".5.0"
	- ".4.4"
	- ".4.2"
	- ".4.0"

PYTHON-3466 Test crypt_shared with older server versions #1133

PYTHON-3466 Test crypt_shared with older server versions #1133

Uh oh!

Conversation

juliusgeo commented Jan 11, 2023

Uh oh!

blink1073 left a comment

Choose a reason for hiding this comment

Uh oh!

ShaneHarvey left a comment

Choose a reason for hiding this comment

Uh oh!

ShaneHarvey commented Jan 11, 2023

Uh oh!

ShaneHarvey Jan 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusgeo Jan 11, 2023

Choose a reason for hiding this comment

Uh oh!

ShaneHarvey Jan 11, 2023

Choose a reason for hiding this comment

Uh oh!

juliusgeo Jan 11, 2023

Choose a reason for hiding this comment

Uh oh!

juliusgeo commented Jan 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blink1073 Jan 11, 2023

Choose a reason for hiding this comment

Uh oh!

juliusgeo Jan 11, 2023

Choose a reason for hiding this comment

Uh oh!

ShaneHarvey Jan 11, 2023

Choose a reason for hiding this comment

Uh oh!

blink1073 commented Jan 11, 2023

Uh oh!

ShaneHarvey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ShaneHarvey Jan 11, 2023 •

edited

Loading

juliusgeo commented Jan 11, 2023 •

edited

Loading