Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GODRIVER-2492 use consistent versions of crypt_shared and server #1021

Merged
merged 16 commits into from
Jul 20, 2022
Merged

GODRIVER-2492 use consistent versions of crypt_shared and server #1021

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
2810078
use fork of drivers-evergreen-tools
kevinAlbs Jul 13, 2022
c639acf
echo VERSION expansion in prepare-resources
kevinAlbs Jul 13, 2022
758ee2b
download crypt_shared using download-mongodb.sh
kevinAlbs Jul 13, 2022
a93f716
add empty expansion.yml if downloading crypt_shared succeeds
kevinAlbs Jul 13, 2022
532df50
use correct working dir for expansions.update
kevinAlbs Jul 13, 2022
9646582
remove unnecessary SKIP_CRYPT_SHARED_LIB_DOWNLOAD expansions
kevinAlbs Jul 13, 2022
ba9b5b0
default to "latest" for VERSION
kevinAlbs Jul 13, 2022
49eeaf4
disambiguate VERSION
kevinAlbs Jul 14, 2022
6eae9d9
Revert "use fork of drivers-evergreen-tools"
kevinAlbs Jul 14, 2022
4e445a6
TEMP: use fork of drivers-evergreen-tools
kevinAlbs Jul 15, 2022
9756bba
Merge branch 'master' into DRIVERS-2355
kevinAlbs Jul 15, 2022
5aed681
document where variables are defined
kevinAlbs Jul 19, 2022
c492eeb
make CRYPT_SHARED_LIB_PATH an evergreen expansion
kevinAlbs Jul 19, 2022
02b4f15
replace SKIP_CRYPT_SHARED_LIB_DOWNLOAD with SKIP_CRYPT_SHARED_LIB
kevinAlbs Jul 19, 2022
4340df2
Revert "TEMP: use fork of drivers-evergreen-tools"
kevinAlbs Jul 19, 2022
68dcdc3
add more verbose echo statements
kevinAlbs Jul 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 45 additions & 51 deletions .evergreen/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,26 +192,42 @@ functions:
script: |
${PREPARE_SHELL}

# Skip the crypt_shared library download if there is no crypt_shared build for the current
# platform. Don't try to determine this automatically to prevent misconfiguration errors
# from silently skipping the crypt_shared download and changing the testing conditions.
if [ "${SKIP_CRYPT_SHARED_LIB_DOWNLOAD}" = "true" ]; then
echo "There is no crypt_shared library for this platform, skipping download..."
exit 0
MONGODB_VERSION=${VERSION}
if [ -z "$MONGODB_VERSION" ]; then
# default to latest to match behavior of run-orchestration.sh.
MONGODB_VERSION=latest
fi

# Download the crypt_shared dynamic library for the current platform to the current working
# directory. The run-tests.sh and versioned API test scripts expect to find a
# mongo_crypt_v1.* file in the "src/go.mongodb.org/mongo-driver" working directory.
# TODO(GODRIVER-2437): Update version to "latest-stable" once the crypt_shared library has a
# feature-complete stable release.
${PYTHON3_BINARY} $DRIVERS_TOOLS/.evergreen/mongodl.py \
--component crypt_shared \
--version latest \
--edition enterprise \
--out . \
--only "**/mongo_crypt_v1.*" \
--strip-path-components 1
. $DRIVERS_TOOLS/.evergreen/download-mongodb.sh
get_distro
# get_distro defines $DISTRO.
get_mongodb_download_url_for "$DISTRO" "$MONGODB_VERSION"
# get_mongodb_download_url_for defines $MONGO_CRYPT_SHARED_DOWNLOAD_URL and $EXTRACT.
if [ -z "$MONGO_CRYPT_SHARED_DOWNLOAD_URL" ]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: Add a note about where $MONGO_CRYPT_SHARED_DOWNLOAD_URL and $EXTRACT come from. I had to do some searching to figure out that get_mongodb_download_url_for sets those env vars.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good thinking. Done.

echo "There is no crypt_shared library for distro='$DISTRO' and version='$MONGODB_VERSION'".
touch expansion.yml
else
echo "Downloading crypt_shared package from $MONGO_CRYPT_SHARED_DOWNLOAD_URL"
download_and_extract_crypt_shared "$MONGO_CRYPT_SHARED_DOWNLOAD_URL" "$EXTRACT"
CRYPT_SHARED_LIB_PATH="$(find $(pwd) -maxdepth 1 -type f \
-name 'mongo_crypt_v1.so' -o \
-name 'mongo_crypt_v1.dll' -o \
-name 'mongo_crypt_v1.dylib')"
# Expect that we always find a crypt_shared library file and set the CRYPT_SHARED_LIB_PATH
# environment variable. If we didn't, print an error message and exit.
if [ -z "$CRYPT_SHARED_LIB_PATH" ]; then
echo 'CRYPT_SHARED_LIB_PATH is empty. Exiting.'
exit 1
fi
# If we're on Windows, convert the "cygdrive" path to Windows-style paths.
if [ "Windows_NT" = "$OS" ]; then
CRYPT_SHARED_LIB_PATH=$(cygpath -m $CRYPT_SHARED_LIB_PATH)
fi
echo "CRYPT_SHARED_LIB_PATH: $CRYPT_SHARED_LIB_PATH" > expansion.yml
fi
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find with expansions.update! Should we move the find command from run-tests.sh here and add CRYPT_SHARED_LIB_PATH to the expansions instead of setting SKIP_CRYPT_SHARED_LIB_DOWNLOAD if we didn't download it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I have updated to move the find command here and set a CRYPT_SHARED_LIB_PATH expansion.

- command: expansions.update
params:
file: src/go.mongodb.org/mongo-driver/expansion.yml

install-linters:
- command: shell.exec
Expand Down Expand Up @@ -518,31 +534,14 @@ functions:
fi
. ${DRIVERS_TOOLS}/.evergreen/csfle/set-temp-creds.sh

# If the task doesn't have the SKIP_CRYPT_SHARED_LIB_DOWNLOAD variable set, try to find the
# crypt_shared library downloaded in the "prepare-resources" task and set the CRYPT_SHARED_LIB_PATH
# environment variable with a path to the file.
if [ "${SKIP_CRYPT_SHARED_LIB_DOWNLOAD}" != "true" ]; then
# Find the crypt_shared library file in the current directory and set the CRYPT_SHARED_LIB_PATH to
# the path of that file. Only look for .so, .dll, or .dylib files to prevent matching any other
# downloaded files.
export CRYPT_SHARED_LIB_PATH="$(find $(pwd) -maxdepth 1 -type f \
-name 'mongo_crypt_v1.so' -o \
-name 'mongo_crypt_v1.dll' -o \
-name 'mongo_crypt_v1.dylib')"

# Expect that we always find a crypt_shared library file and set the CRYPT_SHARED_LIB_PATH
# environment variable. If we didn't, print an error message and exit.
if [ -z "$CRYPT_SHARED_LIB_PATH" ]; then
echo 'SKIP_CRYPT_SHARED_LIB_DOWNLOAD is not "true", but CRYPT_SHARED_LIB_PATH is empty. Exiting.'
exit 1
fi

# If we're on Windows, convert the "cygdrive" path to Windows-style paths.
if [ "Windows_NT" = "$OS" ]; then
export CRYPT_SHARED_LIB_PATH=$(cygpath -m $CRYPT_SHARED_LIB_PATH)
fi

echo "CRYPT_SHARED_LIB_PATH=$CRYPT_SHARED_LIB_PATH"
if [ "${SKIP_CRYPT_SHARED_LIB}" = "true" ]; then
CRYPT_SHARED_LIB_PATH=""
echo "crypt_shared library is skipped"
elif [ -z "${CRYPT_SHARED_LIB_PATH}" ]; then
echo "crypt_shared library path is empty"
else
CRYPT_SHARED_LIB_PATH=${CRYPT_SHARED_LIB_PATH}
echo "crypt_shared library will be loaded from path: $CRYPT_SHARED_LIB_PATH"
fi

export GOFLAGS=-mod=vendor
Expand All @@ -564,6 +563,7 @@ functions:
GCP_EMAIL="${cse_gcp_email}" \
GCP_PRIVATE_KEY="${cse_gcp_private_key}" \
REQUIRE_API_VERSION="${REQUIRE_API_VERSION}" \
CRYPT_SHARED_LIB_PATH="$CRYPT_SHARED_LIB_PATH" \
make evg-test-versioned-api \
PKG_CONFIG_PATH=$PKG_CONFIG_PATH \
LD_LIBRARY_PATH=$LD_LIBRARY_PATH
Expand Down Expand Up @@ -1638,9 +1638,9 @@ tasks:
TOPOLOGY: "replica_set"
AUTH: "auth"
SSL: "ssl"
# Don't download the crypt_shared library, which should cause all of the tests to fall
# Don't use the crypt_shared library, which should cause all of the tests to fall
# back to using mongocryptd instead of crypt_shared.
SKIP_CRYPT_SHARED_LIB_DOWNLOAD: "true"
SKIP_CRYPT_SHARED_LIB: "true"

- name: test-replicaset-auth-nossl
tags: ["test", "replicaset", "authssl"]
Expand Down Expand Up @@ -2038,9 +2038,6 @@ axes:
# to prevent attempting to link the client-side encryption (libmongocrypt) binaries when
# running Go tests.
GO_BUILD_TAGS: ""
# There is no crypt_shared library download for Ubuntu 14.04. Skip downloading it and let the
# tests use mongocryptd instead.
SKIP_CRYPT_SHARED_LIB_DOWNLOAD: "true"

# OSes that require >= 3.2 for SSL
- id: os-ssl-32
Expand All @@ -2061,9 +2058,6 @@ axes:
variables:
GO_DIST: "/opt/golang/go1.17"
PYTHON3_BINARY: "/opt/python/3.8/bin/python3"
# There is no crypt_shared library download for Ubuntu 16.04. Skip downloading it and let the
# tests use mongocryptd instead.
SKIP_CRYPT_SHARED_LIB_DOWNLOAD: "true"
- id: "osx-go-1-17"
display_name: "MacOS 10.15"
run_on: macos-1015
Expand Down
38 changes: 11 additions & 27 deletions .evergreen/run-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,33 +69,6 @@ if [ -z ${GO_BUILD_TAGS+x} ]; then
GO_BUILD_TAGS="cse"
fi

# If the task doesn't have the SKIP_CRYPT_SHARED_LIB_DOWNLOAD variable set, try to find the
# crypt_shared library downloaded in the "prepare-resources" task and set the CRYPT_SHARED_LIB_PATH
# environment variable with a path to the file.
if [ "${SKIP_CRYPT_SHARED_LIB_DOWNLOAD}" != "true" ]; then
# Find the crypt_shared library file in the current directory and set the CRYPT_SHARED_LIB_PATH to
# the path of that file. Only look for .so, .dll, or .dylib files to prevent matching any other
# downloaded files.
export CRYPT_SHARED_LIB_PATH="$(find $(pwd) -maxdepth 1 -type f \
-name 'mongo_crypt_v1.so' -o \
-name 'mongo_crypt_v1.dll' -o \
-name 'mongo_crypt_v1.dylib')"

# Expect that we always find a crypt_shared library file and set the CRYPT_SHARED_LIB_PATH
# environment variable. If we didn't, print an error message and exit.
if [ -z "$CRYPT_SHARED_LIB_PATH" ]; then
echo 'SKIP_CRYPT_SHARED_LIB_DOWNLOAD is not "true", but CRYPT_SHARED_LIB_PATH is empty. Exiting.'
exit 1
fi

# If we're on Windows, convert the "cygdrive" path to Windows-style paths.
if [ "Windows_NT" = "$OS" ]; then
export CRYPT_SHARED_LIB_PATH=$(cygpath -m $CRYPT_SHARED_LIB_PATH)
fi

echo "CRYPT_SHARED_LIB_PATH=$CRYPT_SHARED_LIB_PATH"
fi

# Ensure mock KMS servers are running before starting tests.
await_server() {
for i in $(seq 300); do
Expand All @@ -113,6 +86,16 @@ await_server "KMS", 5698

echo "finished awaiting servers"

if [ "${SKIP_CRYPT_SHARED_LIB}" = "true" ]; then
CRYPT_SHARED_LIB_PATH=""
echo "crypt_shared library is skipped"
elif [ -z "${CRYPT_SHARED_LIB_PATH}" ]; then
echo "crypt_shared library path is empty"
else
CRYPT_SHARED_LIB_PATH=${CRYPT_SHARED_LIB_PATH}
echo "crypt_shared library will be loaded from path: $CRYPT_SHARED_LIB_PATH"
fi

AUTH=${AUTH} \
SSL=${SSL} \
MONGO_GO_DRIVER_CA_FILE=${MONGO_GO_DRIVER_CA_FILE} \
Expand All @@ -136,6 +119,7 @@ GCP_EMAIL="${cse_gcp_email}" \
GCP_PRIVATE_KEY="${cse_gcp_private_key}" \
CSFLE_TLS_CA_FILE="$DRIVERS_TOOLS/.evergreen/x509gen/ca.pem" \
CSFLE_TLS_CERTIFICATE_KEY_FILE="$DRIVERS_TOOLS/.evergreen/x509gen/client.pem" \
CRYPT_SHARED_LIB_PATH=$CRYPT_SHARED_LIB_PATH \
make evg-test \
PKG_CONFIG_PATH=$PKG_CONFIG_PATH \
LD_LIBRARY_PATH=$LD_LIBRARY_PATH