Skip to content

SWIFT-1328 Test against load balancers on Evergreen #668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 1, 2021
Merged

SWIFT-1328 Test against load balancers on Evergreen #668

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 66 additions & 2 deletions .evergreen/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,40 @@ functions:
LINUX_SOURCEKIT_LIB_PATH=${PROJECT_DIRECTORY}/opt/swiftenv/versions/${SWIFT_VERSION}/usr/lib \
${PROJECT_DIRECTORY}/opt/swiftlint/.build/release/swiftlint --strict --quiet

start-load-balancer:
- command: shell.exec
params:
script: |
DRIVERS_TOOLS=${DRIVERS_TOOLS} MONGODB_URI=${MONGODB_URI} \
bash ${DRIVERS_TOOLS}/.evergreen/run-load-balancer.sh start
- command: expansions.update
params:
file: lb-expansion.yml

stop-load-balancer:
- command: shell.exec
params:
script: |
DRIVERS_TOOLS=${DRIVERS_TOOLS} MONGODB_URI=${MONGODB_URI} \
bash ${DRIVERS_TOOLS}/.evergreen/run-load-balancer.sh stop

run-lb-tests:
- command: shell.exec
type: test
params:
working_dir: src
script: |
${PREPARE_SHELL}
MONGODB_URI="${MONGODB_URI}" \
TOPOLOGY="${TOPOLOGY}" \
SSL=${SSL} \
AUTH=${AUTH} \
SWIFT_VERSION=${SWIFT_VERSION} \
MONGODB_API_VERSION="${MONGODB_API_VERSION}" \
SINGLE_MONGOS_LB_URI="${SINGLE_MONGOS_LB_URI}" \
MULTI_MONGOS_LB_URI="${MULTI_MONGOS_LB_URI}" \
${PROJECT_DIRECTORY}/.evergreen/run-tests.sh

"sourcery":
- command: shell.exec
type: test
Expand Down Expand Up @@ -579,6 +613,21 @@ tasks:
MONGODB_VERSION: "latest"
TOPOLOGY: "sharded_cluster"
- func: "run tests"

- name: "test-latest-load_balancer"
tags: ["latest", "load-balancer"]
commands:
- func: "prepare resources"
- func: "fix absolute paths"
- func: "bootstrap mongo-orchestration"
vars:
MONGODB_VERSION: "latest"
Copy link
Contributor Author

@kmahar kmahar Aug 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

technically, we are supposed to test against "each server version that supports load balanced clusters".

this is kind of weird now because no servers actually support it, yet, with the serviceId mocking support they all appear to.

for now, I opted to just test against latest since that's the closest server version to actual support, but I suppose once 5.1 comes out with serviceId support (and we get a "test drivers against 5.1" ticket) we can start testing against both 5.1 and latest.

TOPOLOGY: "sharded_cluster"
- func: start-load-balancer
- func: "run-lb-tests"
vars:
TOPOLOGY: "load_balanced"
- func: stop-load-balancer

- name: "test-atlas-connectivity"
tags: ["atlas-connect"]
Expand Down Expand Up @@ -1136,7 +1185,7 @@ buildvariants:
ssl-auth: "*"
display_name: "${swift-version} ${os-fully-featured} ${ssl-auth}"
tasks:
- ".latest"
- ".latest !.load-balancer"
- ".5.0"
- ".4.4"
- ".4.2"
Expand Down Expand Up @@ -1165,7 +1214,7 @@ buildvariants:
ssl-auth: "*"
display_name: "${swift-version} ${os-fully-featured} ${ssl-auth}"
tasks:
- ".latest"
- ".latest !.load-balancer"
- ".5.0"
- ".4.4"
- ".4.2"
Expand Down Expand Up @@ -1220,6 +1269,21 @@ buildvariants:
tasks:
- ".atlas-connect"

- matrix_name: "load-balancer-all"
Copy link
Contributor Author

@kmahar kmahar Aug 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered doing what we have done for other matrices and having a separate matrix for testing Swift 5.1 (this is necessary since Ubuntu 20.04 doesn't support 5.1). however, I think this is sufficient coverage, and I also relatedly just filed SWIFT-1337 about dropping Swift 5.1 support anyway, per our recent convo / SwiftNIO's decision to do so

Copy link
Contributor

@patrickfreed patrickfreed Sep 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in general we could be a bit looser about what we test, too. The SSWG graduation criteria require that we test at least the two latest versions, and I think if we also include a test that we compile on our minimum supported version, we can be generally confident that the entire range of minimum -> maximum works as expected. In our case that would be compile on 5.1 and test on 5.3 and 5.4. This could help reduce the multiplicative factor of our evergreen config.

Copy link
Contributor Author

@kmahar kmahar Sep 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, that's a good point and I agree that just making sure everything builds on those versions seems sufficient, especially given we don't have any branching logic based on language version as far as I can think of.

I filed SWIFT-1340 about reducing the matrix size, seems like a good GBF candidate

matrix_spec:
os-fully-featured:
- "ubuntu-18.04"
Copy link
Contributor Author

@kmahar kmahar Aug 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haproxy isn't installed by default on macOS like it is on these hosts, and other drivers seem to have skipped running these tests on macOS, so I think just testing on Ubuntu should be sufficient.

- "ubuntu-20.04"
ssl-auth: "*"
swift-version:
- "5.2"
- "5.3"
- "5.4"
- "5.5-dev"
display_name: "Load Balancer ${swift-version} ${os-fully-featured} ${ssl-auth}"
tasks:
- ".load-balancer"

- matrix_name: "serverless"
matrix_spec:
os-fully-featured:
Expand Down
6 changes: 5 additions & 1 deletion .evergreen/run-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ set -o errexit # Exit the script with error if any of the commands fail
# variables
PROJECT_DIRECTORY=${PROJECT_DIRECTORY:-$PWD}
MONGODB_URI=${MONGODB_URI:-"NO_URI_PROVIDED"}
SINGLE_MONGOS_LB_URI=${SINGLE_MONGOS_LB_URI:-"NO_URI_PROVIDED"}
MULTI_MONGOS_LB_URI=${MULTI_MONGOS_LB_URI:-"NO_URI_PROVIDED"}
SWIFT_VERSION=${SWIFT_VERSION:-5.2.5}
INSTALL_DIR="${PROJECT_DIRECTORY}/opt"
TOPOLOGY=${TOPOLOGY:-single}
Expand Down Expand Up @@ -58,7 +60,9 @@ if [ "$TEST_FILTER" != "NO_FILTER" ]; then
FILTER_STATEMENT="--filter ${TEST_FILTER}"
fi

MONGODB_TOPOLOGY=${TOPOLOGY} MONGODB_URI=$MONGODB_URI MONGODB_API_VERSION=$MONGODB_API_VERSION swift test $FILTER_STATEMENT 2>&1 | tee ${RAW_TEST_RESULTS}
MONGODB_TOPOLOGY=${TOPOLOGY} MONGODB_URI=$MONGODB_URI SINGLE_MONGOS_LB_URI=$SINGLE_MONGOS_LB_URI \
MULTI_MONGOS_LB_URI=$MULTI_MONGOS_LB_URI MONGODB_API_VERSION=$MONGODB_API_VERSION \
swift test $FILTER_STATEMENT 2>&1 | tee ${RAW_TEST_RESULTS}

# save tests exit code
EXIT_CODE=$?
Expand Down
11 changes: 10 additions & 1 deletion Sources/TestsCommon/CommonTestUtils.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ open class MongoSwiftTestCase: XCTestCase {
}

public static var multipleMongosLoadBalancedURI: String? {
ProcessInfo.processInfo.environment["MULTIPLE_MONGOS_LB_URI"]
ProcessInfo.processInfo.environment["MULTI_MONGOS_LB_URI"]
Copy link
Contributor Author

@kmahar kmahar Aug 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is actually what the evergreen setup tools name the env variable, I had it wrong

}

/// Indicates that we are running the tests with SSL enabled, determined by the environment variable $SSL.
Expand Down Expand Up @@ -297,6 +297,15 @@ public struct TestRequirement: Decodable {
)
]

public static let changeStreamOnCollectionSupport = TestRequirement(
Copy link
Contributor Author

@kmahar kmahar Aug 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this and the various usages of it are not exactly related to the Evergreen work, however as I was looking through the logs to make sure tests ran correctly I noticed we were incorrectly skipping a bunch of tests which we had hardcoded the supported topologies for and I forgot to add .loadBalanced to so I updated those as well so they would actually run.

acceptableTopologies: [.replicaSet, .sharded, .shardedReplicaSet, .loadBalanced]
)

public static let changeStreamOnDBOrClientSupport = TestRequirement(
minServerVersion: ServerVersion(major: 4, minor: 0, patch: 0),
acceptableTopologies: [.replicaSet, .sharded, .shardedReplicaSet, .loadBalanced]
)

public init(
minServerVersion: ServerVersion? = nil,
maxServerVersion: ServerVersion? = nil,
Expand Down
96 changes: 20 additions & 76 deletions Tests/MongoSwiftSyncTests/SyncChangeStreamTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,10 +358,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
/// Prose test 1 of change stream spec.
/// "ChangeStream must continuously track the last seen resumeToken"
func testChangeStreamTracksResumeToken() throws {
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(testRequirements)
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -396,11 +393,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
* is < 8, this is a driver-side error; for 8+, this is a server-side error).
*/
func testChangeStreamMissingId() throws {
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded, .shardedReplicaSet]
)

let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(testRequirements)
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -436,7 +429,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
let testRequirements = TestRequirement(
// TODO: SWIFT-1257: remove server version requirement
maxServerVersion: ServerVersion(major: 4, minor: 9, patch: 0),
acceptableTopologies: [.replicaSet, .sharded]
acceptableTopologies: [.replicaSet, .sharded, .shardedReplicaSet, .loadBalanced]
)

let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(testRequirements)
Expand Down Expand Up @@ -506,11 +499,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
* ChangeStream will not attempt to resume on any error encountered while executing an aggregate command.
*/
func testChangeStreamFailedAggregate() throws {
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(testRequirements)
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -562,11 +551,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
* closed on the driver side.
*/
func testChangeStreamDoesntCloseOnEmptyBatch() throws {
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(testRequirements)
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -594,11 +579,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
* part of our evergreen matrix.
*/
func testChangeStreamFailedKillCursors() throws {
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(testRequirements)
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -655,11 +636,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
* - If resumeAfter was not specified, the getResumeToken result must be empty.
*/
func testChangeStreamResumeTokenUpdatesEmptyBatch() throws {
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(testRequirements)
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -700,11 +677,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
* - getResumeToken must return the _id of the previous document returned.
*/
func testChangeStreamResumeTokenUpdatesNonemptyBatch() throws {
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(testRequirements)
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -735,7 +708,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
let client = try MongoClient.makeTestClient()
let testRequirements = TestRequirement(
minServerVersion: ServerVersion(major: 4, minor: 0),
acceptableTopologies: [.replicaSet, .sharded]
acceptableTopologies: [.replicaSet, .sharded, .shardedReplicaSet, .loadBalanced]
)

let unmetRequirement = try client.getUnmetRequirement(testRequirements)
Expand Down Expand Up @@ -791,7 +764,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
let client = try MongoClient.makeTestClient()
let testRequirements = TestRequirement(
minServerVersion: ServerVersion(major: 4, minor: 0),
acceptableTopologies: [.replicaSet, .sharded]
acceptableTopologies: [.replicaSet, .sharded, .shardedReplicaSet, .loadBalanced]
)

let unmetRequirement = try client.getUnmetRequirement(testRequirements)
Expand Down Expand Up @@ -832,11 +805,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {

func testChangeStreamOnACollection() throws {
let client = try MongoClient.makeTestClient()
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try client.getUnmetRequirement(testRequirements)
let unmetRequirement = try client.getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -882,11 +851,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {

func testChangeStreamWithPipeline() throws {
let client = try MongoClient.makeTestClient()
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try client.getUnmetRequirement(testRequirements)
let unmetRequirement = try client.getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -915,11 +880,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {

func testChangeStreamResumeToken() throws {
let client = try MongoClient.makeTestClient()
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try client.getUnmetRequirement(testRequirements)
let unmetRequirement = try client.getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -994,11 +955,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {

func testChangeStreamWithEventType() throws {
let client = try MongoClient.makeTestClient()
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try client.getUnmetRequirement(testRequirements)
let unmetRequirement = try client.getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -1046,11 +1003,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
let expectedDoc1 = MyFullDocumentType(id: 1, x: 1, y: 2)

let client = try MongoClient.makeTestClient()
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try client.getUnmetRequirement(testRequirements)
let unmetRequirement = try client.getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -1095,11 +1048,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {

func testChangeStreamOnACollectionWithCodableType() throws {
let client = try MongoClient.makeTestClient()
let testRequirements = TestRequirement(
acceptableTopologies: [.replicaSet, .sharded]
)

let unmetRequirement = try client.getUnmetRequirement(testRequirements)
let unmetRequirement = try client.getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -1165,9 +1114,9 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
func testDecodingInvalidateEventsOnCollection() throws {
// invalidated change stream on a collection
try self.withTestNamespace { client, _, collection in
let unmetRequirement = try MongoClient.makeTestClient().getUnmetRequirement(
TestRequirement(acceptableTopologies: [.replicaSet, .sharded])
)
let unmetRequirement = try MongoClient
.makeTestClient()
.getUnmetRequirement(.changeStreamOnCollectionSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down Expand Up @@ -1202,12 +1151,7 @@ final class SyncChangeStreamTests: MongoSwiftTestCase {
// invalidated change stream on a DB
try self.withTestNamespace { client, db, collection in
// DB change streams are supported as of 4.0
let unmetRequirement = try client.getUnmetRequirement(
TestRequirement(
minServerVersion: ServerVersion(major: 4, minor: 0),
acceptableTopologies: [.replicaSet, .sharded]
)
)
let unmetRequirement = try client.getUnmetRequirement(.changeStreamOnDBOrClientSupport)
guard unmetRequirement == nil else {
printSkipMessage(testName: self.name, unmetRequirement: unmetRequirement!)
return
Expand Down
Loading