Skip to content

Commit

Permalink
Sync to latest change stream tests
Browse files Browse the repository at this point in the history 
There is one behavioral change in these tests: to treat CursorNotFound errors as resumable.
The reason there is no corresponding change in the driver is because it was already doing this,
incorrectly it turns out.  But now it turns it that it is the correct thing to do.

JAVA-3781
JAVA-3776
  • Loading branch information
@jyemin
jyemin committed Jul 10, 2020
1 parent 7b9a753 commit 674a404
Show file tree
Hide file tree
Showing 4 changed files with 219 additions and 66 deletions.
102 changes: 84 additions & 18 deletions driver-core/src/test/resources/change-streams/README.rst
View file
Expand Up @@ -63,7 +63,7 @@ The definition of MATCH or MATCHES in the Spec Test Runner is as follows:

- MATCH takes two values, ``expected`` and ``actual``
- Notation is "Assert [actual] MATCHES [expected]
- Assertion passes if ``expected`` is a subset of ``actual``, with the values ``42`` and ``"42"`` acting as placeholders for "any value"
- Assertion passes if ``expected`` is a subset of ``actual``, with the value ``42`` acting as placeholders for "any value"

Pseudocode implementation of ``actual`` MATCHES ``expected``:

Expand Down Expand Up @@ -93,7 +93,10 @@ Spec Test Runner

Before running the tests

- Create a MongoClient ``globalClient``, and connect to the server
- Create a MongoClient ``globalClient``, and connect to the server.
When executing tests against a sharded cluster, ``globalClient`` must only connect to one mongos. This is because tests
that set failpoints will only work consistently if both the ``configureFailPoint`` and failing commands are sent to the
same mongos.

For each YAML file, for each element in ``tests``:

Expand All @@ -110,13 +113,10 @@ For each YAML file, for each element in ``tests``:

- Create a new MongoClient ``client``
- Begin monitoring all APM events for ``client``. (If the driver uses global listeners, filter out all events that do not originate with ``client``). Filter out any "internal" commands (e.g. ``isMaster``)
- Using ``client``, create a changeStream ``changeStream`` against the specified ``target``. Use ``changeStreamPipeline`` and ``changeStreamOptions`` if they are non-empty
- Using ``globalClient``, run every operation in ``operations`` in serial against the server
- Wait until either:

- An error occurs
- All operations have been successful AND the changeStream has received as many changes as there are in ``result.success``

- Using ``client``, create a changeStream ``changeStream`` against the specified ``target``. Use ``changeStreamPipeline`` and ``changeStreamOptions`` if they are non-empty. Capture any error.
- If there was no error, use ``globalClient`` and run every operation in ``operations`` in serial against the server until all operations have been executed or an error is thrown. Capture any error.
- If there was no error and ``result.error`` is set, iterate ``changeStream`` once and capture any error.
- If there was no error and ``result.success`` is non-empty, iterate ``changeStream`` until it returns as many changes as there are elements in the ``result.success`` array or an error is thrown. Capture any error.
- Close ``changeStream``
- If there was an error:

Expand All @@ -131,8 +131,8 @@ For each YAML file, for each element in ``tests``:
- If there are any ``expectations``

- For each (``expected``, ``idx``) in ``expectations``

- Assert that ``actual[idx]`` MATCHES ``expected``
- If ``actual[idx]`` is a ``killCursors`` event, skip it and move to ``actual[idx+1]``.
- Else assert that ``actual[idx]`` MATCHES ``expected``

- Close the MongoClient ``client``

Expand All @@ -142,20 +142,86 @@ After running all tests
- Drop database ``database_name``
- Drop database ``database2_name``

Iterating the Change Stream
---------------------------

Although synchronous drivers must provide a `non-blocking mode of iteration <../change-streams.rst#not-blocking-on-iteration>`_, asynchronous drivers may not have such a mechanism. Those drivers with only a blocking mode of iteration should be careful not to iterate the change stream unnecessarily, as doing so could cause the test runner to block indefinitely. For this reason, the test runner procedure above advises drivers to take a conservative approach to iteration.

If the test expects an error and one was not thrown by either creating the change stream or executing the test's operations, iterating the change stream once allows for an error to be thrown by a ``getMore`` command. If the test does not expect any error, the change stream should be iterated only until it returns as many result documents as are expected by the test.

Testing on Sharded Clusters
---------------------------

When writing data on sharded clusters, majority-committed data does not always show up in the response of the first
``getMore`` command after the data is written. This is because in sharded clusters, no data from shard A may be returned
until all other shard reports an entry that sorts after the change in shard A.

To account for this, drivers MUST NOT rely on change stream documents in certain batches. For example, if expecting two
documents in a change stream, these may not be part of the same ``getMore`` response, or even be produced in two
subsequent ``getMore`` responses. Drivers MUST allow for a ``getMore`` to produce empty batches when testing on a
sharded cluster. By default, this can take up to 10 seconds, but can be controlled by enabling the ``writePeriodicNoops``
server parameter and configuring the ``periodNoopIntervalSecs`` parameter. Choosing lower values allows for running
change stream tests with smaller timeouts.

Prose Tests
===========

The following tests have not yet been automated, but MUST still be tested
The following tests have not yet been automated, but MUST still be tested. All tests SHOULD be run on both replica sets and sharded clusters unless otherwise specified:

#. ``ChangeStream`` must continuously track the last seen ``resumeToken``
#. ``ChangeStream`` will throw an exception if the server response is missing the resume token
#. ``ChangeStream`` will automatically resume one time on a resumable error (including `not master`) with the initial pipeline and options, except for the addition/update of a ``resumeToken``.
#. ``ChangeStream`` will not attempt to resume on any error encountered while executing an ``aggregate`` command.
#. ``ChangeStream`` will not attempt to resume after encountering error code 11601 (Interrupted), 136 (CappedPositionLost), or 237 (CursorKilled) while executing a ``getMore`` command.
#. ``ChangeStream`` will throw an exception if the server response is missing the resume token (if wire version is < 8, this is a driver-side error; for 8+, this is a server-side error)
#. After receiving a ``resumeToken``, ``ChangeStream`` will automatically resume one time on a resumable error with the initial pipeline and options, except for the addition/update of a ``resumeToken``.
#. ``ChangeStream`` will not attempt to resume on any error encountered while executing an ``aggregate`` command. Note that retryable reads may retry ``aggregate`` commands. Drivers should be careful to distinguish retries from resume attempts. Alternatively, drivers may specify `retryReads=false` or avoid using a [retryable error](../../retryable-reads/retryable-reads.rst#retryable-error) for this test.
#. **Removed**
#. ``ChangeStream`` will perform server selection before attempting to resume, using initial ``readPreference``
#. Ensure that a cursor returned from an aggregate command with a cursor id and an initial empty batch is not closed on the driver side.
#. The ``killCursors`` command sent during the "Resume Process" must not be allowed to throw an exception.
#. ``$changeStream`` stage for ``ChangeStream`` against a server ``>=4.0`` that has not received any results yet MUST include a ``startAtOperationTime`` option when resuming a changestream.
#. ``ChangeStream`` will resume after a ``killCursors`` command is issued for its child cursor.
#. ``$changeStream`` stage for ``ChangeStream`` against a server ``>=4.0`` and ``<4.0.7`` that has not received any results yet MUST include a ``startAtOperationTime`` option when resuming a change stream.
#. **Removed**
#. For a ``ChangeStream`` under these conditions:

- Running against a server ``>=4.0.7``.
- The batch is empty or has been iterated to the last document.

Expected result:

- ``getResumeToken`` must return the ``postBatchResumeToken`` from the current command response.

#. For a ``ChangeStream`` under these conditions:

- Running against a server ``<4.0.7``.
- The batch is empty or has been iterated to the last document.

Expected result:

- ``getResumeToken`` must return the ``_id`` of the last document returned if one exists.
- ``getResumeToken`` must return ``resumeAfter`` from the initial aggregate if the option was specified.
- If ``resumeAfter`` was not specified, the ``getResumeToken`` result must be empty.

#. For a ``ChangeStream`` under these conditions:

- The batch is not empty.
- The batch has been iterated up to but not including the last element.

Expected result:

- ``getResumeToken`` must return the ``_id`` of the previous document returned.

#. For a ``ChangeStream`` under these conditions:

- The batch is not empty.
- The batch hasn’t been iterated at all.
- Only the initial ``aggregate`` command has been executed.

Expected result:

- ``getResumeToken`` must return ``startAfter`` from the initial aggregate if the option was specified.
- ``getResumeToken`` must return ``resumeAfter`` from the initial aggregate if the option was specified.
- If neither the ``startAfter`` nor ``resumeAfter`` options were specified, the ``getResumeToken`` result must be empty.

Note that this test cannot be run against sharded topologies because in that case the initial ``aggregate`` command only establishes cursors on the shards and always returns an empty ``firstBatch``.

#. **Removed**
#. **Removed**
#. ``$changeStream`` stage for ``ChangeStream`` started with ``startAfter`` against a server ``>=4.1.1`` that has not received any results yet MUST include a ``startAfter`` option and MUST NOT include a ``resumeAfter`` option when resuming a change stream.
#. ``$changeStream`` stage for ``ChangeStream`` started with ``startAfter`` against a server ``>=4.1.1`` that has received at least one result MUST include a ``resumeAfter`` option and MUST NOT include a ``startAfter`` option when resuming a change stream.
19 changes: 5 additions & 14 deletions driver-core/src/test/resources/change-streams/change-streams-errors.json
View file
Expand Up @@ -102,15 +102,12 @@
],
"result": {
"error": {
"code": 280,
"errorLabels": [
"NonResumableChangeStreamError"
]
"code": 280
}
}
},
{
"description": "change stream errors on MaxTimeMSExpired",
"description": "change stream errors on ElectionInProgress",
"minServerVersion": "4.2",
"failPoint": {
"configureFailPoint": "failCommand",
Expand All @@ -121,7 +118,7 @@
"failCommands": [
"getMore"
],
"errorCode": 50,
"errorCode": 216,
"closeConnection": false
}
},
Expand All @@ -130,13 +127,7 @@
"replicaset",
"sharded"
],
"changeStreamPipeline": [
{
"$project": {
"_id": 0
}
}
],
"changeStreamPipeline": [],
"changeStreamOptions": {},
"operations": [
{
Expand All @@ -152,7 +143,7 @@
],
"result": {
"error": {
"code": 50
"code": 216
}
}
}
Expand Down
34 changes: 17 additions & 17 deletions driver-core/src/test/resources/change-streams/change-streams-resume-errorLabels.json
View file
Expand Up @@ -53,7 +53,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -146,7 +146,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -239,7 +239,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -332,7 +332,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -425,7 +425,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -518,7 +518,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -611,7 +611,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -704,7 +704,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -797,7 +797,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -890,7 +890,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -983,7 +983,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -1076,7 +1076,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -1169,7 +1169,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -1262,7 +1262,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -1355,7 +1355,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -1448,7 +1448,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down Expand Up @@ -1547,7 +1547,7 @@
{
"command_started_event": {
"command": {
"getMore": "42",
"getMore": 42,
"collection": "test"
},
"command_name": "getMore",
Expand Down

0 comments on commit 674a404

Please sign in to comment.