DRIVERS-1677, DRIVERS-1673: Add general logging specification and command logging

source/command-logging-and-monitoring/command-logging-and-monitoring.rst

@@ -5,8 +5,7 @@
 Command Logging and Monitoring
 ==============================
 
-:Title: Command Logging and Monitoring
-:Status: Approved
+:Status: Accepted
 :Minimum Server Version: 2.4
 
 .. contents::
@@ -149,10 +148,28 @@ the architecture needs of each driver.
 Security
 --------
 
-Some commands and replies will contain sensitive data and in order to not risk the leaking of this
-data to external sources or logs their commands AND replies AND corresponding error messages/exceptions 
-MUST be redacted from the events and log messages. The value MUST be replaced with an empty BSON document.
-The list is as follows:
+Some commands and replies will contain sensitive data relating to authentication.
+
+In order to not risk leaking this data to external sources or logs, for these commands:
+
+- The "command" field in ``CommandStartedEvent`` and "command started" log messages MUST 
+  be replaced with an empty BSON document.
+- The "reply" field in ``CommandSucceededEvent`` and "command succeeded" log messages MUST 
+  be replaced with an empty BSON document.
+- If the error is a server-side error, the "failure" field in ``CommandFailedEvent`` and
+  "command failed" log messages MUST have all fields besides the following redacted:
+
+  - ``code``
+  - ``codeName``
+  - ``errorLabels``
+
+  The exact implementation of redaction is flexible depending on the type the driver uses
+  to represent a failure in these events and log messages. For example, a driver could choose
+  to set all properties besides these on an error object to null. Alternatively, a driver
+  that uses strings to represent failures could replace relevant portions of the string with
+  "REDACTED".
+
+The list of sensitive commands is as follows:
 
 .. list-table::
    :header-rows: 1
@@ -513,52 +530,26 @@ See the README in the test directory for requirements and guidance.
 Changelog
 =========
 
-16 SEP 2015:
-  - Removed ``limit`` from find test with options to support 3.2.
-  - Changed find test read preference to ``primaryPreferred``.
-
-1 OCT 2015:
-  - Changed find test with a kill cursors to not run on server versions greater than 3.0
-  - Added a find test with no kill cursors command which only runs on 3.1 and higher.
-  - Added notes on which tests should run based on server versions.
-
-19 OCT 2015:
-  - Changed batchSize in the 3.2 find tests to expect the remaining value.
-
-31 OCT 2015:
-  - Changed find test on 3.1 and higher to ignore being run on sharded clusters.
-
-22 NOV 2015:
-  - Specify how to merge OP_MSG document sequences into command-started events.
-
-29 MAR 2016:
-  - Added note on guarantee of the request ids.
-
-2 NOV 2016:
-  - Added clause for not upconverting commands larger than maxBsonSize.
-
-16 APR 2018:
-  - Made inclusion of BSON serialization/deserialization in command durations
-    to be optional.
-
-12 FEB 2020:
-  - Added legacy hello ``speculativeAuthenticate`` to the list of values that should be redacted.
-
-15 APR 2021:
-  - Added ``serviceId`` field to events.
-
-5 MAY 2021:
-  - Updated to use hello and legacy hello.
-
-30 AUG 2021:
-  - Added ``serverConnectionId`` field to ``CommandStartedEvent``, ``CommandSucceededEvent`` and
-    ``CommandFailedEvent``.
-
-18 MAY 2022:
-  - Converted legacy tests to the unified test format.
-
-2 SEPTEMBER 2022:
-  - Remove material that only applies to MongoDB versions < 3.6.
-
-7 SEPTEMBER 2022:
-  - Add command logging information and tests.
+:2015-09-16: Removed ``limit`` from find test with options to support 3.2.
+             Changed find test read preference to ``primaryPreferred``.
+:2015-10-01: Changed find test with a kill cursors to not run on server versions
+             greater than 3.0. Added a find test with no kill cursors command
+             which only runs on 3.1 and higher. Added notes on which tests
+             should run based on server versions.
+:2015-10-19: Changed batchSize in the 3.2 find tests to expect the remaining value.
+:2015-10-31: Changed find test on 3.1 and higher to ignore being run on sharded clusters.
+:2015-11-22: Specify how to merge OP_MSG document sequences into command-started events.
+:2016-03-29: Added note on guarantee of the request ids.
+:2016-11-02: Added clause for not upconverting commands larger than maxBsonSize.
+:2018-04-16: Made inclusion of BSON serialization/deserialization in command
+             durations to be optional.
+:2020-02-12: Added legacy hello ``speculativeAuthenticate`` to the list of
+             values that should be redacted.
+:2021-04-15: Added ``serviceId`` field to events.
+:2021-05-05: Updated to use hello and legacy hello.
+:2021-08-30: Added ``serverConnectionId`` field to ``CommandStartedEvent``,
+             ``CommandSucceededEvent`` and ``CommandFailedEvent``.
+:2022-05-18: Converted legacy tests to the unified test format.
+:2022-09-02: Remove material that only applies to MongoDB versions < 3.6.
+:2022-10-05: Remove spec front matter and reformat changelog.
+:2022-10-11: Add command logging information and tests.

source/command-logging-and-monitoring/tests/README.rst

@@ -15,11 +15,14 @@ Testing
 Automated Tests
 ^^^^^^^^^^^^^^^
 There are tests in the `Unified Test Format <../../unified-test-format/unified-test-format.rst>`__ for both logging and
-monitoring in `/logging </logging>`_ and `/monitoring </monitoring>`_, respectively.
-
+monitoring in `/logging </logging>`_ and `/monitoring </monitoring>`_, respectively. Drivers MUST run the logging
+tests with their max document length setting (as described in the 
+`logging specification <../../logging/logging.rst#truncation-of-large-documents>`__) set to a large value e.g. 10,000;
+this is necessary in order for the driver to emit the full server reply (and to allow matching against that reply) on
+certain MongoDB versions and topologies.
 
 Prose Tests
-~~~~~~~~~~~
+^^^^^^^^^^^
 Drivers MUST implement the following logging prose tests. These tests require the ability to capture log message data in a
 structured form as described in the 
 `Unified Test Format specification <../../unified-test-format/unified-test-format.rst#expectedLogMessage>`__.
@@ -34,7 +37,7 @@ to support for specifying max document length as discussed in the
 2. Construct an array ``docs`` containing the document ``{"x" : "y"}`` repeated 100 times.
 3. Insert ``docs`` to a collection via ``insertMany``.
 4. Inspect the resulting "command started" log message and assert that the "command" value is a string of length 1000 + (length of trailing ellipsis).
-5. Inspect the resulting "command succeeded" log message and assert that the "reply" value is a string of length <= 1000.
+5. Inspect the resulting "command succeeded" log message and assert that the "reply" value is a string of length <= 1000 + (length of trailing ellipsis).
 6. Run ``find()`` on the collection where the document was inserted.
 7. Inspect the resulting "command succeeded" log message and assert that the reply is a string of length 1000 + (length of trailing ellipsis).
 

source/command-logging-and-monitoring/tests/logging/command.json

@@ -1,6 +1,6 @@
 {
   "description": "command-logging",
-  "schemaVersion": "1.11",
+  "schemaVersion": "1.12",
   "createEntities": [
     {
       "client": {

source/command-logging-and-monitoring/tests/logging/command.yml

@@ -1,6 +1,6 @@
 description: "command-logging"
 
-schemaVersion: "1.11"
+schemaVersion: "1.12"
 
 createEntities:
   - client:

source/command-logging-and-monitoring/tests/logging/driver-connection-id.json

@@ -1,6 +1,6 @@
 {
   "description": "driver-connection-id",
-  "schemaVersion": "1.11",
+  "schemaVersion": "1.12",
   "createEntities": [
     {
       "client": {

source/command-logging-and-monitoring/tests/logging/driver-connection-id.yml

@@ -1,7 +1,7 @@
 # This is a separate test so that drivers that do not implement CMAP can easily skip it.
 description: "driver-connection-id"
 
-schemaVersion: "1.11"
+schemaVersion: "1.12"
 
 createEntities:
   - client:

source/command-logging-and-monitoring/tests/logging/no-handshake-messages.json

@@ -1,6 +1,6 @@
 {
   "description": "no-handshake-command-logs",
-  "schemaVersion": "1.11",
+  "schemaVersion": "1.12",
   "tests": [
     {
       "description": "Handshake commands should not generate log messages",

source/command-logging-and-monitoring/tests/logging/no-handshake-messages.yml

@@ -1,6 +1,6 @@
 description: "no-handshake-command-logs"
 
-schemaVersion: "1.11"
+schemaVersion: "1.12"
 
 tests:
   - description: "Handshake commands should not generate log messages"

source/command-logging-and-monitoring/tests/logging/no-heartbeat-messages.json

@@ -1,6 +1,6 @@
 {
   "description": "no-heartbeat-command-logs",
-  "schemaVersion": "1.11",
+  "schemaVersion": "1.12",
   "runOnRequirements": [
     {
       "topologies": [

source/command-logging-and-monitoring/tests/logging/no-heartbeat-messages.yml

@@ -1,6 +1,6 @@
 description: "no-heartbeat-command-logs"
 
-schemaVersion: "1.11"
+schemaVersion: "1.12"
 
 # no heartbeats in load balanced mode.
 runOnRequirements:

source/command-logging-and-monitoring/tests/logging/operation-id.json

@@ -1,6 +1,6 @@
 {
   "description": "operation-id",
-  "schemaVersion": "1.11",
+  "schemaVersion": "1.12",
   "createEntities": [
     {
       "client": {

source/command-logging-and-monitoring/tests/logging/operation-id.yml

@@ -2,7 +2,7 @@
 # together bulk writes.
 description: "operation-id"
 
-schemaVersion: "1.11"
+schemaVersion: "1.12"
 
 createEntities:
   - client:

source/command-logging-and-monitoring/tests/logging/pre-42-server-connection-id.json

@@ -1,6 +1,6 @@
 {
   "description": "pre-42-server-connection-id",
-  "schemaVersion": "1.11",
+  "schemaVersion": "1.12",
   "runOnRequirements": [
     {
       "maxServerVersion": "4.0.99"

source/command-logging-and-monitoring/tests/logging/pre-42-server-connection-id.yml

@@ -1,6 +1,6 @@
 description: "pre-42-server-connection-id"
 
-schemaVersion: "1.11"
+schemaVersion: "1.12"
 
 runOnRequirements:
   - maxServerVersion: "4.0.99"