From aa63be8b91fb32b97cdbbb5aa413f1a8f65d3ad6 Mon Sep 17 00:00:00 2001 From: Jocelyn Mendez Date: Tue, 2 Nov 2021 11:21:28 -0400 Subject: [PATCH 1/5] DOCSP-16551 updates and deletes limitations on time-series collections --- source/core/timeseries-collections.txt | 2 ++ .../timeseries/timeseries-limitations.txt | 15 ++++++++++--- source/reference/command/findAndModify.txt | 2 ++ .../reference/method/db.collection.update.txt | 4 ++++ source/reference/method/js-collection.txt | 2 ++ source/release-notes/5.1.txt | 21 +++++++++++++------ 6 files changed, 37 insertions(+), 9 deletions(-) diff --git a/source/core/timeseries-collections.txt b/source/core/timeseries-collections.txt index 3f50a673e01..79a95f88e4d 100644 --- a/source/core/timeseries-collections.txt +++ b/source/core/timeseries-collections.txt @@ -79,6 +79,8 @@ command: } ) +.. _time-series-fields: + When creating a time series collection, specify the following options: .. list-table:: diff --git a/source/core/timeseries/timeseries-limitations.txt b/source/core/timeseries/timeseries-limitations.txt index 4e8c5186134..73c5a4182e1 100644 --- a/source/core/timeseries/timeseries-limitations.txt +++ b/source/core/timeseries/timeseries-limitations.txt @@ -24,9 +24,18 @@ The maximum size of a measurement document is 4 MB. Updates and Deletes ~~~~~~~~~~~~~~~~~~~ -:ref:`Time series collections ` only -support insert operations and read queries. Updates and manual delete operations -result in an error. +:ref:`Time series collections ` support update +and delete operations with limitations. The following operations are not supported +on time series collections: + +- :dbcommand:`findAndModify`. +- Updates that modify the ``timeField``. +- Updates with ``multi:false`` that modify the ``metaField``. +- Updates with ``upsert:true``. +- Updates or deletes in multi-document transactions. + +MongoDB supports all other update and delete operations. For more information, +see :ref:``. To automatically delete old data, :ref:`set up automatic removal (TTL) `. diff --git a/source/reference/command/findAndModify.txt b/source/reference/command/findAndModify.txt index aa35cb0daf4..9e1567a6bf9 100644 --- a/source/reference/command/findAndModify.txt +++ b/source/reference/command/findAndModify.txt @@ -1,3 +1,5 @@ +.. _find-and-modify: + ============= findAndModify ============= diff --git a/source/reference/method/db.collection.update.txt b/source/reference/method/db.collection.update.txt index dd735a0e027..adaae84749c 100644 --- a/source/reference/method/db.collection.update.txt +++ b/source/reference/method/db.collection.update.txt @@ -1,3 +1,5 @@ +.. _collection-update: + ====================== db.collection.update() ====================== @@ -51,6 +53,8 @@ The :method:`db.collection.update()` method has the following form: } ) +.. _update-parameters: + Parameters ~~~~~~~~~~ diff --git a/source/reference/method/js-collection.txt b/source/reference/method/js-collection.txt index 4e8b7ba8a48..a905a9bc2d5 100644 --- a/source/reference/method/js-collection.txt +++ b/source/reference/method/js-collection.txt @@ -1,3 +1,5 @@ +.. _collection-method: + ================== Collection Methods ================== diff --git a/source/release-notes/5.1.txt b/source/release-notes/5.1.txt index 6af9e164185..0a95655d87a 100644 --- a/source/release-notes/5.1.txt +++ b/source/release-notes/5.1.txt @@ -59,6 +59,21 @@ Starting in MongoDB 5.1, :expression:`$dateSubtract` and :expression:`$dateAdd` report an error when an overflow is detected for ``amount`` values. +Time Series Collections +----------------------- + +Geo Indexing for Time Series collections +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Starting in MongoDB 5.1, you can use :ref:`geo indexes +` on the ``metaField`` of time +series collections. + +Updates and Deletes +~~~~~~~~~~~~~~~~~~~ +Starting in MongoDB 5.1, time series collections support +:ref:`update and delete operations ` with limitations. + .. _5.1-rel-notes-sbe: Slot-Based Query Execution Engine @@ -135,12 +150,6 @@ Opcode Counters Resharding Statistics - :serverstatus:`shardingStatistics.resharding.lastOpEndingChunkImbalance` -Geo Indexing for Time Series collections -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Starting in MongoDB 5.1, you can use :ref:`geo indexes -` on the ``metaField`` of time -series collections. Schema Validation Errors Contain Description Field ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 6aa1b859200a64ec7acb835a0911a371bc93ed8c Mon Sep 17 00:00:00 2001 From: Jeff Allen Date: Fri, 5 Nov 2021 16:54:11 -0400 Subject: [PATCH 2/5] (DOCSP-18792): Sharding support for time series collections --- source/core/timeseries/timeseries-granularity.txt | 5 +++++ source/core/timeseries/timeseries-limitations.txt | 7 +++++-- .../includes/5.1/5.1-release-notes-sharded-time-series.rst | 3 +++ source/release-notes/5.1.txt | 5 +++++ 4 files changed, 18 insertions(+), 2 deletions(-) create mode 100644 source/includes/5.1/5.1-release-notes-sharded-time-series.rst diff --git a/source/core/timeseries/timeseries-granularity.txt b/source/core/timeseries/timeseries-granularity.txt index d577c0ba540..d8931978d72 100644 --- a/source/core/timeseries/timeseries-granularity.txt +++ b/source/core/timeseries/timeseries-granularity.txt @@ -143,3 +143,8 @@ a time. From ``"seconds"`` to ``"minutes"`` or from ``"minutes"`` to ``"hours"``. Other changes are not allowed. If you need to change the ``granularity`` from ``"seconds"`` to ``"hours"``, first increase the ``granularity`` to ``"minutes"`` and then to ``"hours"``. + +.. note:: + + You cannot modify the ``granularity`` of a sharded time series + collection. diff --git a/source/core/timeseries/timeseries-limitations.txt b/source/core/timeseries/timeseries-limitations.txt index 73c5a4182e1..81e1f96285c 100644 --- a/source/core/timeseries/timeseries-limitations.txt +++ b/source/core/timeseries/timeseries-limitations.txt @@ -126,11 +126,14 @@ Client-Side Field Level Encryption ` is not supported for :ref:`time series collections `. +.. _time-series-limitations-sharding: + Sharding ~~~~~~~~ -:ref:`Time series collections ` cannot -currently be sharded. +Sharded time series collections are supported starting in MongoDB 5.1. + +You cannot modify the ``granularity`` of a sharded time series collection. Aggregation $out and $merge ~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/includes/5.1/5.1-release-notes-sharded-time-series.rst b/source/includes/5.1/5.1-release-notes-sharded-time-series.rst new file mode 100644 index 00000000000..ebe24e702ba --- /dev/null +++ b/source/includes/5.1/5.1-release-notes-sharded-time-series.rst @@ -0,0 +1,3 @@ +MongoDB 5.1 provides support for sharded :ref:`time series collections +`. For more information, see +:ref:`Time Series Limitations `. diff --git a/source/release-notes/5.1.txt b/source/release-notes/5.1.txt index 0a95655d87a..d714ea302e5 100644 --- a/source/release-notes/5.1.txt +++ b/source/release-notes/5.1.txt @@ -149,6 +149,11 @@ Opcode Counters Resharding Statistics - :serverstatus:`shardingStatistics.resharding.lastOpEndingChunkImbalance` + +Sharded Time Series Collections +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. include:: /includes/5.1/5.1-release-notes-sharded-time-series.rst Schema Validation Errors Contain Description Field From 0778f0ca4e8f33f7ad412b01aa6dcdada96cac9a Mon Sep 17 00:00:00 2001 From: Jeff Allen Date: Mon, 8 Nov 2021 09:55:56 -0500 Subject: [PATCH 3/5] copy tweaks --- source/core/timeseries/timeseries-limitations.txt | 2 +- .../5.1/5.1-release-notes-sharded-time-series.rst | 4 ++-- source/release-notes/5.1.txt | 15 ++++++++------- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/source/core/timeseries/timeseries-limitations.txt b/source/core/timeseries/timeseries-limitations.txt index 81e1f96285c..8595e09908e 100644 --- a/source/core/timeseries/timeseries-limitations.txt +++ b/source/core/timeseries/timeseries-limitations.txt @@ -131,7 +131,7 @@ Client-Side Field Level Encryption Sharding ~~~~~~~~ -Sharded time series collections are supported starting in MongoDB 5.1. +Starting in MongoDB 5.1, sharded time series collections are supported. You cannot modify the ``granularity`` of a sharded time series collection. diff --git a/source/includes/5.1/5.1-release-notes-sharded-time-series.rst b/source/includes/5.1/5.1-release-notes-sharded-time-series.rst index ebe24e702ba..14fe277a3d2 100644 --- a/source/includes/5.1/5.1-release-notes-sharded-time-series.rst +++ b/source/includes/5.1/5.1-release-notes-sharded-time-series.rst @@ -1,3 +1,3 @@ MongoDB 5.1 provides support for sharded :ref:`time series collections -`. For more information, see -:ref:`Time Series Limitations `. +`. See :ref:`Time Series Limitations +`. diff --git a/source/release-notes/5.1.txt b/source/release-notes/5.1.txt index d714ea302e5..c59ae74f20b 100644 --- a/source/release-notes/5.1.txt +++ b/source/release-notes/5.1.txt @@ -71,8 +71,15 @@ series collections. Updates and Deletes ~~~~~~~~~~~~~~~~~~~ + Starting in MongoDB 5.1, time series collections support -:ref:`update and delete operations ` with limitations. +:ref:`update and delete operations ` with +limitations. + +Sharded Time Series Collections +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. include:: /includes/5.1/5.1-release-notes-sharded-time-series.rst .. _5.1-rel-notes-sbe: @@ -149,12 +156,6 @@ Opcode Counters Resharding Statistics - :serverstatus:`shardingStatistics.resharding.lastOpEndingChunkImbalance` - -Sharded Time Series Collections -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. include:: /includes/5.1/5.1-release-notes-sharded-time-series.rst - Schema Validation Errors Contain Description Field ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 9ed1574c7e6c28d47a2b2fc618c5fac6a3e6e057 Mon Sep 17 00:00:00 2001 From: Jeff Allen Date: Mon, 8 Nov 2021 15:18:47 -0500 Subject: [PATCH 4/5] updates per Arun's feedback --- source/core/timeseries-collections.txt | 31 +------- .../timeseries/timeseries-limitations.txt | 11 ++- .../5.1-release-notes-sharded-time-series.rst | 8 +- .../fact-granularity-description.rst | 23 ++++++ .../fact-meta-field-description.rst | 7 ++ .../fact-time-field-description.rst | 3 + source/reference/command/shardCollection.txt | 76 ++++++++++++++++++- source/release-notes/5.1.txt | 4 +- 8 files changed, 129 insertions(+), 34 deletions(-) create mode 100644 source/includes/time-series/fact-granularity-description.rst create mode 100644 source/includes/time-series/fact-meta-field-description.rst create mode 100644 source/includes/time-series/fact-time-field-description.rst diff --git a/source/core/timeseries-collections.txt b/source/core/timeseries-collections.txt index 79a95f88e4d..fece12536fd 100644 --- a/source/core/timeseries-collections.txt +++ b/source/core/timeseries-collections.txt @@ -97,44 +97,19 @@ When creating a time series collection, specify the following options: - string - - Required. The name of the field which contains the date in each - time series document. Documents in a time series collection must - have a valid BSON date as the value for the ``timeField``. + - .. include:: /includes/time-series/fact-time-field-description.rst * - ``timeseries.metaField`` - string - - Optional. The name of the field which contains metadata in each - time series document. The metadata in the specified field should - be data that is used to label a unique series of documents. The - metadata should rarely, if ever, change. - - The name of the specified field may not be ``_id`` or the same as - the ``timeseries.timeField``. The field can be of any type. + - .. include:: /includes/time-series/fact-meta-field-description.rst * - ``timeseries.granularity`` - string - - Optional. Possible values are ``"seconds"``, ``"minutes"``, and - ``"hours"``. By default, MongoDB sets the ``granularity`` to - ``"seconds"`` for high-frequency ingestion. - - Manually set the ``granularity`` parameter to improve performance - by optimizing how data in the time series collection is stored - internally. To select a value for ``granularity``, choose the - closest match to the time span between consecutive incoming - measurements. - - If you specify the ``timeseries.metaField``, consider the time - span between consecutive incoming measurements that have the same - unique value for the ``metaField`` field. Measurements often have - the same unique value for the ``metaField`` field if they come - from the same source. - - If you do not specify ``timeseries.metaField``, consider the time - span between all measurements that are inserted in the collection. + - .. include:: /includes/time-series/fact-granularity-description.rst * - ``expireAfterSeconds`` diff --git a/source/core/timeseries/timeseries-limitations.txt b/source/core/timeseries/timeseries-limitations.txt index 8595e09908e..ef9527952d1 100644 --- a/source/core/timeseries/timeseries-limitations.txt +++ b/source/core/timeseries/timeseries-limitations.txt @@ -21,6 +21,8 @@ Constraints The maximum size of a measurement document is 4 MB. +.. _timeseries-limitations-updates-deletes: + Updates and Deletes ~~~~~~~~~~~~~~~~~~~ @@ -132,8 +134,15 @@ Sharding ~~~~~~~~ Starting in MongoDB 5.1, sharded time series collections are supported. +When using sharded time series collections, you cannot: + +- Modify the ``granularity`` of a sharded time series + collection. + +- Run sharding administration commands, including: -You cannot modify the ``granularity`` of a sharded time series collection. + - :dbcommand:`moveChunk` + - :dbcommand:`splitChunk` Aggregation $out and $merge ~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/includes/5.1/5.1-release-notes-sharded-time-series.rst b/source/includes/5.1/5.1-release-notes-sharded-time-series.rst index 14fe277a3d2..b5ce5b70533 100644 --- a/source/includes/5.1/5.1-release-notes-sharded-time-series.rst +++ b/source/includes/5.1/5.1-release-notes-sharded-time-series.rst @@ -1,3 +1,7 @@ MongoDB 5.1 provides support for sharded :ref:`time series collections -`. See :ref:`Time Series Limitations -`. +`. + +See: + +- :dbcommand:`shardCollection` +- :ref:`Time Series Limitations ` diff --git a/source/includes/time-series/fact-granularity-description.rst b/source/includes/time-series/fact-granularity-description.rst new file mode 100644 index 00000000000..cf726c6a24b --- /dev/null +++ b/source/includes/time-series/fact-granularity-description.rst @@ -0,0 +1,23 @@ +Optional. Possible values are: + +- ``"seconds"`` +- ``"minutes"`` +- ``"hours"`` + +By default, MongoDB sets the ``granularity`` to ``"seconds"`` for +high-frequency ingestion. + +Manually set the ``granularity`` parameter to improve performance +by optimizing how data in the time series collection is stored +internally. To select a value for ``granularity``, choose the +closest match to the time span between consecutive incoming +measurements. + +If you specify the ``timeseries.metaField``, consider the time +span between consecutive incoming measurements that have the same +unique value for the ``metaField`` field. Measurements often have +the same unique value for the ``metaField`` field if they come +from the same source. + +If you do not specify ``timeseries.metaField``, consider the time +span between all measurements that are inserted in the collection. diff --git a/source/includes/time-series/fact-meta-field-description.rst b/source/includes/time-series/fact-meta-field-description.rst new file mode 100644 index 00000000000..8bd763988fb --- /dev/null +++ b/source/includes/time-series/fact-meta-field-description.rst @@ -0,0 +1,7 @@ +Optional. The name of the field which contains metadata in each +time series document. The metadata in the specified field should +be data that is used to label a unique series of documents. The +metadata should rarely, if ever, change. + +The name of the specified field may not be ``_id`` or the same as +the ``timeseries.timeField``. The field can be of any type. diff --git a/source/includes/time-series/fact-time-field-description.rst b/source/includes/time-series/fact-time-field-description.rst new file mode 100644 index 00000000000..81bf90b12c5 --- /dev/null +++ b/source/includes/time-series/fact-time-field-description.rst @@ -0,0 +1,3 @@ +Required. The name of the field which contains the date in each +time series document. Documents in a time series collection must +have a valid BSON date as the value for the ``timeField``. diff --git a/source/reference/command/shardCollection.txt b/source/reference/command/shardCollection.txt index b4c7489cd16..a9692a50a47 100644 --- a/source/reference/command/shardCollection.txt +++ b/source/reference/command/shardCollection.txt @@ -35,7 +35,8 @@ Definition unique: , numInitialChunks: , presplitHashedZones: , - collation: { locale: "simple" } + collation: { locale: "simple" }, + timeseries: } :dbcommand:`shardCollection` has the following fields: @@ -170,6 +171,64 @@ Definition .. versionadded:: 4.4 + * - :ref:`timeseries ` + + - object + + - .. _cmd-shard-collection-timeseries: + + Optional. Specify this option to create a new sharded + :ref:`time series collection `. + + To shard an existing time series collection, omit this + parameter. When you omit this parameter and specify a time + series collection in the ``shardCollection`` parameter, + MongoDB automatically uses the values from the time series + collection as the values for the ``timeseries`` field. + + For detailed syntax, see + :ref:`sharded-time-series-collection-options`. + + .. versionadded:: 5.1 + +.. _sharded-time-series-collection-options: + +Time Series Options +~~~~~~~~~~~~~~~~~~~ + +.. versionadded:: 5.1 + +Specify the :ref:`timeseries ` option +to :dbcommand:`shardCollection` to create a new sharded +:ref:`time series collection `. + +The :ref:`timeseries ` option takes +the following fields: + +.. list-table:: + :header-rows: 1 + :widths: 20 20 80 + + * - Field + - Type + - Description + + * - ``timeField`` + - string + - .. include:: /includes/time-series/fact-time-field-description.rst + + * - ``metaField`` + - string + - .. include:: /includes/time-series/fact-meta-field-description.rst + + * - ``granularity`` + - string + - .. include:: /includes/time-series/fact-granularity-description.rst + + * - ``bucketMaxSpanSeconds`` + - integer + - Optional. The maximum range of time values for a bucket, + in seconds. Considerations -------------- @@ -194,6 +253,20 @@ avoid scalability and perfomance issues. - :ref:`sharding-shard-key-selection` - :ref:`sharding-shard-key` +Shard Keys on Time Series Collections +````````````````````````````````````` + +When sharding time series collections, you can only specify the +``metaField``, ``timeField``, or both in the shard key. No other fields, +including ``_id``, are allowed in the shard key pattern. + +- ``metaField`` can be either a :ref:`hashed shard key + ` or a :ref:`ranged shard key + `. + +- ``timeField`` can only be a :ref:`ranged shard key + `. + .. _hashed-shard-keys: Hashed Shard Keys @@ -274,6 +347,7 @@ in the ``records`` database and uses the ``zipcode`` field as the .. code-block:: javascript db.adminCommand( { shardCollection: "records.people", key: { zipcode: 1 } } ) + .. seealso:: diff --git a/source/release-notes/5.1.txt b/source/release-notes/5.1.txt index c59ae74f20b..b4eadc2a613 100644 --- a/source/release-notes/5.1.txt +++ b/source/release-notes/5.1.txt @@ -73,8 +73,8 @@ Updates and Deletes ~~~~~~~~~~~~~~~~~~~ Starting in MongoDB 5.1, time series collections support -:ref:`update and delete operations ` with -limitations. +:ref:`update and delete operations +` with limitations. Sharded Time Series Collections ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From b8695ea3218aaf6c45f0a17c2bc4660e7f03cee9 Mon Sep 17 00:00:00 2001 From: Jeff Allen Date: Tue, 9 Nov 2021 17:31:32 -0500 Subject: [PATCH 5/5] Updates per review part 2 --- source/reference/command/shardCollection.txt | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/source/reference/command/shardCollection.txt b/source/reference/command/shardCollection.txt index a9692a50a47..d183a0cfd69 100644 --- a/source/reference/command/shardCollection.txt +++ b/source/reference/command/shardCollection.txt @@ -257,15 +257,16 @@ Shard Keys on Time Series Collections ````````````````````````````````````` When sharding time series collections, you can only specify the -``metaField``, ``timeField``, or both in the shard key. No other fields, -including ``_id``, are allowed in the shard key pattern. +``metaField`` (or sub-fields of ``metaField``), ``timeField``, or both +in the shard key. No other fields, including ``_id``, are allowed in the +shard key pattern. - ``metaField`` can be either a :ref:`hashed shard key ` or a :ref:`ranged shard key `. - ``timeField`` can only be a :ref:`ranged shard key - `. + ` and must be at the end of the shard key pattern. .. _hashed-shard-keys: