Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/reference/data-streams/downsampling.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ and it inherits its settings (for example, the number of shards and replicas).
supported.

* The downsampling configuration is extracted from the time series data stream
<<tsds-create-mappings-component-template,index mapping>>. The only additional
<<create-tsds-index-template,index mapping>>. The only additional
required setting is the downsampling `fixed_interval`.

[discrete]
Expand Down
134 changes: 36 additions & 98 deletions docs/reference/data-streams/set-up-tsds.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,6 @@ To set up a <<tsds,time series data stream (TSDS)>>, follow these steps:

. Check the <<tsds-prereqs,prerequisites>>.
. <<tsds-ilm-policy>>.
. <<tsds-create-mappings-component-template>>.
. <<tsds-create-index-settings-component-template>>.
. <<create-tsds-index-template>>.
. <<create-tsds>>.
. <<secure-tsds>>.
Expand Down Expand Up @@ -109,34 +107,52 @@ PUT _ilm/policy/my-weather-sensor-lifecycle-policy
----

[discrete]
[[tsds-create-mappings-component-template]]
==== Create a mappings component template
[[create-tsds-index-template]]
==== Create an index template

To setup a TSDS create an index template with the following details:

* One or more index patterns that match the TSDS's name. We recommend
using our {fleet-guide}/data-streams.html#data-streams-naming-scheme[data stream
naming scheme].

* Enable data streams.

* Specify a mapping that defines your dimensions and metrics:

A TSDS requires a matching index template. In most cases, you compose this index
template using one or more component templates. You typically use separate
component templates for mappings and index settings. This lets you reuse the
component templates in multiple index templates.
** One or more <<time-series-dimension,dimension fields>> with a `time_series_dimension` value of `true`.
At least one of these dimensions must be a plain `keyword` field.

For a TSDS, the mappings component template must include mappings for:
** One or more <<time-series-metric,metric fields>>, marked using the `time_series_metric` mapping parameter.

* One or more <<time-series-dimension,dimension fields>> with a
`time_series_dimension` value of `true`. At least one of these dimensions must
be a plain `keyword` field.
** Optional: A `date` or `date_nanos` mapping for the `@timestamp` field. If you don’t specify a mapping,
Elasticsearch maps `@timestamp` as a `date` field with default options.

Optionally, the template can also include mappings for:
* Define index settings:

* One or more <<time-series-metric,metric fields>>, marked using the
`time_series_metric` mapping parameter.
** Set `index.mode` setting to `time_series`.

* A `date` or `date_nanos` mapping for the `@timestamp` field. If you don’t
specify a mapping, Elasticsearch maps `@timestamp` as a `date` field with
default options.
** Your lifecycle policy in the `index.lifecycle.name` index setting.

** Optional: Other index settings, such as <<dynamic-index-number-of-replicas,`index.number_of_replicas`>>,
for your TSDS's backing indices.

* A priority higher than `200` to avoid collisions with built-in templates.
See <<avoid-index-pattern-collisions>>.

* Optional: Component templates containing your mappings and other index settings.

[source,console]
----
PUT _component_template/my-weather-sensor-mappings
PUT _index_template/my-weather-sensor-index-template
{
"index_patterns": ["metrics-weather_sensors-*"],
"data_stream": { },
"template": {
"settings": {
"index.mode": "time_series",
"index.lifecycle.name": "my-lifecycle-policy"
},
"mappings": {
"properties": {
"sensor_id": {
Expand All @@ -156,88 +172,11 @@ PUT _component_template/my-weather-sensor-mappings
"time_series_metric": "gauge"
},
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time"
"type": "date"
}
}
}
},
"_meta": {
"description": "Mappings for weather sensor data"
}
}
----
// TEST[continued]

[discrete]
[[tsds-create-index-settings-component-template]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 This id is being referenced in another file, so docs CI is failing

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx, updated the link.

==== Create an index settings component template

Optionally, the index settings component template for a TSDS can include:

* Your lifecycle policy in the `index.lifecycle.name` index setting.
* Other index settings, such as <<dynamic-index-number-of-replicas,`index.number_of_replicas`>>, for your TSDS's
backing indices.

IMPORTANT: Don't specify the `index.routing_path` index setting in a component
template. If you choose, you can configure `index.routing_path` directly in the
index template, as shown in the following step.

[source,console]
----
PUT _component_template/my-weather-sensor-settings
{
"template": {
"settings": {
"index.lifecycle.name": "my-lifecycle-policy"
}
},
"_meta": {
"description": "Index settings for weather sensor data"
}
}
----
// TEST[continued]

[discrete]
[[create-tsds-index-template]]
==== Create an index template

Use your component templates to create an index template. In the index template,
specify:

* One or more index patterns that match the TSDS's name. We recommend
using our {fleet-guide}/data-streams.html#data-streams-naming-scheme[data stream
naming scheme].

* That the template is data stream enabled.

* An `index.mode` object set to `time_series`.

* Optional: The `index.routing_path` index setting. The setting value should
only match plain `keyword` dimension fields and should be set directly in the
index template. When not defined explicitly, the `index.routing_path` setting is
generated from the mapping using all mappings that are set with
`time_series_dimension` set to `true`.

* The component templates containing your mappings and other index settings.

* A priority higher than `200` to avoid collisions with built-in templates.
See <<avoid-index-pattern-collisions>>.

[source,console]
----
PUT _index_template/my-weather-sensor-index-template
{
"index_patterns": ["metrics-weather_sensors-*"],
"data_stream": { },
"template": {
"settings": {
"index.mode": "time_series",
"index.routing_path": [ "sensor_id", "location" ]
}
},
"composed_of": [ "my-weather-sensor-mappings", "my-weather-sensor-settings" ],
"priority": 500,
"_meta": {
"description": "Template for my weather sensor data"
Expand All @@ -251,7 +190,6 @@ PUT _index_template/my-weather-sensor-index-template
----
DELETE _data_stream/*
DELETE _index_template/*
DELETE _component_template/my-*
DELETE _ilm/policy/my-weather-sensor-lifecycle-policy
----
// TEST[continued]
Expand Down
8 changes: 4 additions & 4 deletions docs/reference/data-streams/tsds.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -226,9 +226,9 @@ to not be actively written anymore in order to provide good performance. These a
- <<ilm-readonly>>
- <<ilm-searchable-snapshot>>
- <<ilm-shrink>>
{ilm-cap} will **not** proceed with executing these actions until the upper time-bound
for accepting writes, represented by the <<index-time-series-end-time,`index.time_series.end_time`>>
index setting, has lapsed.
{ilm-cap} will **not** proceed with executing these actions until the upper time-bound
for accepting writes, represented by the <<index-time-series-end-time,`index.time_series.end_time`>>
index setting, has lapsed.

If no backing index can accept a document's `@timestamp` value, {es} rejects the
document.
Expand Down Expand Up @@ -290,7 +290,7 @@ created the initial backing index has:

Only data that falls inside that range can be indexed.

In our <<tsds-create-index-settings-component-template,TSDS example>>,
In our <<create-tsds-index-template,TSDS example>>,
`index.look_ahead_time` is set to three hours, so only documents with a
`@timestamp` value that is within three hours previous or subsequent to the
present time are accepted for indexing.
Expand Down