diff --git a/solutions/images/logs-streams-patterns.png b/solutions/images/logs-streams-patterns.png index 0dfd280b50..aa1df44582 100644 Binary files a/solutions/images/logs-streams-patterns.png and b/solutions/images/logs-streams-patterns.png differ diff --git a/solutions/observability/streams/management/advanced.md b/solutions/observability/streams/management/advanced.md index 4baed472b4..0482121143 100644 --- a/solutions/observability/streams/management/advanced.md +++ b/solutions/observability/streams/management/advanced.md @@ -6,6 +6,4 @@ navigation_title: Configure advanced settings --- # Configure advanced settings for streams [streams-advanced-settings] -The **Advanced** tab on the **Manage stream** page shows the underlying configuration details of your stream. While Streams simplifies many configurations, it doesn't support modifying all pipelines and templates. From the **Advanced** tab, you can manually interact with the index or component templates or modify other ingest pipelines that used by the stream. - -This UI is intended for advanced users. \ No newline at end of file +The **Advanced** tab shows the underlying {{es}} configuration details of your stream. While Streams simplifies many configurations, it doesn't support modifying all pipelines and templates. From the **Advanced** tab, you can manually interact with the index or component templates or modify other ingest pipelines that used by the stream. \ No newline at end of file diff --git a/solutions/observability/streams/management/data-quality.md b/solutions/observability/streams/management/data-quality.md new file mode 100644 index 0000000000..c761dc9b3c --- /dev/null +++ b/solutions/observability/streams/management/data-quality.md @@ -0,0 +1,15 @@ +--- +applies_to: + serverless: preview + stack: preview 9.1, ga 9.2 +--- + +# Manage data quality [streams-data-retention] + +Use the **Data quality** tab to find failed and degraded documents in your stream. The **Data quality** tab is made up of the following components: + +- **Degraded documents**: Documents with the `ignored` property usually because of malformed fields or exceeding the limit of total fields when `ignore_above:false`. This component shows the total number of degraded documents, the percentage, and status (**Good**, **Degraded**, **Poor**). +- **Failed documents**: Documents that were rejected during ingestion. +- **Issues**: {applies_to}`stack: preview 9.2`Find issues with specific fields, how often they've occurred, and when they've occurred. + +For more information on data quality, refer to the [data set quality](../../data-set-quality-monitoring.md) documentation. \ No newline at end of file diff --git a/solutions/observability/streams/management/extract.md b/solutions/observability/streams/management/extract.md index d1b4dcfdbe..8400ac8573 100644 --- a/solutions/observability/streams/management/extract.md +++ b/solutions/observability/streams/management/extract.md @@ -5,32 +5,42 @@ applies_to: --- # Extract fields [streams-extract-fields] -Unstructured log messages must be parsed into meaningful fields before you can filter and analyze them effectively. Commonly extracted fields include `@timestamp` and the `log.level`, but you can also extract information like IP addresses, usernames, and ports. +Extracting meaningful fields from your log messages lets you filter and analyze them effectively. For example, you might want to use [Discover](../../../../explore-analyze/discover.md) to filter for log messages with a `WARNING` or `ERROR` log level that occurred during a certain time period to diagnose an issue. If you haven't extracted log level and timestamp fields from your messages, you won't get meaningful results. -Use the **Processing** tab on the **Manage stream** page to process your data. The UI simulates your changes and provides an immediate preview that's tested end-to-end. +From the **Processing** tab, you can add the [processors](#streams-extract-processors) you need to extract these structured fields. The UI then simulates your changes and provides an immediate [preview](#streams-preview-changes) that's tested end-to-end. -The UI also shows indexing problems, such as mapping conflicts, so you can address them before applying changes. +Streams also shows when you have indexing problems, such as [mapping conflicts](#streams-processing-mapping-conflicts), so you can address them before applying changes. + +After creating your processor, all future ingested data will be parsed into structured fields accordingly. :::{note} Applied changes aren't retroactive and only affect *future ingested data*. ::: +## Supported processors [streams-extract-processors] + +Streams supports the following processors: + +- [Date](./extract/date.md): convert date strings into timestamps with options for timezone, locale, and output format settings. +- [Dissect](./extract/dissect.md): extract fields from structured log messages using defined delimiters instead of patterns, making it faster than Grok and ideal for consistently formatted logs. +- [Grok](./extract/grok.md): extract fields from unstructured log messages using predefined or custom patterns, supports multiple match attempts in sequence, and can automatically generate patterns with an LLM connector. +- [Set](./extract/set.md): assign a specific value to a field, creating the field if it doesn’t exist or overwriting its value if it does. +- [Rename](./extract/rename.md): change the name of a field, moving its value to a new field name and removing the original. +- [Append](./extract/append.md): add a value to an existing array field, or create the field as an array if it doesn’t exist. + ## Add a processor [streams-add-processors] -Streams uses {{es}} ingest pipelines to process your data. Ingest pipelines are made up of processors that transform your data. +Streams uses [{{es}} ingest pipelines](../../../../manage-data/ingest/transform-enrich/ingest-pipelines.md) made up of processors to transform your data, without requiring you to switch interfaces and manually update pipelines. -To add a processor: +To add a processor from the **Processing** tab: -1. Select **Add processor** to open a list of supported processors. -1. Select a processor from the list: - - [Date](./extract/date.md) - - [Dissect](./extract/dissect.md) - - [Grok](./extract/grok.md) - - GeoIP - - Rename - - Set - - URL Decode -1. Select **Add Processor** to save the processor. +1. Select **Create** → **Create processor** to open a list of supported processors. +1. Select a processor from the **Processor** menu. +1. Configure the processor and select **Create** to save the processor. + +After adding all desired processors and conditions, make sure to **Save changes**. + +Refer to individual [supported processors](#streams-extract-processors) for more on configuring specific processors. :::{note} Editing processors with JSON is planned for a future release, and additional processors may be supported over time. @@ -38,8 +48,18 @@ Editing processors with JSON is planned for a future release, and additional pro ### Add conditions to processors [streams-add-processor-conditions] -You can provide a condition for each processor under **Optional fields**. Conditions are boolean expressions that are evaluated for each document. Provide a field, a value, and a comparator. -Processors support these comparators: +You can provide a condition for each processor under **Optional fields**. Conditions are boolean expressions that are evaluated for each document. + +To add a condition: +1. Select **Create** → **Create condition**. +1. Provide a **Field**, a **Value**, and a comparator. Expand the following dropdown for supported comparators. +1. Select **Create condition**. + +After adding all desired processors and conditions, make sure to **Save changes**. + +:::{dropdown} Supported comparators +Streams processors support the following comparators: + - equals - not equals - less than @@ -51,10 +71,11 @@ Processors support these comparators: - ends with - exists - not exists +::: ### Preview changes [streams-preview-changes] -Under **Processors for field extraction**, when you set pipeline processors to modify your documents, **Data preview** shows you a preview of the results with additional filtering options depending on the outcome of the simulation. +After creating processors, the **Data preview** tab shows a preview of the results with additional filtering options depending on the outcome of the simulation. When you add or edit processors, the **Data preview** updates automatically. @@ -63,27 +84,26 @@ To avoid unexpected results, we recommend adding processors rather than removing ::: **Data preview** loads 100 documents from your existing data and runs your changes using them. -For any newly added processors, this simulation is reliable. You can save individual processors during the preview, and even reorder them. -Selecting **Save changes** applies your changes to the data stream. +For any newly created processors and conditions, the preview is reliable. You can create and reorder individual processors and conditions during the preview. -If you edit the stream again, note the following: -- Adding more processors to the end of the list will work as expected. -- Changing existing processors or re-ordering them may cause unexpected results. Because the pipeline may have already processed the documents used for sampling, the UI cannot accurately simulate changes to existing data. -- Adding a new processor and moving it before an existing processor may cause unexpected results. The UI only simulates the new processor, not the existing ones, so the simulation may not accurately reflect changes to existing data. +Select **Save changes** to apply your changes to the data stream. -![Screenshot of the Grok processor UI](<../../../images/logs-streams-grok.png>) +If you edit the stream after saving your changes, note the following: +- Adding more processors to the end of the list will work as expected. +- Editing or reordering existing processors may cause unexpected results. Because the pipeline may have already processed the documents used for sampling, **Data preview** cannot accurately simulate changes to existing data. +- Adding a new processor and moving it before an existing processor may cause unexpected results. **Data preview** only simulates the new processor, not the existing ones, so the simulation may not accurately reflect changes to existing data. ### Ignore failures [streams-ignore-failures] -Turn on **Ignore failure** to ignore the processor if it fails. This is useful if you want to continue processing the document even if the processor fails. +Each processor has the option to **Ignore failures**. When enabled, processing of the document continues when the processor fails. ### Ignore missing fields [streams-ignore-missing-fields] -Turn on **Ignore missing fields** to ignore the processor if the field is not present. This is useful if you want to continue processing the document even if the field is not present. +Dissect, grok, and rename processors include the **Ignore missing fields** option. When enabled, processing of the document continues when a source field is missing. ## Detect and handle failures [streams-detect-failures] -Documents fail processing for different reasons. Streams helps you to easily find and handle failures before deploying changes. +Documents fail processing for different reasons. Streams helps you to find and handle failures before deploying changes. In the following screenshot, the **Failed** percentage shows that not all messages matched the provided Grok pattern: @@ -97,11 +117,11 @@ Failures are displayed at the bottom of the process editor: ![Screenshot showing failure notifications](<../../../images/logs-streams-processor-failures.png>) -These failures may require action, but in some cases, they serve more as warnings. +These failures may require action, or serve as a warning. -### Mapping conflicts +### Mapping conflicts [streams-processing-mapping-conflicts] -As part of processing, Streams also checks for mapping conflicts by simulating the change end to end. If a mapping conflict is detected, Streams marks the processor as failed and displays a failure message like the following: +As part of processing, Streams also checks for mapping conflicts by simulating the change end-to-end. When Streams detects a mapping conflict, it marks the processor as failed and displays a failure message like the following: ![Screenshot showing mapping conflict notifications](<../../../images/logs-streams-mapping-conflicts.png>) @@ -113,12 +133,14 @@ Once saved, the processor provides a quick look at the processor's success rate ![Screenshot showing field stats](<../../../images/logs-streams-field-stats.png>) -## Advanced: How and where do these changes get applied to the underlying datastream? [streams-applied-changes] +## Advanced: How and where do these changes get applied to the underlying data stream? [streams-applied-changes] + +% make sure this is all still accurate. -When you save processors, Streams modifies the "best matching" ingest pipeline for the data stream. In short, Streams either chooses the best matching pipeline ending in `@custom` that is already part of your data stream, or it adds one for you. +When you save processors, Streams modifies the "best-matching" ingest pipeline for the data stream. In short, Streams either chooses the best-matching pipeline ending in `@custom` that is already part of your data stream, or it adds one for you. Streams identifies the appropriate @custom pipeline (for example, `logs-myintegration@custom` or `logs@custom`). -It checks the default_pipeline that is set on the datastream. +It checks the `default_pipeline` that is set on the data stream. You can view the default pipeline at **Manage stream** → **Advanced** under **Ingest pipeline**. In this default pipeline, we locate the last processor that calls a pipeline ending in `@custom`. For integrations, this would result in a pipeline name like `logs-myintegration@custom`. Without an integration, the only `@custom` pipeline available may be `logs@custom`. @@ -150,7 +172,4 @@ You can still add your own processors manually to the `@custom` pipeline if need ## Known limitations [streams-known-limitations] - Streams does not support all processors. We are working on adding more processors in the future. -- Streams does not support all processor options. We are working on adding more options in the future. -- The data preview simulation may not accurately reflect the changes to the existing data when editing existing processors or re-ordering them. -- Dots in field names are not supported. You can use the dot expand processor in the `@custom` pipeline as a workaround. You need to manually add the dot expand processor. -- Providing any arbitrary JSON in the Streams UI is not supported. We are working on adding this in the future. +- The data preview simulation may not accurately reflect the changes to the existing data when editing existing processors or re-ordering them. We will allow proper simulations using original documents in a future version. \ No newline at end of file diff --git a/solutions/observability/streams/management/extract/append.md b/solutions/observability/streams/management/extract/append.md new file mode 100644 index 0000000000..bd8c899559 --- /dev/null +++ b/solutions/observability/streams/management/extract/append.md @@ -0,0 +1,16 @@ +--- +applies_to: + serverless: ga + stack: preview 9.1, ga 9.2 +--- +# Append processor [streams-append-processor] +% Need use cases + +Use the append processor to add a value to an existing array field, or create the field as an array if it doesn’t exist. + +To use an append processor: + +1. Set **Source Field** to the field you want append values to. +1. Set **Target field** to the values you want to append to the **Source Field**. + +This functionality uses the {{es}} rename pipeline processor. Refer to the [rename processor](elasticsearch://reference/enrich-processor/rename-processor.md) {{es}} documentation for more information. \ No newline at end of file diff --git a/solutions/observability/streams/management/extract/date.md b/solutions/observability/streams/management/extract/date.md index 2d79d22fa3..2e135592e2 100644 --- a/solutions/observability/streams/management/extract/date.md +++ b/solutions/observability/streams/management/extract/date.md @@ -6,11 +6,16 @@ applies_to: # Date processor [streams-date-processor] -The date processor parses date strings and uses them as the timestamp of the document. +The date processor parses dates from fields, and then uses the date or timestamp as the timestamp for the document. + +To extract a timestamp field using the date processor: + +1. Set the **Source Field** to the field containing the timestamp. +1. Set the **Format** field to one of the accepted date formats (ISO8602, UNIX, UNIX_MS, or TAI64N) or use a Java time pattern. Refer to the [example formats](#streams-date-examples) for more information. This functionality uses the {{es}} date pipeline processor. Refer to the [date processor](elasticsearch://reference/enrich-processor/date-processor.md) {{es}} documentation for more information. -## Examples +## Example formats [streams-date-examples] The following list provides some common examples of date formats and how to parse them. @@ -34,9 +39,8 @@ Sunday, October 15, 2023 => EEEE, MMMM dd, yyyy 2023-10-15 14:30:00 => yyyy-MM-dd HH:mm:ss ``` - ## Optional fields [streams-date-optional-fields] -The following fields are optional for the date processor: +You can set the following optional fields for the date processor in the **Advanced settings**: | Field | Description| | ------- | --------------- | diff --git a/solutions/observability/streams/management/extract/dissect.md b/solutions/observability/streams/management/extract/dissect.md index 72a08ad0a2..c9f3527b00 100644 --- a/solutions/observability/streams/management/extract/dissect.md +++ b/solutions/observability/streams/management/extract/dissect.md @@ -5,20 +5,26 @@ applies_to: --- # Dissect processor [streams-dissect-processor] -The dissect processor parses structured log messages and extracts fields from them. Unlike Grok, it does not use a set of predefined patterns to match the log messages. Instead, it uses a set of delimiters to split the log message into fields. -Dissect is much faster than Grok and is ideal for log messages that follow a consistent, structured format. +The dissect processor parses structured log messages and extracts fields from them. It uses a set of delimiters to split the log message into fields instead of predefined patterns to match the log messages. + +Dissect is much faster than Grok, and is recommend for log messages that follow a consistent, structured format. + +To parse a log message with a dissect processor: +1. Set the **Source Field** to the field you want to dissect +1. Set the delimiters you want to use in the **Pattern** field. Refer to the [example pattern](#streams-dissect-example) for more information on setting delimiters. This functionality uses the {{es}} dissect pipeline processor. Refer to the [dissect processor](elasticsearch://reference/enrich-processor/dissect-processor.md) {{es}} documentation for more information. -To parse a log message, simply name the field and list the delimiters you want to use. The dissect processor will then split the log message into fields based on the delimiters provided. +## Example dissect pattern [streams-dissect-example] -Example: +The following example shows the dissect pattern for an unstructured log message. -Log Message +**Log message:** ``` 2025-04-04T09:04:45+00:00 ERROR 160.200.87.105 127.79.135.127 21582 ``` -Dissect Pattern + +**Dissect Pattern:** ``` %{timestamp} %{log.level} %{source.ip} %{destination.ip} %{destination.port} ``` \ No newline at end of file diff --git a/solutions/observability/streams/management/extract/grok.md b/solutions/observability/streams/management/extract/grok.md index 27fd6d30ca..29fa44a8e0 100644 --- a/solutions/observability/streams/management/extract/grok.md +++ b/solutions/observability/streams/management/extract/grok.md @@ -5,18 +5,26 @@ applies_to: --- # Grok processor [streams-grok-processor] -The Grok processor parses unstructured log messages and extracts fields from them. It uses a set of predefined patterns to match the log messages and extract the fields. The Grok processor is very powerful and can parse a wide variety of log formats. +The grok processor parses unstructured log messages using a set of predefined patterns to match the log messages and extract the fields. The Grok processor is very powerful and can parse a wide variety of log formats. -You can provide multiple patterns to the Grok processor. The Grok processor will try to match the log message against each pattern in the order they are provided. If a pattern matches, the fields will be extracted and the remaining patterns will not be used. -If a pattern does not match, the Grok processor will try the next pattern. If no patterns match, the Grok processor will fail and you can troubleshoot the issue. Refer to [generate patterns](#streams-grok-patterns) for more information. +You can provide multiple patterns to the grok processor. The Grok processor will try to match the log message against each pattern in the order they are provided. If a pattern matches, the fields will be extracted and the remaining patterns will not be used. -Start with the most common patterns first and then add more specific patterns later. This reduces the number of runs the Grok processor has to do and improves the performance of the pipeline. +If a pattern doesn't match, the grok processor will try the next pattern. If no patterns match, the Grok processor will fail and you can troubleshoot the issue. Instead of writing grok patterns, you can have streams generate patterns for you. Refer to [generate patterns](#streams-grok-patterns) for more information. + +:::{tip} +To improve pipeline performance, start with the most common patterns first, then add more specific patterns. This reduces the number times the grok processor has to run. +::: + +To parse a log message with a dissect processor: + +1. Set the **Source Field** to the field you want to search for grok matches. +1. Set the patterns you want to use in the **Grok patterns** field. Refer to the [example pattern](#streams-grok-example) for more information on patterns. This functionality uses the {{es}} Grok pipeline processor. Refer to the [Grok processor](elasticsearch://reference/enrich-processor/grok-processor.md) {{es}} documentation for more information. -The Grok processor uses a set of predefined patterns to match the log messages and extract the fields. -You can also define your own pattern definitions by expanding the `Optional fields` section. You can then define your own patterns and use them in the Grok processor. -The patterns are defined in the following format: +## Example grok pattern [streams-grok-example] + +Grok patterns are defined in the following format: ``` { @@ -30,18 +38,28 @@ The previous pattern can then be used in the processor. ``` ## Generate patterns [streams-grok-patterns] +:::{note} Requires an LLM Connector to be configured. +::: + Instead of writing the Grok patterns by hand, you can use the **Generate Patterns** button to generate the patterns for you. -% TODO Elastic LLM? +Generated patterns work best on semi-structured data. For very custom logs with a lot of text, creating manual patterns general creates more accurate results. ![generated patterns](<../../../../images/logs-streams-patterns.png>) -Click the plus icon next to the pattern to accept it and add it to the list of patterns used by the Grok processor. +To add a generated grok pattern: + +1. Select **Create** → **Create processor**. +1. Select **Grok** from the **Processor** menu. +1. Select **Generate pattern**. +1. Select **Accept** to add a generated pattern to the list of patterns used by the grok processor. + +### How does **Generate patterns** work? [streams-grok-pattern-generation] +% need to check to make sure this is still accurate. -### How does the pattern generation work? [streams-grok-pattern-generation] Under the hood, the 100 samples on the right side are grouped into categories of similar messages. For each category, a Grok pattern is generated by sending a few samples to the LLM. Matching patterns are then shown in the UI. :::{note} -This can incur additional costs, depending on the LLM connector you are using. Typically a single iteration uses between 1000 and 5000 tokens, depending on the number of identified categories and the length of the messages. +This can incur additional costs, depending on the LLM connector you are using. Typically a single iteration uses between 1000 and 5000 tokens depending on the number of identified categories and the length of the messages. ::: \ No newline at end of file diff --git a/solutions/observability/streams/management/extract/manual-pipeline-configuration.md b/solutions/observability/streams/management/extract/manual-pipeline-configuration.md new file mode 100644 index 0000000000..3d55b2f63b --- /dev/null +++ b/solutions/observability/streams/management/extract/manual-pipeline-configuration.md @@ -0,0 +1,6 @@ +--- +applies_to: + serverless: ga + stack: preview 9.1, ga 9.2 +--- +# Manual pipeline configuration [streams-manual-pipeline-configuration] \ No newline at end of file diff --git a/solutions/observability/streams/management/extract/rename.md b/solutions/observability/streams/management/extract/rename.md new file mode 100644 index 0000000000..ea6e7dd033 --- /dev/null +++ b/solutions/observability/streams/management/extract/rename.md @@ -0,0 +1,16 @@ +--- +applies_to: + serverless: ga + stack: preview 9.1, ga 9.2 +--- +# Rename processor [streams-rename-processor] +% need use cases + +Use the rename processor to change the name of a field, moving its value to a new field name and removing the original. + +To use a rename processor: + +1. Set **Source Field** to the field you want to rename. +1. Set **Target field** to the new name you want to use for the **Source Field**. + +This functionality uses the {{es}} rename pipeline processor. Refer to the [rename processor](elasticsearch://reference/enrich-processor/rename-processor.md) {{es}} documentation for more information. \ No newline at end of file diff --git a/solutions/observability/streams/management/extract/set.md b/solutions/observability/streams/management/extract/set.md new file mode 100644 index 0000000000..c31a08ec38 --- /dev/null +++ b/solutions/observability/streams/management/extract/set.md @@ -0,0 +1,16 @@ +--- +applies_to: + serverless: ga + stack: preview 9.1, ga 9.2 +--- +# Set processor [streams-set-processor] +% need use cases + +Use the set processor to assign a specific value to a field, creating the field if it doesn’t exist or overwriting its value if it does. + +To use a set processor: + +1. Set **Source Field** to the field you want to insert, upsert, or update +1. Set **Value** to the value you want the source field to be set to. + +This functionality uses the {{es}} set pipeline processor. Refer to the [set processor](elasticsearch://reference/enrich-processor/set-processor.md) {{es}} documentation for more information. \ No newline at end of file diff --git a/solutions/observability/streams/management/retention.md b/solutions/observability/streams/management/retention.md index 52bfbcd1e3..1e932312ea 100644 --- a/solutions/observability/streams/management/retention.md +++ b/solutions/observability/streams/management/retention.md @@ -5,40 +5,43 @@ applies_to: stack: preview 9.1 --- -# Manage data retention for log streams [streams-data-retention] +# Manage data retention for streams [streams-data-retention] -Use the **Data retention** tab on the **Manage stream** page to set how long your stream retains data and to get insight into your stream's data ingestion and storage size. +Use the **Retention** tab to set how long your stream retains data and to get insight into your stream's data ingestion and storage size. -![Screenshot of the data retention UI](<../../../images/logs-streams-retention.png>) +The **Retention** tab contains the following components to help you determine how long you want your stream to retain data: -The **Data retention** page is made up of the following components that can help you determine how long you want your stream to retain data: +- **Retention**: The current retention policy, including the source of the policy. +- **Storage size**: The total size and number of documents in the stream. +- **Ingestion averages**: Estimated ingestion per day and month, calculated based on the total size of all data in the stream divided by the stream's age. +- **ILM policy data tiers**: {applies_to}`stack: preview 9.1, ga 9.2` The amount of data in each data tier (**Hot**, **Warm**, **Cold**). +- **Ingestion over time**: Estimated ingestion rate per time bucket. The bucket interval is dynamic and adjusts based on the selected time range. The ingestion rate is calculated using the average document size in the stream multiplied by the number of documents in each bucket. This is an estimate, and the actual ingestion rate may vary. -- **Retention period**: The minimum number of days after which the data is deleted -- **Source**: The origin of the data retention policy. -- **Last updated**: When data retention was last updated for the selected stream. -- **Ingestion**: Estimated ingestion per day and month, calculated based on the total size of all data in the stream divided by the stream's age. This is an estimate, and the actual ingestion may vary. -- **Total doc count**: The total number of documents in the stream. -- **Ingestion Rate**: Estimated ingestion rate per time bucket. The bucket interval is dynamic and adjusts based on the selected time range. The ingestion rate is calculated using the average document size in the stream multiplied by the number of documents in each bucket. This is an estimate, and the actual ingestion rate may vary. -- **Policy summary**: {applies_to}`stack: preview 9.1` The amount of data ingested per phase (hot, warm, cold). +For more information on data retention, refer to [Data stream lifecycle](../../../../manage-data/lifecycle/data-stream.md). ## Edit the data retention [streams-update-data-retention] -From any stream page, select **Edit data retention** to change how long your data stream retains data. +From the **Retention** tab, select **Edit data retention** to change how long your data stream retains data. + +### Inherit from index template +When enabled, your stream uses the retention configuration from its index template. ### Set a specific retention period The **Retention period** is the minimum number of days after which the data is deleted. To set data retention to a specific time period: -1. Select **Edit data retention** → **Set specific retention days**. -1. From here, set the period of time you want to retain data for this stream. +1. From the **Retention** tab, select **Edit data retention**. +1. Turn off **Inherit from index template** if enabled. +1. Select **Custom period**. +1. Set the period of time you want to retain data for this stream. To define a global default retention policy, refer to [project settings](../../../../deploy-manage/deploy/elastic-cloud/project-settings.md). ### Follow an ILM policy ```{applies_to} -stack: ga 9.1 +stack: preview 9.1, ga 9.2 ``` [ILM policies](../../../../manage-data/lifecycle/index-lifecycle-management.md) let you automate and standardize data retention across streams and other data streams. To have your streams follow an existing policy: -1. Select **Edit data retention** → **Use a lifecycle policy**. +1. From the **Retention** tab, select **Edit data retention**. 1. Select a pre-defined ILM policy from the list. You can also create a new ILM policy. Refer to [Configure a lifecycle policy](../../../../manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) for more information. \ No newline at end of file diff --git a/solutions/observability/streams/management/schema.md b/solutions/observability/streams/management/schema.md new file mode 100644 index 0000000000..3078efe47b --- /dev/null +++ b/solutions/observability/streams/management/schema.md @@ -0,0 +1,12 @@ +--- +navigation_title: Manage field mapping +applies_to: + serverless: ga + stack: preview 9.1, ga 9.2 +--- + +# Manage field mapping using the Schema tab [streams-schema] + +## Permissions to edit and add fields [streams-schema-permissions] + +## Field statuses [streams-schema-statuses] \ No newline at end of file diff --git a/solutions/observability/streams/management/significant-events.md b/solutions/observability/streams/management/significant-events.md new file mode 100644 index 0000000000..cdda082d54 --- /dev/null +++ b/solutions/observability/streams/management/significant-events.md @@ -0,0 +1,16 @@ +--- +applies_to: + serverless: ga + stack: preview 9.1, ga 9.2 +--- + +# Add significant events + +Significant events surface noteworthy occurrences within a Stream’s data. These can include events like out-of-memory errors, startup and shutdown messages, and configuration changes. + +You can define significant events by either: + +- **Generate significant events with AI:** If you don't know what you're looking for, let AI suggest queries based on your data. Then, select the suggestions that make sense to you. +- **Create significant events from a query:** If you know what you're looking for, write your own query to add an event. + +% ## Query Examples - need to add some. \ No newline at end of file diff --git a/solutions/observability/streams/streams.md b/solutions/observability/streams/streams.md index 417228488d..21c5e1bef5 100644 --- a/solutions/observability/streams/streams.md +++ b/solutions/observability/streams/streams.md @@ -1,20 +1,17 @@ --- applies_to: - serverless: preview - stack: preview 9.1 + serverless: ga + stack: preview 9.1, ga 9.2 --- -:::{warning} -Streams is currently in Technical Preview. This feature may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. -::: - # Streams +% can we document when to use streams vs when to not use streams? + +Streams provides a single, centralized UI within {{kib}} that streamlines common tasks like extracting fields, setting data retention, and rerouting data, so you don't need to use multiple applications or manually configure underlying {{es}} components. -Streams provides a single, centralized UI within {{kib}} that streamlines common tasks, reducing the need to navigate multiple applications or manually configure underlying {{es}} components. Key workflows include: -- [Extract fields](../streams/management/extract.md) from your documents. -- [Change the data retention](../streams/management/retention.md) of a stream. +A Stream directly corresponds to an {{es}} data stream (for example, `logs-myapp-default`). Any updates you perform in Streams configures a specific [data stream](../../../manage-data/data-store/data-streams.md). -A Stream directly corresponds to an {{es}} data stream (for example, `logs-myapp-default`). Operations performed in the Streams UI configure that specific data stream. +% need to add a wired vs classic streams section ## Required permissions @@ -23,25 +20,20 @@ Streams requires the following Elastic Cloud Serverless roles: - Admin: ability to manage all Streams. - Editor/Viewer: limited access, unable to perform all actions. -## Access the Streams UI - -In {{obs-serverless}}, Streams is automatically available. - -In {{stack}} version 9.1 and later, you can enable Streams in the {{observability}} Advanced Settings. To do this: - -1. Go to **Management** > **Stack Management** > **Advanced Settings**, or search for "Advanced Settings" in the [global search field](../../../explore-analyze/find-and-organize/find-apps-and-objects.md). -1. Enable **Streams UI** under **Observability**. +## Access Streams -In {{serverless-short}} or after enabling Streams in {{stack}}, access the UI in one of the following ways: +Open Streams from one of the following places in {{kib}}: - Select **Streams** from the navigation menu or use the [global search field](../../../explore-analyze/find-and-organize/find-apps-and-objects.md). -- From **Discover**, expand a document's details flyout and select **Stream** or an action associated with the document's data stream. Streams will open filtered to only the selected stream. This only works for documents stored in a data stream. +- You can open the data stream for a specific document from **Discover**. To do this, expand a document's details flyout and select **Stream** or an action associated with the document's data stream. Streams will open filtered to the selected data stream. This only works for documents stored in a data stream. ## Manage individual streams [streams-management-tab] Interact with and configure your streams in the following ways: -- [Data retention](./management/retention.md): Manage how your stream retains data and get insight into data ingestion and storage size under the **Data retention** tab. -- [Processing](./management/extract.md): Parse and extract information from log messages into dedicated fields under the **Processing** tab. -- [Advanced](./management/advanced.md): Review and manually modify the inner workings of your stream under the **Advanced** tab. \ No newline at end of file +- [Retention](./management/retention.md): Manage how your stream retains data and get insight into data ingestion and storage size. +- [Processing](./management/extract.md): Parse and extract information from documents into dedicated fields. +- [Schema](./management/schema.md): Manage field mappings. +- [Data quality](./management/data-quality.md): Get information about failed and degraded documents in your stream. +- [Advanced](./management/advanced.md): Review and manually modify underlying {{es}} components of your stream. \ No newline at end of file diff --git a/solutions/toc.yml b/solutions/toc.yml index 7c3c9ad6af..0ccf42b97a 100644 --- a/solutions/toc.yml +++ b/solutions/toc.yml @@ -455,15 +455,22 @@ toc: - file: observability/logs/logs-index-template-reference.md children: - file: observability/logs/logs-index-template-defaults.md - - file: observability/streams/streams.md - children: - - file: observability/streams/management/extract.md - children: - - file: observability/streams/management/extract/date.md - - file: observability/streams/management/extract/dissect.md - - file: observability/streams/management/extract/grok.md - - file: observability/streams/management/retention.md - - file: observability/streams/management/advanced.md + - file: observability/streams/streams.md + children: + - file: observability/streams/management/extract.md + children: + - file: observability/streams/management/extract/date.md + - file: observability/streams/management/extract/dissect.md + - file: observability/streams/management/extract/grok.md + - file: observability/streams/management/extract/set.md + - file: observability/streams/management/extract/rename.md + - file: observability/streams/management/extract/append.md + - file: observability/streams/management/extract/manual-pipeline-configuration.md + - file: observability/streams/management/retention.md + - file: observability/streams/management/schema.md + - file: observability/streams/management/data-quality.md + - file: observability/streams/management/significant-events.md + - file: observability/streams/management/advanced.md - file: observability/incident-management.md children: - file: observability/incident-management/alerting.md