Streamlang ref

solutions/observability/ai/observability-ai-assistant.md

@@ -58,7 +58,7 @@ The [**GenAI settings**](/explore-analyze/ai-features/manage-access-to-ai-assist
 
 - Manage which AI connectors are available in your environment.
 - Enable or disable AI Assistant and other AI-powered features in your environment.
-- {applies_to}`stack: ga 9.2` {applies_to}`serverless: unavailable` Specify in which Elastic solutions the `AI Assistant for {{observability}} and Search` and the `AI Assistant for Security` appear.
+- {applies_to}`stack: ga 9.2` {applies_to}`serverless: unavailable` Specify in which Elastic solutions the `AI Assistant for Observability and Search` and the `AI Assistant for Security` appear.
 
 ## Your data and the AI Assistant [data-information]
 

solutions/observability/streams/management/extract.md

@@ -34,25 +34,32 @@
 
 Streams supports the following processors:
 
-- [**Drop**](./extract/drop.md): Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on a condition.
-- [**Remove**](./extract/remove.md): Removes existing fields.
-- [**Date**](./extract/date.md): Converts date strings into timestamps, with options for timezone, locale, and output formatting.
+- [**Append**](./extract/append.md): Adds a value to an existing array field, or creates the field as an array if it doesn't exist.
+- [**Concat**](./extract/concat.md): Concatenates a mix of field values and literal strings into a single field.
 - [**Convert**](./extract/convert.md): Converts a field in the currently ingested document to a different type, such as converting a string to an integer.
-- [**Replace**](./extract/replace.md): Replaces parts of a string field according to a regular expression pattern with a replacement string.
+- [**Date**](./extract/date.md): Converts date strings into timestamps, with options for timezone, locale, and output formatting.
 - [**Dissect**](./extract/dissect.md): Extracts fields from structured log messages using defined delimiters instead of patterns, making it faster than Grok and ideal for consistently formatted logs.
+- [**Drop**](./extract/drop.md): Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on a condition.
 - [**Grok**](./extract/grok.md): Extracts fields from unstructured log messages using predefined or custom patterns, supports multiple match attempts in sequence, and can automatically generate patterns with an [LLM connector](/explore-analyze/ai-features/llm-guides/llm-connectors.md).
-- [**Set**](./extract/set.md): Assigns a specific value to a field, creating the field if it doesn’t exist or overwriting its value if it does.
+- [**Join**](./extract/join.md): Concatenates the values of multiple fields with a delimiter.
+- [**Lowercase**](./extract/lowercase.md): Converts a string field to lowercase.
 - [**Math**](./extract/math.md): Evaluates arithmetic or logical expressions.
+- [**Network direction**](./extract/network-direction.md): Determines network traffic direction based on source and destination IP addresses.
+- [**Redact**](./extract/redact.md): Redacts sensitive data in a string field by matching grok patterns.
+- [**Remove**](./extract/remove.md): Removes existing fields or removes fields by prefix.
 - [**Rename**](./extract/rename.md): Changes the name of a field, moving its value to a new field name and removing the original.
-- [**Append**](./extract/append.md): Adds a value to an existing array field, or creates the field as an array if it doesn’t exist.
+- [**Replace**](./extract/replace.md): Replaces parts of a string field according to a regular expression pattern with a replacement string.
+- [**Set**](./extract/set.md): Assigns a specific value to a field, creating the field if it doesn't exist or overwriting its value if it does.
+- [**Trim**](./extract/trim.md): Removes leading and trailing whitespace from a string field.
+- [**Uppercase**](./extract/uppercase.md): Converts a string field to uppercase.
 
 ### Processor limitations and inconsistencies [streams-processor-inconsistencies]
 
-Streams exposes a Streamlang configuration, but internally it relies on {{es}} ingest pipeline processors and ES|QL. Streamlang doesn’t always have 1:1 parity with the ingest processors because it needs to support options that work in both ingest pipelines and ES|QL. In most cases, you won’t need to worry about these details, but the underlying design decisions still affect the UI and available configuration options. The following are some limitations and inconsistencies when using Streamlang processors:
+Streams exposes a Streamlang configuration, but internally it relies on {{es}} ingest pipeline processors and ES|QL. Streamlang doesn't always have 1:1 parity with the ingest processors because it needs to support options that work in both ingest pipelines and ES|QL. In most cases, you won't need to worry about these details, but the underlying design decisions still affect the UI and available configuration options. The following are some limitations and inconsistencies when using Streamlang processors:
 
-- **Consistently typed fields**: ES|QL requires one consistent type per column, so workflows that produce mixed types across documents won’t transpile.
+- **Consistently typed fields**: ES|QL requires one consistent type per column, so workflows that produce mixed types across documents won't transpile.
 - **Conversion of types**: ES|QL and ingest pipelines accept different conversion combinations and strictness (especially for strings), so `convert` can behave differently across targets.
-- **Multi-value commands/functions**: Fields can contain one or multiple values. ES|QL and ingest processors don’t always handle these cases the same way. For example, grok in ES|QL handles multiple values automatically, while the grok processor does not
+- **Multi-value commands/functions**: Fields can contain one or multiple values. ES|QL and ingest processors don't always handle these cases the same way. For example, grok in ES|QL handles multiple values automatically, while the grok processor does not
 - **Conditional execution**: ES|QL's enforced table shape limits conditional casting, parsing, and wildcard field operations that ingest pipelines can do per-document.
 - **Arrays of objects / flattening**: Ingest pipelines preserve nested JSON arrays, while ES|QL flattens to columns, so operations like rename and delete on parent objects can differ or fail.
 
@@ -152,6 +159,8 @@
 - Users who prefer working with code
 - Advanced configurations with complex or deeply nested conditions
 
+Refer to the [Streamlang reference](./streamlang.md) for the complete syntax, including all available processors, condition operators, and examples.
+
 
 ### Preview changes [streams-preview-changes]
 
@@ -273,4 +282,4 @@
 
 - Streams does not support all processors. More processors will be added in future versions.
 - The data preview simulation might not accurately reflect the changes to the existing data when editing existing processors or re-ordering them. Streams will allow proper simulations using original documents in a future version.
-- Streams can't properly handle arrays. While it supports basic actions like appending or renaming, it can't access individual array elements. For classic streams, the workaround is to use the [manual pipeline configuration](./extract/manual-pipeline-configuration.md) that supports Painless scripting and all ingest processors.
+- Streams can't properly handle arrays. While it supports basic actions like appending or renaming, it can't access individual array elements. For classic streams, the workaround is to use the [manual pipeline configuration](./extract/manual-pipeline-configuration.md) that supports Painless scripting and all ingest processors.

solutions/observability/streams/management/extract/append.md

@@ -13,9 +13,8 @@ products:
   - id: elastic-stack
 ---
 # Append processor [streams-append-processor]
-% Need use cases
 
-Use the **Append** processor to add a value to an existing array field, or create the field as an array if it doesn’t exist.
+Use the **Append** processor to add a value to an existing array field, or create the field as an array if it doesn't exist.
 
 To use an append processor:
 
@@ -24,4 +23,22 @@ To use an append processor:
 1. Set **Source Field** to the field you want append values to.
 1. Set **Target field** to the values you want to append to the **Source Field**.
 
-This functionality uses the {{es}} [append processor](elasticsearch://reference/enrich-processor/append-processor.md) internally, but you configure it in Streamlang. Streamlang doesn’t always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
+This functionality uses the {{es}} [append processor](elasticsearch://reference/enrich-processor/append-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
+
+## YAML reference [streams-append-yaml-reference]
+
+In [YAML mode](../extract.md#streams-editing-yaml-mode), configure the append processor using the following parameters. For the complete Streamlang syntax, refer to the [Streamlang reference](../streamlang.md).
+
+| Parameter | Type | Required | Description |
+| --- | --- | --- | --- |
+| `to` | string | Yes | Array field to append to. |
+| `value` | array | Yes | Values to append. |
+| `allow_duplicates` | boolean | No | When `false`, duplicate values are not appended. |
+
+```yaml
+- action: append
+  to: attributes.tags
+  value:
+    - processed
+    - reviewed
+```

solutions/observability/streams/management/extract/concat.md

@@ -0,0 +1,51 @@
+---
+applies_to:
+  serverless: ga
+  stack: ga 9.4+
+products:
+  - id: observability
+  - id: elasticsearch
+  - id: kibana
+  - id: cloud-serverless
+  - id: cloud-hosted
+  - id: cloud-enterprise
+  - id: cloud-kubernetes
+  - id: elastic-stack
+---
+
+# Concat processor [streams-concat-processor]
+
+The **Concat** processor concatenates a mix of field values and literal strings into a single field.
+
+To concatenate values:
+
+1. Select **Create** → **Create processor**.
+1. Select **Concat** from the **Processor** menu.
+1. Set the items to concatenate. Each item is either a field reference or a literal string value.
+1. Set the **Target field** where the concatenated result is stored.
+
+## YAML reference [streams-concat-yaml-reference]
+
+In [YAML mode](../extract.md#streams-editing-yaml-mode), configure the concat processor using the following parameters. For the complete Streamlang syntax, refer to the [Streamlang reference](../streamlang.md).
+
+| Parameter | Type | Required | Description |
+| --- | --- | --- | --- |
+| `from` | array | Yes | Items to concatenate. Each item is either `{ type: "field", value: "<field_name>" }` or `{ type: "literal", value: "<text>" }`. |
+| `to` | string | Yes | Target field. |
+| `ignore_missing` | boolean | No | When `true`, skip this processor if any referenced field is missing. |
+
+```yaml
+- action: concat
+  from:
+    - type: literal
+      value: "User: "
+    - type: field
+      value: attributes.username
+    - type: literal
+      value: " (ID: "
+    - type: field
+      value: attributes.user_id
+    - type: literal
+      value: ")"
+  to: attributes.user_summary
+```

solutions/observability/streams/management/extract/convert.md

@@ -28,4 +28,26 @@ To convert a field to a different data type:
 If you add a **Convert** processor inside a condition group (a **WHERE** block), you must set a **Target field**.
 ::::
 
-This functionality uses the {{es}} [Convert processor](elasticsearch://reference/enrich-processor/convert-processor.md) internally, but you configure it in Streamlang. Streamlang doesn’t always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
+This functionality uses the {{es}} [Convert processor](elasticsearch://reference/enrich-processor/convert-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
+
+## YAML reference [streams-convert-yaml-reference]
+
+In [YAML mode](../extract.md#streams-editing-yaml-mode), configure the convert processor using the following parameters. For the complete Streamlang syntax, refer to the [Streamlang reference](../streamlang.md).
+
+| Parameter | Type | Required | Description |
+| --- | --- | --- | --- |
+| `from` | string | Yes | Source field containing the value to convert. |
+| `type` | string | Yes | Target data type: `integer`, `long`, `double`, `boolean`, or `string`. |
+| `to` | string | No | Target field for the converted value. Defaults to the source field. |
+| `ignore_missing` | boolean | No | When `true`, skip this processor if the source field is missing. |
+
+:::{note}
+When using `convert` inside a condition (`where` block), you must set a `to` field that is different from `from`.
+:::
+
+```yaml
+- action: convert
+  from: attributes.status_code
+  type: integer
+  to: attributes.status_code_int
+```

solutions/observability/streams/management/extract/date.md

@@ -24,7 +24,7 @@ To extract a timestamp field using the date processor:
 1. Set the **Source Field** to the field containing the timestamp.
 1. Set the **Format** field to one of the accepted date formats (ISO8602, UNIX, UNIX_MS, or TAI64N) or use a Java time pattern. Refer to the [example formats](#streams-date-examples) for more information.
 
-This functionality uses the {{es}} [Date processor](elasticsearch://reference/enrich-processor/date-processor.md) internally, but you configure it in Streamlang. Streamlang doesn’t always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
+This functionality uses the {{es}} [Date processor](elasticsearch://reference/enrich-processor/date-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
 
 ## Example formats [streams-date-examples]
 
@@ -58,4 +58,28 @@ You can set the following optional fields for the date processor in the **Advanc
 | Target field | The field that will hold the parsed date. Defaults to `@timestamp`. |
 | Timezone | The timezone to use when parsing the date. Supports template snippets. Defaults to `UTC`. |
 | Locale | The locale to use when parsing the date, relevant when parsing month names or weekdays. Supports template snippets. Defaults to `ENGLISH`. |
-| Output format | The format to use when writing the date to `target_field`. Must be a valid Java time pattern. Defaults to `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. |
+| Output format | The format to use when writing the date to `target_field`. Must be a valid Java time pattern. Defaults to `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. |
+
+## YAML reference [streams-date-yaml-reference]
+
+In [YAML mode](../extract.md#streams-editing-yaml-mode), configure the date processor using the following parameters. For the complete Streamlang syntax, refer to the [Streamlang reference](../streamlang.md).
+
+| Parameter | Type | Required | Description |
+| --- | --- | --- | --- |
+| `from` | string | Yes | Source field containing the date string. |
+| `formats` | string[] | Yes | Date formats to try, in order (for example, `ISO8601`, `UNIX`, or a Java time pattern). |
+| `to` | string | No | Target field for the parsed date. Defaults to `@timestamp`. |
+| `output_format` | string | No | Format for the output date string. Must be a valid Java time pattern. |
+| `timezone` | string | No | Timezone to use when parsing. Defaults to `UTC`. |
+| `locale` | string | No | Locale to use when parsing month names or weekdays. |
+
+```yaml
+- action: date
+  from: attributes.timestamp
+  formats:
+    - "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
+    - "yyyy-MM-dd HH:mm:ss"
+  to: attributes.parsed_time
+  output_format: "yyyy-MM-dd"
+  timezone: "America/New_York"
+```

solutions/observability/streams/management/extract/dissect.md

@@ -60,4 +60,21 @@ To add a generated dissect pattern:
 ### How does **Generate patterns** work? [streams-dissect-pattern-generation]
 
 :::{include} ../../../../_snippets/streams-suggestions.md
-:::
+:::
+
+## YAML reference [streams-dissect-yaml-reference]
+
+In [YAML mode](../extract.md#streams-editing-yaml-mode), configure the dissect processor using the following parameters. For the complete Streamlang syntax, refer to the [Streamlang reference](../streamlang.md).
+
+| Parameter | Type | Required | Description |
+| --- | --- | --- | --- |
+| `from` | string | Yes | Source field to parse. |
+| `pattern` | string | Yes | Dissect pattern with `%{field}` placeholders. |
+| `append_separator` | string | No | Separator used when concatenating target fields. |
+| `ignore_missing` | boolean | No | When `true`, skip this processor if the source field is missing. |
+
+```yaml
+- action: dissect
+  from: body.message
+  pattern: "%{attributes.timestamp} %{attributes.level} %{attributes.message}"
+```

solutions/observability/streams/management/extract/drop.md

@@ -27,4 +27,21 @@ To configure a condition for dropping documents:
   The default is the `always` condition. Not setting a specific condition results in every document that matches the drop condition getting dropped from indexing.
   :::
 
-This functionality uses the {{es}} [Drop processor](elasticsearch://reference/enrich-processor/drop-processor.md) internally, but you configure it in Streamlang. Streamlang doesn’t always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
+This functionality uses the {{es}} [Drop processor](elasticsearch://reference/enrich-processor/drop-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
+
+## YAML reference [streams-drop-yaml-reference]
+
+In [YAML mode](../extract.md#streams-editing-yaml-mode), configure the drop document processor using the following parameters. For the complete Streamlang syntax, refer to the [Streamlang reference](../streamlang.md).
+
+The `drop_document` processor has no additional parameters beyond the common options. Use a `where` [condition](../streamlang.md#streams-streamlang-conditions) to specify when documents should be dropped.
+
+:::{warning}
+If no condition is set, the default `always` condition drops every document.
+:::
+
+```yaml
+- action: drop_document
+  where:
+    field: attributes.path
+    eq: "/health"
+```

solutions/observability/streams/management/extract/grok.md

@@ -69,4 +69,24 @@ To add a generated grok pattern:
 ### How does **Generate patterns** work? [streams-grok-pattern-generation]
 
 :::{include} ../../../../_snippets/streams-suggestions.md
-:::
+:::
+
+## YAML reference [streams-grok-yaml-reference]
+
+In [YAML mode](../extract.md#streams-editing-yaml-mode), configure the grok processor using the following parameters. For the complete Streamlang syntax, refer to the [Streamlang reference](../streamlang.md).
+
+| Parameter | Type | Required | Description |
+| --- | --- | --- | --- |
+| `from` | string | Yes | Source field to parse. |
+| `patterns` | string[] | Yes | One or more grok patterns, tried in order. |
+| `pattern_definitions` | object | No | Custom pattern definitions as key-value pairs. |
+| `ignore_missing` | boolean | No | When `true`, skip this processor if the source field is missing. |
+
+```yaml
+- action: grok
+  from: body.message
+  patterns:
+    - "%{IP:attributes.client_ip} %{WORD:attributes.method} %{URIPATHPARAM:attributes.path}"
+  pattern_definitions:
+    MY_PATTERN: "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}"
+```