diff --git a/apl/apl-features.mdx b/apl/apl-features.mdx index a892de7e..f9754c90 100644 --- a/apl/apl-features.mdx +++ b/apl/apl-features.mdx @@ -280,10 +280,15 @@ keywords: ['axiom documentation', 'documentation', 'axiom', 'APL', 'axiom proces | Tabular operator | [join](/apl/tabular-operators/join-operator) | Returns a dataset containing rows from two different tables based on conditions. | | Tabular operator | [limit](/apl/tabular-operators/limit-operator) | Returns the top N rows from the input dataset. | | Tabular operator | [lookup](/apl/tabular-operators/lookup-operator) | Returns a dataset where rows from one dataset are enriched with matching columns from a lookup table based on conditions. | +| Tabular operator | [make-series](/apl/tabular-operators/make-series) | Returns a dataset where the specified field is aggregated into a time series. | +| Tabular operator | [mv-expand](/apl/tabular-operators/mv-expand) | Returns a dataset where the specified field is expanded into multiple rows. | | Tabular operator | [order](/apl/tabular-operators/order-operator) | Returns the input dataset, sorted according to the specified fields and order. | | Tabular operator | [parse](/apl/tabular-operators/parse-operator) | Returns the input dataset with new fields added based on the specified parsing pattern. | +| Tabular operator | [parse-kv](/apl/tabular-operators/parse-kv) | Returns a dataset where key-value pairs are extracted from a string field into individual columns. | +| Tabular operator | [parse-where](/apl/tabular-operators/parse-where) | Returns a dataset where values from a string are extracted based on a pattern. | | Tabular operator | [project-away](/apl/tabular-operators/project-away-operator) | Returns the input dataset excluding the specified fields. | | Tabular operator | [project-keep](/apl/tabular-operators/project-keep-operator) | Returns a dataset with only the specified fields. | +| Tabular operator | [project-rename](/apl/tabular-operators/project-rename) | Returns a dataset where the specified field is renamed according to the specified pattern. | | Tabular operator | [project-reorder](/apl/tabular-operators/project-reorder-operator) | Returns a table with the specified fields reordered as requested followed by any unspecified fields in their original order. | | Tabular operator | [project](/apl/tabular-operators/project-operator) | Returns a dataset containing only the specified fields. | | Tabular operator | [redact](/apl/tabular-operators/redact-operator) | Returns the input dataset with sensitive data replaced or hashed. | diff --git a/apl/scalar-functions/conversion-functions/toarray.mdx b/apl/scalar-functions/conversion-functions/toarray.mdx index faf85e06..f673aa7d 100644 --- a/apl/scalar-functions/conversion-functions/toarray.mdx +++ b/apl/scalar-functions/conversion-functions/toarray.mdx @@ -3,7 +3,7 @@ title: toarray description: 'This page explains how to use the toarray function in APL.' --- -Use the `toarray` function in APL to convert a dynamic-typed input—such as a bag, property bag, or JSON array—into a regular array. This is helpful when you want to process the elements individually with array functions like `array_length`, `array_index_of`, or `mv-expand`. +Use the `toarray` function in APL to convert a dynamic-typed input—such as a bag, property bag, or JSON array—into a regular array. This is helpful when you want to process the elements individually with array functions like `array_length` or `array_index_of`. You typically use `toarray` when working with semi-structured data, especially after parsing JSON from log fields or external sources. It lets you access and manipulate nested collections using standard array operations. diff --git a/apl/tabular-operators/make-series.mdx b/apl/tabular-operators/make-series.mdx new file mode 100644 index 00000000..c316d001 --- /dev/null +++ b/apl/tabular-operators/make-series.mdx @@ -0,0 +1,178 @@ +--- +title: make-series +description: 'This page explains how to use the make-series operator in APL.' +--- + +## Introduction + +The `make-series` operator transforms event data into array-based time series. Instead of producing one row per time bucket, `make-series` encodes the values and corresponding timestamps into arrays stored in table fields. This makes it possible to apply `series_*` functions for advanced manipulations such as moving averages, smoothing, anomaly detection, or other time-series computations. + +You find this operator useful when you want to: + +- Turn event data into array-encoded time series for further analysis. +- Apply `series_*` functions (for example, `series_fir`, `series_stats`) to aggregated data. +- Postprocess and then expand arrays back into rows with `mv-expand` for visualization or downstream queries. + +Unlike `summarize`, which produces row-based aggregations, `make-series` is designed specifically for creating and manipulating array-based time series. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk SPL, the `timechart` command creates row-based time series, with one row per time bucket. In APL, the `make-series` operator instead encodes the series into arrays, which you can later manipulate or expand. This is a key difference from SPL’s row-based approach. + + +```sql Splunk example +index=sample-http-logs +| timechart span=1m avg(req_duration_ms) +```` + +```kusto APL equivalent +['sample-http-logs'] +| make-series avg(req_duration_ms) default=0 on _time from ago(1h) to now() step 1m +``` + + + + + + +In ANSI SQL, you typically use `GROUP BY` with a generated series or calendar table to create row-based time buckets. In APL, `make-series` creates arrays of values and timestamps in a single row. This lets you perform array-based computations on the time series before optionally expanding back into rows. + + +```sql SQL example +SELECT + time_bucket('1 minute', _time) AS minute, + AVG(req_duration_ms) AS avg_duration +FROM sample_http_logs +WHERE _time > NOW() - interval '1 hour' +GROUP BY minute +ORDER BY minute +``` + +```kusto APL equivalent +['sample-http-logs'] +| make-series avg(req_duration_ms) default=0 on _time from ago(1h) to now() step 1m +``` + + + + + + +## Usage + +### Syntax + +```kusto +make-series [Aggregation [, ...]] + [default = DefaultValue] + on TimeField + [in Range] + step StepSize + [by GroupingField [, ...]] +``` + +### Parameters + +| Parameter | Description | +| ---------------- | --------------------------------------------------------------------------------------------------------------- | +| `Aggregation` | One or more aggregation functions (for example, `avg()`, `count()`, `sum()`) applied to each time bin, producing arrays of values. | +| `default` | A value to use when no records exist in a time bin. | +| `TimeField` | The field containing timestamps used for binning. | +| `Range` | An optional range expression specifying the start and end of the series (for example, `from ago(1h) to now()`). | +| `StepSize` | The size of each time bin (for example, `1m`, `5m`, `1h`). | +| `GroupingField` | Optional fields to split the series by, producing parallel arrays for each group. | + +### Returns + +The operator returns a table where each aggregation produces an array of values aligned with an array of time bins. Each row represents a group (if specified), with arrays that encode the entire time series. + +## Use case examples + + + + +You want to create an array-based time series of request counts, then compute a rolling average using a `series_*` function, and finally expand back into rows for visualization. + +**Query** + +```kusto +['sample-http-logs'] +| make-series count() on _time from now()-24h to now() step 5m +| extend moving_avg_count=series_fir(count_, dynamic([1, 1, 1, 1, 1])) +| mv-expand moving_avg_count to typeof(long), count_ to typeof(long), time to typeof(datetime) +| project-rename _time=time +| summarize avg(moving_avg_count), avg(count_) by bin(_time, 5m) +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20make-series%20count()%20on%20_time%20from%20now()-24h%20to%20now()%20step%205m%20%7C%20extend%20moving_avg_count%3Dseries_fir(count_%2C%20dynamic(%5B1%2C%201%2C%201%2C%201%2C%201%5D))%20%7C%20mv-expand%20moving_avg_count%20to%20typeof(long)%2C%20count_%20to%20typeof(long)%2C%20time%20to%20typeof(datetime)%20%7C%20project-rename%20_time%3Dtime%20%7C%20summarize%20avg(moving_avg_count)%2C%20avg(count_)%20by%20bin(_time%2C%205m)%22%2C%22queryOptions%22%3A%7B%22quickRange%22%3A%221d%22%7D%7D) + +**Output** + +| _time | count_ | moving_avg_count | +| ------------------- | ------ | ---------------- | +| 2025-09-29T10:00:00 | 120 | 118 | +| 2025-09-29T10:05:00 | 130 | 122 | +| 2025-09-29T10:10:00 | 110 | 121 | + +The query turns request counts into arrays, applies a smoothing function, and then expands the arrays back into rows for analysis. + + + + +You want to analyze span durations per service, storing them as arrays for later manipulation. + +**Query** + +```kusto +['otel-demo-traces'] +| make-series avg(duration) on _time from ago(2h) to now() step 10m by ['service.name'] +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D%20%7C%20make-series%20avg(duration)%20on%20_time%20from%20ago(2h)%20to%20now()%20step%2010m%20by%20%5B'service.name'%5D%22%7D) + +**Output** + +| service.name | avg_duration | time | +| --------------- | ----------------------------- | ----------------------- | +| frontend | [20ms, 18ms, 22ms, 19ms, ...] | [2025-09-29T08:00, ...] | +| checkout | [35ms, 40ms, 33ms, 37ms, ...] | [2025-09-29T08:00, ...] | + +The query produces array-encoded time series per service, which you can further process with `series_*` functions. + + + + +You want to analyze the rate of HTTP 500 errors in your logs per minute. + +**Query** + +```kusto +['sample-http-logs'] +| where status == '500' +| make-series count() default=0 on _time from ago(30m) to now() step 1m +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20where%20status%20%3D%3D%20'500'%20%7C%20make-series%20count()%20default%3D0%20on%20_time%20from%20ago(30m)%20to%20now()%20step%201m%22%7D) + +**Output** + +| count_ | _time | +| ------------------------ | ---------------------- | +| [1489, 1428, 1517, 1462, 1509, ...] | ["2025-09-30T09:08:14.921301725Z", "2025-09-30T09:09:14.921301725Z", ...] | + +The query generates a time series of HTTP 500 error counts as an array-based time series for further analysis with `series_*` functions. + + + + +## List of related operators + +- [extend](/apl/tabular-operators/extend-operator): Creates new calculated fields, often as preparation before `make-series`. Use `extend` when you want to preprocess data for time series analysis. +- [mv-expand](/apl/tabular-operators/mv-expand): Expands arrays into multiple rows. Use `mv-expand` to work with the arrays returned by `make-series`. +- [summarize](/apl/tabular-operators/summarize-operator): Aggregates rows into groups but does not generate continuous time bins. Use `summarize` when you want flexible grouping without forcing evenly spaced intervals. +- [top](/apl/tabular-operators/top-operator): Returns the top rows by a specified expression, not time series. Use `top` when you want to focus on the most significant values instead of trends over time. diff --git a/apl/tabular-operators/mv-expand.mdx b/apl/tabular-operators/mv-expand.mdx new file mode 100644 index 00000000..3807b314 --- /dev/null +++ b/apl/tabular-operators/mv-expand.mdx @@ -0,0 +1,104 @@ +--- +title: mv-expand +description: 'This page explains how to use the mv-expand operator in APL.' +--- + +The `mv-expand` operator expands dynamic arrays and property bags into multiple rows. Each element of the array or each property of the bag becomes its own row, while other columns are duplicated. + +You use `mv-expand` when you want to analyze or filter individual values inside arrays or objects. This is especially useful when working with logs that include lists of values, OpenTelemetry traces that contain arrays of spans, or security events that group multiple attributes into one field. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk SPL, the `mvexpand` command expands multi-value fields into separate events. The APL `mv-expand` operator works in a very similar way, splitting array values into individual rows. The main difference is that APL explicitly works with dynamic arrays or property bags, while Splunk handles multi-value fields implicitly. + + +```sql Splunk example +... | mvexpand request_uri +```` + +```kusto APL equivalent +['sample-http-logs'] +| mv-expand uri +``` + + + + + + +In ANSI SQL, you use `CROSS JOIN UNNEST` or `CROSS APPLY` to flatten arrays into rows. In APL, `mv-expand` provides a simpler and more direct way to achieve the same result. + + +```sql SQL example +SELECT id, value +FROM logs +CROSS JOIN UNNEST(request_uris) AS t(value) +``` + +```kusto APL equivalent +['sample-http-logs'] +| mv-expand uri +``` + + + + + + +## Usage + +### Syntax + +```kusto +mv-expand [kind=(bag|array)] [with_itemindex=IndexFieldName] FieldName [to typeof(Typename)] [limit Rowlimit] +``` + +### Parameters + +| Parameter | Description | +| -------------------------------- | -------------------------------------------------------------------------------------- | +| `kind` | Optional. Specifies whether the column is a bag (object) or an array. Defaults to `array`. | +| `with_itemindex=IndexFieldName` | Optional. Outputs an additional column with the zero-based index of the expanded item. | +| `FieldName` | Required. The name of the column that contains an array or object to expand. | +| `to typeof(Typename)` | Optional. Converts each expanded element to the specified type. | +| `limit Rowlimit` | Optional. Limits the number of expanded rows per record. | + +### Returns + +The operator returns a table where each element of the expanded array or each property of the expanded object is placed in its own row. Other columns are duplicated for each expanded row. + +## Use case example + +When analyzing logs, some values can be stored as arrays. You can use `mv-expand` to expand them into individual rows for easier filtering. + +**Query** + +```kusto +['sample-http-logs'] +| limit 100 +| mv-expand territories +| summarize count = count() by territory_name = tostring(territories) +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20limit%20100%20%7C%20mv-expand%20territories%20%7C%20summarize%20count%20%3D%20count()%20by%20territory_name%20%3D%20tostring(territories)%22%7D) + +**Output** + +| territory_name | count | +| ---------------- | ------- | +| United States | 67 | +| India | 22 | +| Japan | 12 | + +This query expands the `territories` array into rows and counts the most frequent territories. + +## List of related operators + +- [project](/apl/tabular-operators/project-operator): Selects or computes columns. Use it when you want to reshape data, not expand arrays. +- [summarize](/apl/tabular-operators/summarize-operator): Aggregates data across rows. Use it after expanding arrays to compute statistics. +- [top](/apl/tabular-operators/top-operator): Returns the top N rows by expression. Use it after expansion to find the most frequent values. diff --git a/apl/tabular-operators/overview.mdx b/apl/tabular-operators/overview.mdx index 2fc4a171..f8774625 100644 --- a/apl/tabular-operators/overview.mdx +++ b/apl/tabular-operators/overview.mdx @@ -18,11 +18,16 @@ The table summarizes the tabular operators available in APL. | [join](/apl/tabular-operators/join-operator) | Returns a dataset containing rows from two different tables based on conditions. | | [limit](/apl/tabular-operators/limit-operator) | Returns the top N rows from the input dataset. | | [lookup](/apl/tabular-operators/lookup-operator) | Returns a dataset where rows from one dataset are enriched with matching columns from a lookup table based on conditions. | +| [make-series](/apl/tabular-operators/make-series) | Returns a dataset where the specified field is aggregated into a time series. | +| [mv-expand](/apl/tabular-operators/mv-expand) | Returns a dataset where the specified field is expanded into multiple rows. | | [order](/apl/tabular-operators/order-operator) | Returns the input dataset, sorted according to the specified fields and order. | | [parse](/apl/tabular-operators/parse-operator) | Returns the input dataset with new fields added based on the specified parsing pattern. | +| [parse-kv](/apl/tabular-operators/parse-kv) | Returns a dataset where key-value pairs are extracted from a string field into individual columns. | +| [parse-where](/apl/tabular-operators/parse-where) | Returns a dataset where values from a string are extracted based on a pattern. | | [project](/apl/tabular-operators/project-operator) | Returns a dataset containing only the specified fields. | | [project-away](/apl/tabular-operators/project-away-operator) | Returns the input dataset excluding the specified fields. | | [project-keep](/apl/tabular-operators/project-keep-operator) | Returns a dataset with only the specified fields. | +| [project-rename](/apl/tabular-operators/project-rename) | Returns a dataset where the specified field is renamed according to the specified pattern. | | [project-reorder](/apl/tabular-operators/project-reorder-operator) | Returns a table with the specified fields reordered as requested followed by any unspecified fields in their original order. | | [redact](/apl/tabular-operators/redact-operator) | Returns the input dataset with sensitive data replaced or hashed. | | [sample](/apl/tabular-operators/sample-operator) | Returns a table containing the specified number of rows, selected randomly from the input dataset. | @@ -32,4 +37,4 @@ The table summarizes the tabular operators available in APL. | [take](/apl/tabular-operators/take-operator) | Returns the specified number of rows from the dataset. | | [top](/apl/tabular-operators/top-operator) | Returns the top N rows from the dataset based on the specified sorting criteria. | | [union](/apl/tabular-operators/union-operator) | Returns all rows from the specified tables or queries. | -| [where](/apl/tabular-operators/where-operator) | Returns a filtered dataset containing only the rows where the condition evaluates to true. | \ No newline at end of file +| [where](/apl/tabular-operators/where-operator) | Returns a filtered dataset containing only the rows where the condition evaluates to true. | diff --git a/apl/tabular-operators/parse-kv.mdx b/apl/tabular-operators/parse-kv.mdx new file mode 100644 index 00000000..61f6e0e6 --- /dev/null +++ b/apl/tabular-operators/parse-kv.mdx @@ -0,0 +1,109 @@ +--- +title: parse-kv +description: 'This page explains how to use the parse-kv operator in APL.' +--- + +The `parse-kv` operator parses key-value pairs from a string field into individual columns. You use it when your data is stored in a single string that contains structured information, such as `key=value` pairs. With `parse-kv`, you can extract the values into separate columns to make them easier to query, filter, and analyze. + +This operator is useful in scenarios where logs, traces, or security events contain metadata encoded as key-value pairs. Instead of manually splitting strings, you can use `parse-kv` to transform the data into a structured format. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk, you often use the `kv` or `extract` commands to parse key-value pairs from raw log data. In APL, you achieve similar functionality with the `parse-kv` operator. The difference is that `parse-kv` explicitly lets you define which keys to extract and what delimiters to use. + + +```sql Splunk example +... | kv pairdelim=";" kvdelim="=" keys="key1,key2,key3" +```` + +```kusto APL equivalent +datatable(data:string) +[ + 'key1=a;key2=b;key3=c' +] +| parse-kv data as (key1, key2, key3) with (pair_delimiter=';', kv_delimiter='=') +``` + + + + + + +ANSI SQL does not have a direct equivalent of `parse-kv`. Typically, you would use string functions such as `SUBSTRING` or `SPLIT_PART` to manually extract key-value pairs. In APL, `parse-kv` simplifies this process by automatically extracting multiple keys in one step. + + +```sql SQL example +SELECT + SUBSTRING_INDEX(SUBSTRING_INDEX(data, ';', 1), '=', -1) as key1, + SUBSTRING_INDEX(SUBSTRING_INDEX(data, ';', 2), '=', -1) as key2, + SUBSTRING_INDEX(SUBSTRING_INDEX(data, ';', 3), '=', -1) as key3 +FROM logs; +``` + +```kusto APL equivalent +datatable(data:string) +[ + 'key1=a;key2=b;key3=c' +] +| parse-kv data as (key1, key2, key3) with (pair_delimiter=';', kv_delimiter='=') +``` + + + + + + +## Usage + +### Syntax + +```kusto +parse-kv Expression as (KeysList) with (pair_delimiter = PairDelimiter, kv_delimiter = KvDelimiter [, Options...]) +``` + +### Parameters + +| Parameter | Description | +| ---------------- | ------------------------------------------------------------------------------- | +| `Expression` | The string expression that contains the key-value pairs. | +| `KeysList` | A list of keys to extract into separate columns. | +| `PairDelimiter` | A character or string that separates key-value pairs (for example, `;` or `,`). | +| `KvDelimiter` | A character or string that separates keys and values (for example, `=` or `:`). | +| `Options` | Additional parsing options, such as case sensitivity. | + +### Returns + +A dataset where each specified key is extracted into its own column with the corresponding value. If a key is missing in the original string, the column is empty for that row. + +## Use case example + +When analyzing HTTP logs, you might encounter a field where request metadata is encoded as key-value pairs. You can extract values like status and duration for easier analysis. + +**Query** + +```kusto +['sample-http-logs'] +| parse-kv kvdata as (status, req_duration_ms) with (pair_delimiter=';', kv_delimiter='=') +| project _time, status, req_duration_ms, method, uri +``` + +**Output** + +| _time | status | req_duration_ms | method | uri | +| -------------------- | ------ | ----------------- | ------ | -------- | +| 2024-05-01T10:00:00Z | 200 | 120 | GET | /home | +| 2024-05-01T10:01:00Z | 404 | 35 | GET | /missing | + +This query extracts status and request duration from a concatenated field and projects them alongside other useful fields. + +## List of related operators + +- [extend](/apl/tabular-operators/extend-operator): Adds calculated columns. Use when parsing is not required but you want to create new derived columns. +- [parse](/apl/tabular-operators/parse-operator): Extracts values from a string expression without filtering out non-matching rows. Use when you want to keep all rows, including those that fail to parse. +- [project](/apl/tabular-operators/project-operator): Selects and computes columns without parsing. Use when you want to transform data rather than extract values. +- [where](/apl/tabular-operators/where-operator): Filters rows based on conditions. Use alongside parsing functions if you want more control over filtering logic. diff --git a/apl/tabular-operators/parse-where.mdx b/apl/tabular-operators/parse-where.mdx new file mode 100644 index 00000000..120d4879 --- /dev/null +++ b/apl/tabular-operators/parse-where.mdx @@ -0,0 +1,112 @@ +--- +title: parse-where +description: 'This page explains how to use the parse-where operator in APL.' +--- + +The `parse-where` operator lets you extract values from a string expression based on a pattern and at the same time filter out rows that don’t match the pattern. This operator is useful when you want to ensure that your results contain only rows where the parsing succeeds, reducing the need for an additional filtering step. + +You can use `parse-where` when working with logs or event data that follow a known structure but may contain noise or irrelevant lines. For example, you can parse request logs to extract structured information like HTTP method, status code, or error messages, and automatically discard any rows that don’t match the format. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk SPL, you use the `rex` command with the `max_match=1` option to extract fields and filter out non-matching events. In APL, `parse-where` provides the same functionality in a more direct way. Rows that do not match the pattern are automatically excluded. + + +```sql Splunk example +... | rex field=log_line "\[(?\w+)\] (?.+)" +```` + +```kusto APL equivalent +datatable(log_line:string) +[ + '[INFO] Service started', + 'invalid line' +] +| parse-where log_line with '[', level:string, '] ', message:string +``` + + + + + + +ANSI SQL does not have a direct equivalent to `parse-where`. You often use `LIKE` or `REGEXP` functions to test string patterns and then combine them with `CASE` expressions to extract substrings. In APL, `parse-where` simplifies this by combining extraction and filtering into one operator. + + +```sql SQL example +SELECT + REGEXP_SUBSTR(log_line, '\\[(\\w+)\\]', 1, 1) AS level, + REGEXP_SUBSTR(log_line, '\\] (.+)', 1, 1) AS message +FROM logs +WHERE log_line REGEXP '\\[(\\w+)\\] (.+)'; +``` + +```kusto APL equivalent +datatable(log_line:string) +[ + '[INFO] Service started', + 'invalid line' +] +| parse-where log_line with '[', level:string, '] ', message:string +``` + + + + + + +## Usage + +### Syntax + +```kusto +parse-where [kind=kind [flags=regexFlags]] expression with [*] stringConstant columnName [:columnType] [*] +``` + +### Parameters + +| Parameter | Description | +| ---------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| `kind` | (Optional) Specifies the parsing method. The default is `simple`. You can also specify `regex` for regular expression parsing. | +| `flags` | (Optional) Regex flags to control the behavior of pattern matching. Used only with `kind=regex`. | +| `expression` | The string expression to parse. | +| `stringConstant` | The constant parts of the pattern that must match exactly. | +| `columnName` | The name of the column where the extracted value is stored. | +| `columnType` | (Optional) The type of the extracted value (for example, `string`, `int`, `real`). | + +### Returns + +The operator returns a table with the original columns and the newly extracted columns. Rows that do not match the parsing pattern are removed. + +## Use case example + +You want to extract the HTTP method and status code from request logs while ignoring rows that don’t follow the expected format. + +**Query** + +```kusto +['http-logs'] +| parse-where uri with '/' method:string '/' * 'status=' status:string +| project _time, method, status, uri +``` + +**Output** + +| _time | method | status | uri | +| -------------------- | ------ | ------ | --------------------------- | +| 2025-09-01T12:00:00Z | GET | 200 | /GET/api/items?status=200 | +| 2025-09-01T12:00:05Z | POST | 500 | /POST/api/orders?status=500 | + +This query extracts the method and status from the `uri` field and discards rows where the `uri` does not match the pattern. + +## List of related operators + +- [extend](/apl/tabular-operators/extend-operator): Adds calculated columns. Use when parsing is not required but you want to create new derived columns. +- [parse](/apl/tabular-operators/parse-operator): Extracts values from a string expression without filtering out non-matching rows. Use when you want to keep all rows, including those that fail to parse. +- [project](/apl/tabular-operators/project-operator): Selects and computes columns without parsing. Use when you want to transform data rather than extract values. +- [where](/apl/tabular-operators/where-operator): Filters rows based on conditions. Use alongside parsing functions if you want more control over filtering logic. diff --git a/apl/tabular-operators/project-rename.mdx b/apl/tabular-operators/project-rename.mdx new file mode 100644 index 00000000..d1e4251c --- /dev/null +++ b/apl/tabular-operators/project-rename.mdx @@ -0,0 +1,154 @@ +--- +title: project-rename +description: 'This page explains how to use the project-rename operator in APL.' +--- + +The `project-rename` operator in APL lets you rename columns in a dataset while keeping all existing rows intact. You can use it when you want to make column names clearer, align them with naming conventions, or prepare data for downstream processing. Unlike `project`, which also controls which columns appear in the result, `project-rename` only changes the names of selected columns and keeps the full set of columns in the dataset. + +You find this operator useful when: +- You want to standardize field names across multiple queries. +- You want to replace long or inconsistent column names with simpler ones. +- You want to improve query readability without altering the underlying data. + +## For users of other query languages + +If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. + + + + +In Splunk SPL, renaming fields uses the `rename` command. The `project-rename` operator in APL works in a similar way. Both let you map existing fields to new names without altering the dataset content. + + +```sql Splunk example +... | rename uri AS url, status AS http_status +```` + +```kusto APL equivalent +['sample-http-logs'] +| project-rename url = uri, http_status = status +``` + + + + + + +In ANSI SQL, renaming columns is done with `AS` in a `SELECT` statement. In APL, `project-rename` is the closest equivalent, but unlike SQL, it preserves all columns by default while renaming only the specified ones. + + +```sql SQL example +SELECT uri AS url, status AS http_status, method, id +FROM sample_http_logs; +``` + +```kusto APL equivalent +['sample-http-logs'] +| project-rename url = uri, http_status = status +``` + + + + + + +## Usage + +### Syntax + +```kusto +Table +| project-rename NewName1 = OldName1, NewName2 = OldName2, ... +``` + +### Parameters + +| Name | Type | Description | +| --------- | ------ | --------------------------------------- | +| `NewName` | string | The new column name you want to assign. | +| `OldName` | string | The existing column name to rename. | + +### Returns + +A dataset with the same rows and columns as the input, except that the specified columns have new names. + +## Use case examples + + + + +When analyzing HTTP logs, you might want to rename fields to shorter or more descriptive names before creating dashboards or reports. + +**Query** + +```kusto +['sample-http-logs'] +| project-rename city = ['geo.city'], country = ['geo.country'] +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20project-rename%20city%20%3D%20%5B'geo.city'%5D%2C%20country%20%3D%20%5B'geo.country'%5D%22%7D) + +**Output** + +| _time | req_duration_ms | id | status | uri | method | city | country | +| -------------------- | ----------------- | ----- | ------ | ------ | ------ | ------ | ------- | +| 2025-09-01T10:00:00Z | 120 | user1 | 200 | /home | GET | Paris | FR | +| 2025-09-01T10:01:00Z | 85 | user2 | 404 | /about | GET | Berlin | DE | + +This query renames the `geo.city` and `geo.country` fields to `city` and `country` for easier use in queries. + + + + +When inspecting distributed traces, you can rename service-related fields to match your internal naming conventions. + +**Query** + +```kusto +['otel-demo-traces'] +| project-rename service = ['service.name'] +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D%20%7C%20project-rename%20service%20%3D%20%5B'service.name'%5D%22%7D) + +**Output** + +| _time | duration | span_id | trace_id | service | kind | +| -------------------- | ------------ | -------- | --------- | --------------- | ------ | +| 2025-09-01T09:55:00Z | 00:00:01.200 | abc123 | trace789 | frontend | server | +| 2025-09-01T09:56:00Z | 00:00:00.450 | def456 | trace790 | checkout | client | + +This query renames `service.name` to `service`, making it shorter for downstream filtering. + + + + +For security-related HTTP log analysis, you can rename status and URI fields to match existing security dashboards. + +**Query** + +```kusto +['sample-http-logs'] +| project-rename http_status = status, url = uri +``` + +[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20project-rename%20http_status%20%3D%20status%2C%20url%20%3D%20uri%22%7D) + +**Output** + +| _time | req_duration_ms | id | http_status | url | method | geo.city | geo.country | +| -------------------- | ----------------- | ----- | ------------ | ------ | ------ | ------------- | ---------------- | +| 2025-09-01T11:00:00Z | 150 | user5 | 403 | /admin | POST | Madrid | ES | +| 2025-09-01T11:02:00Z | 200 | user6 | 500 | /login | POST | Rome | IT | + +This query renames `status` to `http_status` and `uri` to `url`, making the output align with security alerting systems. + + + + +## List of related operators + +- [extend](/apl/tabular-operators/extend-operator): Creates new calculated columns. Use it when you want to add columns rather than rename existing ones. +- [project](/apl/tabular-operators/project-operator): Lets you select and rename columns at the same time. Use it when you want to control which columns appear in the result. +- [project-away](/apl/tabular-operators/project-away-operator): Removes specific columns from the dataset. Use it when you want to drop columns rather than rename them. +- [summarize](/apl/tabular-operators/summarize-operator): Aggregates data into groups. Use it when you want to compute metrics rather than adjust column names. diff --git a/docs.json b/docs.json index daf5e5a2..5b99413c 100644 --- a/docs.json +++ b/docs.json @@ -468,11 +468,16 @@ "apl/tabular-operators/join-operator", "apl/tabular-operators/limit-operator", "apl/tabular-operators/lookup-operator", + "apl/tabular-operators/make-series", + "apl/tabular-operators/mv-expand", "apl/tabular-operators/order-operator", "apl/tabular-operators/parse-operator", + "apl/tabular-operators/parse-kv", + "apl/tabular-operators/parse-where", "apl/tabular-operators/project-operator", "apl/tabular-operators/project-away-operator", "apl/tabular-operators/project-keep-operator", + "apl/tabular-operators/project-rename", "apl/tabular-operators/project-reorder-operator", "apl/tabular-operators/redact-operator", "apl/tabular-operators/sample-operator", diff --git a/legal/acceptable-use-policy.mdx b/legal/acceptable-use-policy.mdx index 7ffc9cd4..71cbc096 100644 --- a/legal/acceptable-use-policy.mdx +++ b/legal/acceptable-use-policy.mdx @@ -33,7 +33,7 @@ You may not use the Axiom Service to violate the security or integrity of the Ax * Any attempt to test the vulnerability of a network or system, probe or scan or to breach security or authentication measures without permission. * Any attempt to interfere with, disrupt or disable Service to any user, or network, including, but not limited to, via means of attempts to overload the system, crashing, mail bombing, denial of service attacks, or flooding techniques. * The forging of any TCP/IP packet header, or using a computer system or program that intentionally conceals the source of information. -* Any usage or attempted usage of the Service You are not authorized to use. +* Any usage or attempted usage of the Service You aren’t authorized to use. * Any attempt or action designed to circumvent or alter any use limitations, including but not limited to, storage restrictions or billing measurement on the Services. **Reporting Violations**