[DOCS-15452] Document regex support in Log Explorer - #39368
[DOCS-15452] Document regex support in Log Explorer#39368dd-danieltian wants to merge 17 commits into
Conversation
Preview links (active after the
|
OliviaShoup
left a comment
There was a problem hiding this comment.
thanks for fixing this!
|
/review |
There was a problem hiding this comment.
🤖 Automated review by Claude. AI-generated; verify before acting.
Reviewed the new Regex sections in extractions.md and formulas.md. Content is clear, accurate, and well-structured. One minor readability nit; otherwise looks good.
Reviewed 92a5c569108470ec866b27544a486135d33e336e — workflow run
| | `input` | The text to transform | | ||
| | `pattern` | The regex pattern to match | | ||
| | `replacement` | The regex transformation pattern, often using capture groups | | ||
| | `start` | Optional. The character index to begin matching from, counting from 0. Defaults to `0` | |
There was a problem hiding this comment.
Style: Minor readability nit — counting from 0. Defaults to \0`places two different0`s (index base vs. default value) right next to each other, which can be momentarily confusing. Consider making the indexing explicit:
| | `start` | Optional. The character index to begin matching from, counting from 0. Defaults to `0` | | |
| | `start` | Optional. The zero-based character index to begin matching from. Defaults to `0` | |
| Given this log line: | ||
|
|
||
| ```plaintext | ||
| 10.0.0.14 GET /api/v1/orders 503 | ||
| ``` | ||
|
|
||
| this pattern: | ||
|
|
||
| ```plaintext | ||
| (?<ip>\S+) (?<method>\S+) (?<path>\S+) (?<status>\d+) | ||
| ``` | ||
|
|
||
| produces four fields you can filter, group, and sort by: | ||
|
|
||
| | Field | Value | | ||
| |---|---| | ||
| | `ip` | `10.0.0.14` | | ||
| | `method` | `GET` | | ||
| | `path` | `/api/v1/orders` | | ||
| | `status` | `503` | |
There was a problem hiding this comment.
| Given this log line: | |
| ```plaintext | |
| 10.0.0.14 GET /api/v1/orders 503 | |
| ``` | |
| this pattern: | |
| ```plaintext | |
| (?<ip>\S+) (?<method>\S+) (?<path>\S+) (?<status>\d+) | |
| ``` | |
| produces four fields you can filter, group, and sort by: | |
| | Field | Value | | |
| |---|---| | |
| | `ip` | `10.0.0.14` | | |
| | `method` | `GET` | | |
| | `path` | `/api/v1/orders` | | |
| | `status` | `503` | | |
| **Log line**: | |
| ```plaintext | |
| 10.0.0.14 GET /api/v1/orders 503 | |
| ``` | |
| **Regex pattern**: | |
| ```plaintext | |
| (?<ip>\S+) (?<method>\S+) (?<path>\S+) (?<status>\d+) | |
| ``` | |
| This produces four fields you can filter, group, and sort by: | |
| - `#ip = 10.0.0.14` | |
| - `#method = GET` | |
| - `#path = /api/v1/orders` | |
| - `#status = 503` |
These edits split this section up into cleaner reference + example sections. However, these example fields don't relate to the group rules listed above. We should expand on this example to explain how rules work, and/or create an anti-example to show what invalid patterns would look like.
Split the rules list and example into separate Capture group rules and Example headings, matching the Matchers/Filters/Example shape used by the Grok half of the page. Move the string-typing note up beside the related extraction/formula ordering note. Conform the example to the bold-label format already used by the Grok example. Also fix 'lets you to extract' and link the regex term inline in Overview.
The previous wording attributed both the doubled backslash in \\d and the one in \\$ to the same cause. The $ case stacks two concerns: escaping the character's special meaning, then escaping that backslash for the string literal.
Drop the Regex section opener that restated the Overview almost verbatim. State that unlisted constructs are unsupported, and note that extraction patterns take a single backslash unlike formula arguments. Rework the example to use (?:v\d+) so it exercises the non-capturing group rule instead of only using trivially valid names.
| - Use `(?<name>…)`, not `(…)`. | ||
| - Use `(?:…)` to group without creating a field. | ||
| - Each name must be unique. The name becomes the name of the extracted field. | ||
| - Each name must start with a letter, and contain only letters and digits (`[A-Za-z][A-Za-z0-9]*`). Names like `client_ip`, `http.status`, or `client-ip` are not valid capture group names. |
There was a problem hiding this comment.
Are underscores (_) rejected in capture group names? This would imply we're using a custom validator rather than a stock regex engine. If it's not supported, this section would benefit from adding a table of anti-patterns.
There was a problem hiding this comment.
yes, underscores are rejected in capture group names, but that's different from the patterns - the regex pattern can match against underscores and special characters, it just can't be in a named column, so it can't be in the name of a capture group. is this still what you're referring to?
There was a problem hiding this comment.
Thanks for confirming; I wasn't sure about group names on the Datadog side.
| | Named capture groups | `(?<status>\d+)` | Captures the match under a name. Required for extraction | | ||
| | Quantifiers | `a*`, `a+`, `a?`, `a{2,4}` | Repetition: zero or more, one or more, optional, or a bounded range. Matches as much as possible | | ||
| | Lazy quantifiers | `.*?end` | The same repetition, but matching as little as possible | | ||
| | Anchors | `^ERROR`, `timeout$` | The start or the end of the value | |
There was a problem hiding this comment.
For a multi-line log message such as a stack trace, do ^ and $ match the start and end of the whole value, or of each line?
There was a problem hiding this comment.
They will match the start and end of the whole value, should I add that?
There was a problem hiding this comment.
Added some clarifying notes for being able to match any character "." to newline, and to match the anchors with each line. Should I clarify in line 120 here as well that it doesn't match by line?
There was a problem hiding this comment.
Updated this line to By default, the start or the end of the whole value. Not necessary to add anything else.
The example showed valid output only. It did not connect back to the capture group rules above it. This adds a short check of the example pattern against each rule, and a table of invalid patterns that break each rule.
Updated the rules for capture group names to include digits.
The dot metacharacter does not match newlines, and anchors match the whole value, not each line, unless the (?s) or (?m) inline flag is set. Document both, since a value can span multiple lines. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
What is this PR?
Reopens #39286, which was accidentally reverted by #39345 shortly after merging. This restores the original content unchanged: a new page documenting regex-based field extraction (named capture groups) in the Log Explorer, plus the
regexp_like/regexp_replacecalculated field functions, cross-linked fromextractions.mdandformulas.md.Verification
Reapplied by reverting the revert commit (
git revert e2bb027e8b) on top of currentmaster, which merged with no conflicts. Diffed the reapplied changes against the original merged PR diff — byte-for-byte identical.Limitations
None beyond the original PR's scope. Fixes DOCS-15452.