Skip to content

[DOCS-15452] Document regex support in Log Explorer - #39368

Open
dd-danieltian wants to merge 17 commits into
masterfrom
daniel-tian/document-regex-support-reopen
Open

[DOCS-15452] Document regex support in Log Explorer#39368
dd-danieltian wants to merge 17 commits into
masterfrom
daniel-tian/document-regex-support-reopen

Conversation

@dd-danieltian

@dd-danieltian dd-danieltian commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

What is this PR?

Reopens #39286, which was accidentally reverted by #39345 shortly after merging. This restores the original content unchanged: a new page documenting regex-based field extraction (named capture groups) in the Log Explorer, plus the regexp_like/regexp_replace calculated field functions, cross-linked from extractions.md and formulas.md.

Verification

Reapplied by reverting the revert commit (git revert e2bb027e8b) on top of current master, which merged with no conflicts. Diffed the reapplied changes against the original merged PR diff — byte-for-byte identical.

Limitations

None beyond the original PR's scope. Fixes DOCS-15452.

@dd-danieltian
dd-danieltian requested a review from a team as a code owner August 20, 2026 17:40
@github-actions

Copy link
Copy Markdown
Contributor

@OliviaShoup OliviaShoup left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for fixing this!

@maxvp maxvp added the Do Not Merge Just do not merge this PR :) label Aug 20, 2026
@maxvp maxvp self-assigned this Aug 20, 2026
@maxvp

maxvp commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review by Claude. AI-generated; verify before acting.

Reviewed the new Regex sections in extractions.md and formulas.md. Content is clear, accurate, and well-structured. One minor readability nit; otherwise looks good.

Reviewed 92a5c569108470ec866b27544a486135d33e336eworkflow run

| `input` | The text to transform |
| `pattern` | The regex pattern to match |
| `replacement` | The regex transformation pattern, often using capture groups |
| `start` | Optional. The character index to begin matching from, counting from 0. Defaults to `0` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style: Minor readability nit — counting from 0. Defaults to \0`places two different0`s (index base vs. default value) right next to each other, which can be momentarily confusing. Consider making the indexing explicit:

Suggested change
| `start` | Optional. The character index to begin matching from, counting from 0. Defaults to `0` |
| `start` | Optional. The zero-based character index to begin matching from. Defaults to `0` |

Comment thread hugo/content/en/logs/explorer/calculated_fields/extractions.md Outdated
Comment thread hugo/content/en/logs/explorer/calculated_fields/extractions.md Outdated
Comment thread hugo/content/en/logs/explorer/calculated_fields/extractions.md Outdated
Comment thread hugo/content/en/logs/explorer/calculated_fields/extractions.md Outdated
Comment thread hugo/content/en/logs/explorer/calculated_fields/formulas.md Outdated
Comment thread hugo/content/en/logs/explorer/calculated_fields/extractions.md Outdated
Comment on lines +132 to +151
Given this log line:

```plaintext
10.0.0.14 GET /api/v1/orders 503
```

this pattern:

```plaintext
(?<ip>\S+) (?<method>\S+) (?<path>\S+) (?<status>\d+)
```

produces four fields you can filter, group, and sort by:

| Field | Value |
|---|---|
| `ip` | `10.0.0.14` |
| `method` | `GET` |
| `path` | `/api/v1/orders` |
| `status` | `503` |

@maxvp maxvp Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Given this log line:
```plaintext
10.0.0.14 GET /api/v1/orders 503
```
this pattern:
```plaintext
(?<ip>\S+) (?<method>\S+) (?<path>\S+) (?<status>\d+)
```
produces four fields you can filter, group, and sort by:
| Field | Value |
|---|---|
| `ip` | `10.0.0.14` |
| `method` | `GET` |
| `path` | `/api/v1/orders` |
| `status` | `503` |
**Log line**:
```plaintext
10.0.0.14 GET /api/v1/orders 503
```
**Regex pattern**:
```plaintext
(?<ip>\S+) (?<method>\S+) (?<path>\S+) (?<status>\d+)
```
This produces four fields you can filter, group, and sort by:
- `#ip = 10.0.0.14`
- `#method = GET`
- `#path = /api/v1/orders`
- `#status = 503`

These edits split this section up into cleaner reference + example sections. However, these example fields don't relate to the group rules listed above. We should expand on this example to explain how rules work, and/or create an anti-example to show what invalid patterns would look like.

maxvp added 5 commits August 21, 2026 16:37
Split the rules list and example into separate Capture group rules and
Example headings, matching the Matchers/Filters/Example shape used by the
Grok half of the page. Move the string-typing note up beside the related
extraction/formula ordering note. Conform the example to the bold-label
format already used by the Grok example.

Also fix 'lets you to extract' and link the regex term inline in Overview.
The previous wording attributed both the doubled backslash in \\d and the
one in \\$ to the same cause. The $ case stacks two concerns: escaping the
character's special meaning, then escaping that backslash for the string
literal.
Drop the Regex section opener that restated the Overview almost verbatim.
State that unlisted constructs are unsupported, and note that extraction
patterns take a single backslash unlike formula arguments.

Rework the example to use (?:v\d+) so it exercises the non-capturing group
rule instead of only using trivially valid names.
Comment thread hugo/content/en/logs/explorer/calculated_fields/extractions.md Outdated
- Use `(?<name>…)`, not `(…)`.
- Use `(?:…)` to group without creating a field.
- Each name must be unique. The name becomes the name of the extracted field.
- Each name must start with a letter, and contain only letters and digits (`[A-Za-z][A-Za-z0-9]*`). Names like `client_ip`, `http.status`, or `client-ip` are not valid capture group names.

@maxvp maxvp Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are underscores (_) rejected in capture group names? This would imply we're using a custom validator rather than a stock regex engine. If it's not supported, this section would benefit from adding a table of anti-patterns.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, underscores are rejected in capture group names, but that's different from the patterns - the regex pattern can match against underscores and special characters, it just can't be in a named column, so it can't be in the name of a capture group. is this still what you're referring to?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming; I wasn't sure about group names on the Datadog side.

| Named capture groups | `(?<status>\d+)` | Captures the match under a name. Required for extraction |
| Quantifiers | `a*`, `a+`, `a?`, `a{2,4}` | Repetition: zero or more, one or more, optional, or a bounded range. Matches as much as possible |
| Lazy quantifiers | `.*?end` | The same repetition, but matching as little as possible |
| Anchors | `^ERROR`, `timeout$` | The start or the end of the value |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a multi-line log message such as a stack trace, do ^ and $ match the start and end of the whole value, or of each line?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They will match the start and end of the whole value, should I add that?

@dd-danieltian dd-danieltian Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some clarifying notes for being able to match any character "." to newline, and to match the anchors with each line. Should I clarify in line 120 here as well that it doesn't match by line?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated this line to By default, the start or the end of the whole value. Not necessary to add anything else.

The example showed valid output only. It did not connect back to the
capture group rules above it. This adds a short check of the example
pattern against each rule, and a table of invalid patterns that break
each rule.
Updated the rules for capture group names to include digits.
dd-danieltian and others added 4 commits August 26, 2026 18:06
The dot metacharacter does not match newlines, and anchors match the
whole value, not each line, unless the (?s) or (?m) inline flag is
set. Document both, since a value can span multiple lines.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@maxvp maxvp removed the Do Not Merge Just do not merge this PR :) label Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants