Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 72 additions & 10 deletions content/en/logs/log_configuration/parsing.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,11 +423,25 @@ Note that "id" is an integer and not a string.
MyParsingRule (%{integer:user.id}|%{word:user.firstname}) connected on %{date("MM/dd/yyyy"):connect_date}
```

**Results**:

{{< img src="logs/processing/parsing/parsing_example_4.png" alt="Parsing example 4" style="width:80%;" >}}

{{< img src="logs/processing/parsing/parsing_example_4_bis.png" alt="Parsing example 4 bis" style="width:80%;" >}}
**Results**:<br>
`%{integer:user.id}`
```json
{
"user": {
"id": 12345
},
"connect_date": 1510099200000
}
```
`%{word:user.firstname}`
```json
{
"user": {
"firstname": "john"
},
"connect_date": 1510099200000
}
```

### Optional attribute

Expand All @@ -437,6 +451,7 @@ Some logs contain values that only appear part of the time. In this case, make a

```text
john 1234 connected on 11/08/2017
john connected on 11/08/2017
```

**Rule**:
Expand All @@ -447,9 +462,28 @@ MyParsingRule %{word:user.firstname} (%{integer:user.id} )?connected on %{date("

**Note**: A rule will not match if you include a space after the first word in the optional section.

{{< img src="logs/processing/parsing/parsing_example_5.png" alt="Parsing example 5" style="width:80%;" >}}
**Result**:<br>
`(%{integer:user.id} )?`

{{< img src="logs/processing/parsing/parsing_example_5_bis.png" alt="Parsing example 5 bis" style="width:80%;" >}}
```json
{
"user": {
"firstname": "john",
"id": 1234
},
"connect_date": 1510099200000
}
```

`%{word:user.firstname} (%{integer:user.id} )?`
```json
{
"user": {
"firstname": "john",
},
"connect_date": 1510099200000
}
```

### Nested JSON

Expand All @@ -467,7 +501,17 @@ Sep 06 09:13:38 vagrant program[123]: server.1 {"method":"GET", "status_code":20
parsing_rule %{date("MMM dd HH:mm:ss"):timestamp} %{word:vm} %{word:app}\[%{number:logger.thread_id}\]: %{notSpace:server} %{data::json}
```

{{< img src="logs/processing/parsing/nested_json.png" alt="Nested JSON Parsing example" style="width:80%;" >}}
**Result**:
```json
{
"timestamp": 1567761218000,
"vm": "vagrant",
"app": "program",
"logger": {
"thread_id": 123
}
}
```

### Regex

Expand All @@ -483,7 +527,15 @@ john_1a2b3c4 connected on 11/08/2017
MyParsingRule %{regex("[a-z]*"):user.firstname}_%{regex("[a-zA-Z0-9]*"):user.id} .*
```

{{< img src="logs/processing/parsing/regex_parsing.png" alt="Parsing example 6" style="width:80%;" >}}
**Result**:
```json
{
"user": {
"firstname": "john",
"id": "1a2b3c4"
}
}
```

### List to array

Expand All @@ -501,7 +553,17 @@ Users [John, Oliver, Marc, Tom] have been added to the database
myParsingRule Users %{data:users:array("[]",",")} have been added to the database
```

{{< img src="logs/processing/parsing/array_parsing.png" alt="Parsing example 6" style="width:80%;" >}}
**Result**:
```json
{
"users": [
"John",
" Oliver",
" Marc",
" Tom"
]
}
```

**Log**:

Expand Down
8 changes: 2 additions & 6 deletions content/en/logs/log_configuration/processors.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ In [log configuration settings][1], you can configure processors such as the [Gr

Create custom grok rules to parse the full message or a specific attribute of your raw event. As a best practice, limit your grok parser to 10 parsing rules. For more information on Grok syntax and parsing rules, see [Parsing][10].

{{< img src="logs/log_configuration/processor/grok_parser.png" alt="Grok Parser" style="width:80%;" >}}
{{< img src="/logs/processing/processors/define_parsing_rules_syntax_suggestions.png" alt="Grok parser syntax suggestions in the UI" style="width:90%;" >}}

{{< tabs >}}
{{% tab "UI" %}}
Expand Down Expand Up @@ -154,8 +154,6 @@ Use the [Datadog Log Pipeline API endpoint][1] with the following log date remap

Use the status remapper processor to assign attributes as an official status to your logs. For example, add a log severity level to your logs with the status remapper.

{{< img src="logs/processing/processors/log_post_severity_bis.png" alt="Log severity after remapping" style="width:40%;" >}}

Each incoming status value is mapped as follows:

* Integers from 0 to 7 map to the [Syslog severity standards][4]
Expand Down Expand Up @@ -289,9 +287,7 @@ Use the [Datadog Log Pipeline API endpoint][1] with the following log message re

## Remapper

The remapper processor remaps any source attribute(s) or tags to another target attribute or tag. For example, remap `user` by `firstname` to target your logs in the Log Explorer:

{{< img src="logs/processing/processors/attribute_post_remapping.png" alt="Attribute after remapping" style="width:60%;">}}
The remapper processor remaps any source attribute(s) or tags to another target attribute or tag. For example, remap `user` by `firstname` to target your logs in the Log Explorer.

Constraints on the tag/attribute name are explained in the [attributes and tags documentation][5]. Some additional constraints, applied as `:` or `,`, are not allowed in the target tag/attribute name.

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading