Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cherry-pick #17376 to 7.x: [Filebeat] Improve ECS categorization field mappings for mssql module #17417

Merged
merged 2 commits into from
Apr 2, 2020
Merged

Cherry-pick #17376 to 7.x: [Filebeat] Improve ECS categorization field mappings for mssql module #17417

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4074784
[Filebeat] Improve ECS categorization field mappings for mssql module…
leehinman Apr 1, 2020
dc0b2eb
Merge branch '7.x' into backport_17376_7.x
leehinman Apr 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add source field in k8s events {pull}17209[17209]
- Improve AWS cloudtrail field mappings {issue}16086[16086] {issue}16110[16110] {pull}17155[17155]
- Move azure-eventhub input to GA. {issue}15671[15671] {pull}17313[17313]
- Improve ECS categorization field mappings for mssql module. {issue}16171[16171] {pull}17376[17376]

*Heartbeat*

Expand Down
58 changes: 0 additions & 58 deletions x-pack/filebeat/module/mssql/log/ingest/pipeline.json
View file
Open in desktop

This file was deleted.

50 changes: 50 additions & 0 deletions x-pack/filebeat/module/mssql/log/ingest/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
description: Pipeline to parse MSSQL logs
processors:
- grok:
field: message
patterns:
- '%{MSSQL_DATE:date} %{DATA:mssql.log.origin} [ ]*%{GREEDYDATA:msg_temp}'
pattern_definitions:
MSSQL_DATE: '%{DATA} %{DATA}'
- date:
if: ctx.event.timezone == null
field: date
formats:
- yyyy-MM-dd HH:mm:ss.SS
on_failure:
- append:
field: error.message
value: '{{ _ingest.on_failure_message }}'
- date:
if: ctx.event.timezone != null
field: date
formats:
- yyyy-MM-dd HH:mm:ss.SS
timezone: '{{ event.timezone }}'
on_failure:
- append:
field: error.message
value: '{{ _ingest.on_failure_message }}'
- remove:
field: date
ignore_missing: true
- rename:
field: message
target_field: log.original
- rename:
field: msg_temp
target_field: message
ignore_missing: true
- set:
field: event.kind
value: event
- append:
field: event.category
value: database
- append:
field: event.type
value: info
on_failure:
- set:
field: error.message
value: '{{ _ingest.on_failure_message }}'
2 changes: 1 addition & 1 deletion x-pack/filebeat/module/mssql/log/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ var:
os.linux:
- /var/opt/mssql/log/error*

ingest_pipeline: ingest/pipeline.json
ingest_pipeline: ingest/pipeline.yml
input: config/config.yml
105 changes: 105 additions & 0 deletions x-pack/filebeat/module/mssql/log/test/test.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
[
{
"@timestamp": "2019-05-03T09:01:09.990-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.flags": [
Expand All @@ -17,9 +24,16 @@
},
{
"@timestamp": "2019-05-03T09:01:09.990-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 226,
Expand All @@ -30,9 +44,16 @@
},
{
"@timestamp": "2019-05-03T09:01:09.990-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 282,
Expand All @@ -43,9 +64,16 @@
},
{
"@timestamp": "2019-05-03T09:01:09.990-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 344,
Expand All @@ -56,9 +84,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 400,
Expand All @@ -69,9 +104,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 462,
Expand All @@ -82,9 +124,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.flags": [
Expand All @@ -98,9 +147,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 734,
Expand All @@ -111,9 +167,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 1011,
Expand All @@ -124,9 +187,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 1166,
Expand All @@ -137,9 +207,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 1289,
Expand All @@ -150,9 +227,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.010-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 1373,
Expand All @@ -163,9 +247,16 @@
},
{
"@timestamp": "2019-05-03T09:01:10.200-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 1435,
Expand All @@ -176,9 +267,16 @@
},
{
"@timestamp": "2019-05-03T09:01:11.930-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 1528,
Expand All @@ -189,9 +287,16 @@
},
{
"@timestamp": "2019-05-03T09:01:12.030-02:00",
"event.category": [
"database"
],
"event.dataset": "mssql.log",
"event.kind": "event",
"event.module": "mssql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "log",
"input.type": "log",
"log.offset": 1599,
Expand Down