Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IIS] Add regex and tests for Exchange logs #7559

Merged
merged 16 commits into from Oct 30, 2023
Merged

[IIS] Add regex and tests for Exchange logs #7559

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
c986be8
Initial commit
LaZyDK Aug 28, 2023
bb49d5b
Update PR link
LaZyDK Aug 28, 2023
fb04cec
bump version
LaZyDK Sep 4, 2023
5f45934
Merge branch 'main' into iis10_exchange
LaZyDK Sep 4, 2023
18fec3e
fix changelog PR link
LaZyDK Sep 4, 2023
2472891
Run system tests
LaZyDK Sep 5, 2023
48a0f76
Update custom logs documentation
LaZyDK Sep 5, 2023
fd9207c
Merge branch 'main' into iis10_exchange
LaZyDK Sep 22, 2023
3f3607e
Change readme Logs: custom logging
LaZyDK Oct 8, 2023
dfb874d
Update packages/iis/_dev/build/docs/README.md
LaZyDK Oct 12, 2023
c9bac20
Update packages/iis/data_stream/access/elasticsearch/ingest_pipeline/…
LaZyDK Oct 12, 2023
59d0af6
Add related.hosts and run tests
LaZyDK Oct 12, 2023
6961703
Merge branch 'main' into iis10_exchange
LaZyDK Oct 20, 2023
d9ecca0
Rearrange pipeline and remove duplicates
LaZyDK Oct 20, 2023
435c8cb
Extract user.domain
LaZyDK Oct 20, 2023
04d9306
Reintroduce server_name
LaZyDK Oct 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/iis/_dev/build/docs/README.md
View file
Open in desktop
Expand Up @@ -72,6 +72,8 @@ IIS integration offers certain field combinations shipped automatically into Ela

- Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(cookie) cs(Referer) sc-status sc-substatus sc-win32-status sc-bytes, cs-bytes time-taken

- Fields: date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes, cs-bytes time-taken

{{event "access"}}

The fields reported are:
Expand Down
9 changes: 7 additions & 2 deletions packages/iis/changelog.yml
View file
Open in desktop
@@ -1,5 +1,10 @@
# newer versions go on top
- version: 1.16.0
- version: "1.17.0"
changes:
- description: Add regex for Exchange logs
type: enhancement
link: https://github.com/elastic/integrations/pull/7559
- version: "1.16.0"
changes:
- description: Update the package format_version to 3.0.0.
type: enhancement
Expand All @@ -13,7 +18,7 @@
changes:
- description: Add ability to set condition for logs and metrics.
type: enhancement
link: https://github.com/elastic/integrations/pull/7372
link: https://github.com/elastic/integrations/pull/7373
- version: "1.14.0"
changes:
- description: Update document with supported ingest patterns for access_log
Expand Down
45 changes: 36 additions & 9 deletions packages/iis/data_stream/access/_dev/test/pipeline/test-iis-8.log-expected.json
View file
Open in desktop
Expand Up @@ -24,6 +24,9 @@
"connection"
]
},
"host": {
"name": "se119654"
},
"http": {
"request": {
"body": {
Expand All @@ -41,15 +44,16 @@
},
"iis": {
"access": {
"server_name": "SE119654",
"site_name": "W3SVC1",
"sub_status": 4,
"win32_status": 5
}
},
"related": {
"hosts": [
"se119654"
],
"ip": [
"::1",
"::1"
]
},
Expand Down Expand Up @@ -105,6 +109,9 @@
"connection"
]
},
"host": {
"name": "se119654"
},
"http": {
"request": {
"body": {
Expand All @@ -122,15 +129,16 @@
},
"iis": {
"access": {
"server_name": "SE119654",
"site_name": "W3SVC1",
"sub_status": 4,
"win32_status": 5
}
},
"related": {
"hosts": [
"se119654"
],
"ip": [
"::1",
"::1"
]
},
Expand Down Expand Up @@ -186,6 +194,9 @@
"connection"
]
},
"host": {
"name": "se119654"
},
"http": {
"request": {
"body": {
Expand All @@ -203,15 +214,16 @@
},
"iis": {
"access": {
"server_name": "SE119654",
"site_name": "W3SVC1",
"sub_status": 4,
"win32_status": 5
}
},
"related": {
"hosts": [
"se119654"
],
"ip": [
"::1",
"::1"
]
},
Expand Down Expand Up @@ -266,6 +278,9 @@
"connection"
]
},
"host": {
"name": "se119654"
},
"http": {
"request": {
"body": {
Expand All @@ -283,13 +298,15 @@
},
"iis": {
"access": {
"server_name": "SE119654",
LaZyDK marked this conversation as resolved.
Show resolved Hide resolved
"site_name": "W3SVC1",
"sub_status": 4,
"win32_status": 5
}
},
"related": {
"hosts": [
"se119654"
],
"ip": [
"10.60.74.238",
"10.60.79.142"
Expand Down Expand Up @@ -332,6 +349,9 @@
"connection"
]
},
"host": {
"name": "se119654"
},
"http": {
"request": {
"body": {
Expand All @@ -349,13 +369,15 @@
},
"iis": {
"access": {
"server_name": "SE119654",
"site_name": "W3SVC1",
"sub_status": 4,
"win32_status": 5
}
},
"related": {
"hosts": [
"se119654"
],
"ip": [
"10.60.74.238",
"10.60.79.142"
Expand Down Expand Up @@ -412,6 +434,9 @@
"connection"
]
},
"host": {
"name": "se119654"
},
"http": {
"request": {
"body": {
Expand All @@ -429,13 +454,15 @@
},
"iis": {
"access": {
"server_name": "SE119654",
"site_name": "W3SVC1",
"sub_status": 4,
"win32_status": 5
}
},
"related": {
"hosts": [
"se119654"
],
"ip": [
"10.60.74.238",
"10.60.79.142"
Expand Down
2 changes: 0 additions & 2 deletions packages/iis/data_stream/access/_dev/test/pipeline/test-iis-access-72.log-expected.json
View file
Open in desktop
Expand Up @@ -388,7 +388,6 @@
},
"related": {
"ip": [
"10.44.0.136",
"10.44.0.136"
]
},
Expand Down Expand Up @@ -461,7 +460,6 @@
},
"related": {
"ip": [
"10.44.0.136",
"10.44.0.136"
]
},
Expand Down
19 changes: 15 additions & 4 deletions packages/iis/data_stream/access/_dev/test/pipeline/test-iis-access-75.log-expected.json
View file
Open in desktop
Expand Up @@ -4,19 +4,24 @@
"@timestamp": "2018-08-28T18:24:25.000Z",
"destination": {
"address": "10.100.220.70",
"ip": "10.100.220.70",
"port": 80
},
"ecs": {
"version": "8.5.1"
},
"event": {
"category": [
"web"
"web",
"network"
],
"duration": 792000000,
"kind": "event",
"original": "2018-08-28 18:24:25 [10.100.220.70](http://10.100.220.70) GET / - 80 - [10.100.118.31](http://10.100.118.31) Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.3;+WOW64;+Trident/7.0;+.NET4.0E;+.NET4.0C;+.NET+CLR+3.5.30729;+.NET+CLR[+2.0.50727](tel:+2050727);+.NET+CLR+3.0.30729) 404 4 2 792",
"outcome": "failure"
"outcome": "failure",
"type": [
"connection"
]
},
"http": {
"request": {
Expand All @@ -32,8 +37,15 @@
"win32_status": 2
}
},
"related": {
"ip": [
"10.100.118.31",
"10.100.220.70"
]
},
"source": {
"address": "10.100.118.31"
"address": "10.100.118.31",
"ip": "10.100.118.31"
},
"tags": [
"preserve_original_event"
Expand Down Expand Up @@ -205,7 +217,6 @@
},
"related": {
"ip": [
"2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6",
"2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6"
]
},
Expand Down
7 changes: 6 additions & 1 deletion packages/iis/data_stream/access/_dev/test/pipeline/test-iis-access.log-expected.json
View file
Open in desktop
Expand Up @@ -175,6 +175,9 @@
"connection"
]
},
"host": {
"name": "machine-name"
},
"http": {
"request": {
"body": {
Expand All @@ -192,13 +195,15 @@
},
"iis": {
"access": {
"server_name": "MACHINE-NAME",
"site_name": "W3SVC1",
"sub_status": 0,
"win32_status": 0
}
},
"related": {
"hosts": [
"machine-name"
],
"ip": [
"67.43.156.13",
"127.0.0.1"
Expand Down
6 changes: 5 additions & 1 deletion packages/iis/data_stream/access/_dev/test/pipeline/test-iis10-access.log
View file
Open in desktop
@@ -1,3 +1,7 @@
2022-05-09 17:10:04 10.119.32.8 POST /civault/Cryptology/Cryptology.svc - 443 - 10.119.0.62 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+10.0;+Win64;+x64;+Trident/7.0;+.NET4.0C;+.NET4.0E;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.30729;+.NET+CLR+3.5.30729) - [apcvwp00049.corp.acxiom.net](https://apcvwp00049.corp.acxiom.net/) 200 0 0 26
2022-05-08 01:26:01 10.119.32.8 POST /NamingServiceANSWS/ANSWS.svc - 443 - 10.119.38.250 - - itwebcert.acxiom.com 200 0 0 232
2021-06-10 23:26:57 10.44.0.136 GET / - 80 - 10.46.208.5 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+5.1;+Trident/4.0) BIGipServerYAkgvoMHHYQAGkWEWadfsadfnyAqAere=27246369425.20480.0000 itweb.acx.co 200 0 0 23
2021-06-10 23:26:57 10.44.0.136 GET / - 80 - 10.46.208.5 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+5.1;+Trident/4.0) BIGipServerYAkgvoMHHYQAGkWEWadfsadfnyAqAere=27246369425.20480.0000 itweb.acx.co 200 0 0 23
2023-08-28 07:37:00 exchange01 10.44.0.131 POST /EWS/Exchange.asmx &CorrelationID=<empty>;&cafeReqId=12d62000-56b5-0000-a9f7-a9e934ecb2d5; 443 EXAMPLE\user01 10.119.1.75 OC/16.0.5266.1000+(Skype+for+Business) - 200 0 0 52 10.10.10.7 59050
2023-08-28 08:58:45 exchange02 fe80::d9e3:e981:da2d:427b%3 POST /mapi/emsmdb/ MailboxId=df2fe8fa-492b-0000-a93b-0ce83db65e43@example.com 444 Anonymous fe80::d9e3:e981:da2d:427b%3 Microsoft+Office/16.0+(Windows+NT+6.2;+ucmapi+16.0.5404;+Pro) - 200 0 0 4 10.0.61.69,10.93.4.75 36391
2023-08-28 08:58:33 exchange02 fe80::d9e3:e981:da2d:427b%3 POST /autodiscover/autodiscover.xml &reqId=bc5918ae-b6a0-0000-9b09-bb94ce79a2fc 444 EXAMPLE\user02 fe80::d9e3:e981:da2d:427b%3 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.5404;+Pro) - 200 0 0 78 10.0.40.92,10.93.4.75 51385
2023-08-28 08:58:43 exchange03 fe80::e478:3abc:f96c:ea94%4 POST /mapi/nspi/ MailboxId=33455f09-88cb-0000-a1a5-730f900d10bd@example.com&FrontEnd=EXCHANGE03.EXAMPLE.COM&RequestId=e393c31b-3ae7-0000-ad11-0632a39901b0&ClientRequestInfo=R:{BD33627D-6B8C-4648-9740-AD9172BB0FEB}:106;RT:GetProps;CI:{DC96322E-2662-4791-BE5D-44163F6033E0}:5;CID:{45ADD272-0FE6-4A23-A58E-A9E379DF101C}&ResponseInfo=XRC:0;SC:0;RC:263040&Stage=BegR:2023-08-28T08:58:44.6445336Z;PostAR:1;PreH:1;PostH:3;EndR:3 444 Anonymous fe80::e478:3abc:f96c:ea94%4 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.5404;+Pro) - 200 0 0 3 10.1.2.61,10.93.4.75 56041