Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IIS] Add regex and tests for Exchange logs #7559

Merged
merged 16 commits into from Oct 30, 2023
Merged

[IIS] Add regex and tests for Exchange logs #7559

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
c986be8
Initial commit
LaZyDK Aug 28, 2023
bb49d5b
Update PR link
LaZyDK Aug 28, 2023
fb04cec
bump version
LaZyDK Sep 4, 2023
5f45934
Merge branch 'main' into iis10_exchange
LaZyDK Sep 4, 2023
18fec3e
fix changelog PR link
LaZyDK Sep 4, 2023
2472891
Run system tests
LaZyDK Sep 5, 2023
48a0f76
Update custom logs documentation
LaZyDK Sep 5, 2023
fd9207c
Merge branch 'main' into iis10_exchange
LaZyDK Sep 22, 2023
3f3607e
Change readme Logs: custom logging
LaZyDK Oct 8, 2023
dfb874d
Update packages/iis/_dev/build/docs/README.md
LaZyDK Oct 12, 2023
c9bac20
Update packages/iis/data_stream/access/elasticsearch/ingest_pipeline/…
LaZyDK Oct 12, 2023
59d0af6
Add related.hosts and run tests
LaZyDK Oct 12, 2023
6961703
Merge branch 'main' into iis10_exchange
LaZyDK Oct 20, 2023
d9ecca0
Rearrange pipeline and remove duplicates
LaZyDK Oct 20, 2023
435c8cb
Extract user.domain
LaZyDK Oct 20, 2023
04d9306
Reintroduce server_name
LaZyDK Oct 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/iis/_dev/build/docs/README.md
View file
Open in desktop
Expand Up @@ -72,6 +72,8 @@ IIS integration offers certain field combinations shipped automatically into Ela

- Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(cookie) cs(Referer) sc-status sc-substatus sc-win32-status sc-bytes, cs-bytes time-taken

- Fields: date time s-hostname s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes, cs-bytes time-taken
LaZyDK marked this conversation as resolved.
Show resolved Hide resolved
LaZyDK marked this conversation as resolved.
Show resolved Hide resolved

{{event "access"}}

The fields reported are:
Expand Down
7 changes: 6 additions & 1 deletion packages/iis/changelog.yml
View file
Open in desktop
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.16.0"
changes:
- description: Add regex for Exchange logs
type: enhancement
link: https://github.com/elastic/integrations/pull/7559
- version: "1.15.1"
changes:
- description: Add null check and ignore_missing check to the rename processor
Expand All @@ -8,7 +13,7 @@
changes:
- description: Add ability to set condition for logs and metrics.
type: enhancement
link: https://github.com/elastic/integrations/pull/7372
link: https://github.com/elastic/integrations/pull/7373
- version: "1.14.0"
changes:
- description: Update document with supported ingest patterns for access_log
Expand Down
6 changes: 5 additions & 1 deletion packages/iis/data_stream/access/_dev/test/pipeline/test-iis10-access.log
View file
Open in desktop
@@ -1,3 +1,7 @@
2022-05-09 17:10:04 10.119.32.8 POST /civault/Cryptology/Cryptology.svc - 443 - 10.119.0.62 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+10.0;+Win64;+x64;+Trident/7.0;+.NET4.0C;+.NET4.0E;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.30729;+.NET+CLR+3.5.30729) - [apcvwp00049.corp.acxiom.net](https://apcvwp00049.corp.acxiom.net/) 200 0 0 26
2022-05-08 01:26:01 10.119.32.8 POST /NamingServiceANSWS/ANSWS.svc - 443 - 10.119.38.250 - - itwebcert.acxiom.com 200 0 0 232
2021-06-10 23:26:57 10.44.0.136 GET / - 80 - 10.46.208.5 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+5.1;+Trident/4.0) BIGipServerYAkgvoMHHYQAGkWEWadfsadfnyAqAere=27246369425.20480.0000 itweb.acx.co 200 0 0 23
2021-06-10 23:26:57 10.44.0.136 GET / - 80 - 10.46.208.5 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+5.1;+Trident/4.0) BIGipServerYAkgvoMHHYQAGkWEWadfsadfnyAqAere=27246369425.20480.0000 itweb.acx.co 200 0 0 23
2023-08-28 07:37:00 exchange01 10.44.0.131 POST /EWS/Exchange.asmx &CorrelationID=<empty>;&cafeReqId=12d62000-56b5-0000-a9f7-a9e934ecb2d5; 443 EXAMPLE\user01 10.119.1.75 OC/16.0.5266.1000+(Skype+for+Business) - 200 0 0 52 10.10.10.7 59050
2023-08-28 08:58:45 exchange02 fe80::d9e3:e981:da2d:427b%3 POST /mapi/emsmdb/ MailboxId=df2fe8fa-492b-0000-a93b-0ce83db65e43@example.com 444 Anonymous fe80::d9e3:e981:da2d:427b%3 Microsoft+Office/16.0+(Windows+NT+6.2;+ucmapi+16.0.5404;+Pro) - 200 0 0 4 10.0.61.69,10.93.4.75 36391
2023-08-28 08:58:33 exchange02 fe80::d9e3:e981:da2d:427b%3 POST /autodiscover/autodiscover.xml &reqId=bc5918ae-b6a0-0000-9b09-bb94ce79a2fc 444 EXAMPLE\user02 fe80::d9e3:e981:da2d:427b%3 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.5404;+Pro) - 200 0 0 78 10.0.40.92,10.93.4.75 51385
2023-08-28 08:58:43 exchange03 fe80::e478:3abc:f96c:ea94%4 POST /mapi/nspi/ MailboxId=33455f09-88cb-0000-a1a5-730f900d10bd@example.com&FrontEnd=EXCHANGE03.EXAMPLE.COM&RequestId=e393c31b-3ae7-0000-ad11-0632a39901b0&ClientRequestInfo=R:{BD33627D-6B8C-4648-9740-AD9172BB0FEB}:106;RT:GetProps;CI:{DC96322E-2662-4791-BE5D-44163F6033E0}:5;CID:{45ADD272-0FE6-4A23-A58E-A9E379DF101C}&ResponseInfo=XRC:0;SC:0;RC:263040&Stage=BegR:2023-08-28T08:58:44.6445336Z;PostAR:1;PreH:1;PostH:3;EndR:3 444 Anonymous fe80::e478:3abc:f96c:ea94%4 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.5404;+Pro) - 200 0 0 3 10.1.2.61,10.93.4.75 56041
320 changes: 320 additions & 0 deletions packages/iis/data_stream/access/_dev/test/pipeline/test-iis10-access.log-expected.json
View file
Open in desktop
Expand Up @@ -196,6 +196,326 @@
},
"version": "8.0"
}
},
{
"@timestamp": "2023-08-28T07:37:00.000Z",
"destination": {
"address": "10.44.0.131",
"ip": "10.44.0.131",
"port": 443
},
"ecs": {
"version": "8.5.1"
},
"event": {
"category": [
"web",
"network"
],
"duration": 52000000,
"kind": "event",
"original": "2023-08-28 07:37:00 exchange01 10.44.0.131 POST /EWS/Exchange.asmx \u0026CorrelationID=\u003cempty\u003e;\u0026cafeReqId=12d62000-56b5-0000-a9f7-a9e934ecb2d5; 443 EXAMPLE\\user01 10.119.1.75 OC/16.0.5266.1000+(Skype+for+Business) - 200 0 0 52 10.10.10.7 59050",
"outcome": "success",
"type": [
"connection"
]
},
"host": {
LaZyDK marked this conversation as resolved.
Show resolved Hide resolved
"hostname": "exchange01"
},
"http": {
"request": {
"method": "POST"
},
"response": {
"status_code": 200
}
},
"iis": {
"access": {
"sub_status": 0,
"win32_status": 0
}
},
"network": {
"forwarded_ip": "10.10.10.7"
},
"related": {
"ip": [
"10.119.1.75",
"10.44.0.131"
],
"user": [
"EXAMPLE\\\\user01"
]
},
"source": {
"address": "10.119.1.75",
"ip": "10.119.1.75"
},
"tags": [
"preserve_original_event"
],
"url": {
"extension": "asmx",
"original": "/EWS/Exchange.asmx",
"path": "/EWS/Exchange.asmx",
"query": "\u0026CorrelationID=\u003cempty\u003e;\u0026cafeReqId=12d62000-56b5-0000-a9f7-a9e934ecb2d5;"
},
"user": {
"name": "EXAMPLE\\user01"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "Skype",
"original": "OC/16.0.5266.1000 (Skype for Business)",
"version": "16.0.5266.1000"
}
},
{
"@timestamp": "2023-08-28T08:58:45.000Z",
"destination": {
"address": "fe80::d9e3:e981:da2d:427b%3",
"ip": "fe80::d9e3:e981:da2d:427b",
"port": 444
},
"ecs": {
"version": "8.5.1"
},
"event": {
"category": [
"web",
"network"
],
"duration": 4000000,
"kind": "event",
"original": "2023-08-28 08:58:45 exchange02 fe80::d9e3:e981:da2d:427b%3 POST /mapi/emsmdb/ MailboxId=df2fe8fa-492b-0000-a93b-0ce83db65e43@example.com 444 Anonymous fe80::d9e3:e981:da2d:427b%3 Microsoft+Office/16.0+(Windows+NT+6.2;+ucmapi+16.0.5404;+Pro) - 200 0 0 4 10.0.61.69,10.93.4.75 36391",
"outcome": "success",
"type": [
"connection"
]
},
"host": {
"hostname": "exchange02"
},
"http": {
"request": {
"method": "POST"
},
"response": {
"status_code": 200
}
},
"iis": {
"access": {
"sub_status": 0,
"win32_status": 0
}
},
"network": {
"forwarded_ip": "10.0.61.69"
},
"related": {
"ip": [
"fe80::d9e3:e981:da2d:427b",
"fe80::d9e3:e981:da2d:427b"
],
"user": [
"Anonymous"
]
},
"source": {
"address": "fe80::d9e3:e981:da2d:427b%3",
"ip": "fe80::d9e3:e981:da2d:427b"
},
"tags": [
"preserve_original_event"
],
"url": {
"original": "/mapi/emsmdb/",
"path": "/mapi/emsmdb/",
"query": "MailboxId=df2fe8fa-492b-0000-a93b-0ce83db65e43@example.com"
},
"user": {
"name": "Anonymous"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "Other",
"original": "Microsoft Office/16.0 (Windows NT 6.2; ucmapi 16.0.5404; Pro)",
"os": {
"full": "Windows 8",
"name": "Windows",
"version": "8"
}
}
},
{
"@timestamp": "2023-08-28T08:58:33.000Z",
"destination": {
"address": "fe80::d9e3:e981:da2d:427b%3",
"ip": "fe80::d9e3:e981:da2d:427b",
"port": 444
},
"ecs": {
"version": "8.5.1"
},
"event": {
"category": [
"web",
"network"
],
"duration": 78000000,
"kind": "event",
"original": "2023-08-28 08:58:33 exchange02 fe80::d9e3:e981:da2d:427b%3 POST /autodiscover/autodiscover.xml \u0026reqId=bc5918ae-b6a0-0000-9b09-bb94ce79a2fc 444 EXAMPLE\\user02 fe80::d9e3:e981:da2d:427b%3 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.5404;+Pro) - 200 0 0 78 10.0.40.92,10.93.4.75 51385",
"outcome": "success",
"type": [
"connection"
]
},
"host": {
"hostname": "exchange02"
},
"http": {
"request": {
"method": "POST"
},
"response": {
"status_code": 200
}
},
"iis": {
"access": {
"sub_status": 0,
"win32_status": 0
}
},
"network": {
"forwarded_ip": "10.0.40.92"
},
"related": {
"ip": [
"fe80::d9e3:e981:da2d:427b",
"fe80::d9e3:e981:da2d:427b"
],
"user": [
"EXAMPLE\\\\user02"
]
},
"source": {
"address": "fe80::d9e3:e981:da2d:427b%3",
"ip": "fe80::d9e3:e981:da2d:427b"
},
"tags": [
"preserve_original_event"
],
"url": {
"extension": "xml",
"original": "/autodiscover/autodiscover.xml",
"path": "/autodiscover/autodiscover.xml",
"query": "\u0026reqId=bc5918ae-b6a0-0000-9b09-bb94ce79a2fc"
},
"user": {
"name": "EXAMPLE\\user02"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "Outlook",
"original": "Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.5404; Pro)",
"os": {
"full": "Windows 10",
"name": "Windows",
"version": "10"
},
"version": "2016"
}
},
{
"@timestamp": "2023-08-28T08:58:43.000Z",
"destination": {
"address": "fe80::e478:3abc:f96c:ea94%4",
"ip": "fe80::e478:3abc:f96c:ea94",
"port": 444
},
"ecs": {
"version": "8.5.1"
},
"event": {
"category": [
"web",
"network"
],
"duration": 3000000,
"kind": "event",
"original": "2023-08-28 08:58:43 exchange03 fe80::e478:3abc:f96c:ea94%4 POST /mapi/nspi/ MailboxId=33455f09-88cb-0000-a1a5-730f900d10bd@example.com\u0026FrontEnd=EXCHANGE03.EXAMPLE.COM\u0026RequestId=e393c31b-3ae7-0000-ad11-0632a39901b0\u0026ClientRequestInfo=R:{BD33627D-6B8C-4648-9740-AD9172BB0FEB}:106;RT:GetProps;CI:{DC96322E-2662-4791-BE5D-44163F6033E0}:5;CID:{45ADD272-0FE6-4A23-A58E-A9E379DF101C}\u0026ResponseInfo=XRC:0;SC:0;RC:263040\u0026Stage=BegR:2023-08-28T08:58:44.6445336Z;PostAR:1;PreH:1;PostH:3;EndR:3 444 Anonymous fe80::e478:3abc:f96c:ea94%4 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.5404;+Pro) - 200 0 0 3 10.1.2.61,10.93.4.75 56041",
"outcome": "success",
"type": [
"connection"
]
},
"host": {
"hostname": "exchange03"
},
"http": {
"request": {
"method": "POST"
},
"response": {
"status_code": 200
}
},
"iis": {
"access": {
"sub_status": 0,
"win32_status": 0
}
},
"network": {
"forwarded_ip": "10.1.2.61"
},
"related": {
"ip": [
"fe80::e478:3abc:f96c:ea94",
"fe80::e478:3abc:f96c:ea94"
],
"user": [
"Anonymous"
]
},
"source": {
"address": "fe80::e478:3abc:f96c:ea94%4",
"ip": "fe80::e478:3abc:f96c:ea94"
},
"tags": [
"preserve_original_event"
],
"url": {
"original": "/mapi/nspi/",
"path": "/mapi/nspi/",
"query": "MailboxId=33455f09-88cb-0000-a1a5-730f900d10bd@example.com\u0026FrontEnd=EXCHANGE03.EXAMPLE.COM\u0026RequestId=e393c31b-3ae7-0000-ad11-0632a39901b0\u0026ClientRequestInfo=R:{BD33627D-6B8C-4648-9740-AD9172BB0FEB}:106;RT:GetProps;CI:{DC96322E-2662-4791-BE5D-44163F6033E0}:5;CID:{45ADD272-0FE6-4A23-A58E-A9E379DF101C}\u0026ResponseInfo=XRC:0;SC:0;RC:263040\u0026Stage=BegR:2023-08-28T08:58:44.6445336Z;PostAR:1;PreH:1;PostH:3;EndR:3"
},
"user": {
"name": "Anonymous"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "Outlook",
"original": "Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.5404; Pro)",
"os": {
"full": "Windows 10",
"name": "Windows",
"version": "10"
},
"version": "2016"
}
}
]
}
5 changes: 5 additions & 0 deletions packages/iis/data_stream/access/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Expand Up @@ -60,6 +60,11 @@ processors:
(?:-|%{IPORHOST:source.address}) (?:-|%{NOTSPACE:user_agent.original}) (?:-|%{NOTSPACE:iis.access.cookie}) (?:-|%{NOTSPACE:http.request.referrer})
(?:-|%{NUMBER:http.response.status_code:long}) (?:-|%{NUMBER:iis.access.sub_status:long})
(?:-|%{NUMBER:iis.access.win32_status:long}) (?:-|%{NUMBER:_temp_.duration:long})( (?:-|%{IPORHOST:network.forwarded_ip}))?'
- '%{TIMESTAMP_ISO8601:iis.access.time} (?:-|%{NOTSPACE:host.hostname}) (?:-|%{IPORHOST:destination.address}) (?:-|%{WORD:http.request.method})
LaZyDK marked this conversation as resolved.
Show resolved Hide resolved
(?:-|%{NOTSPACE:_temp_.url_path}) (?:-|%{NOTSPACE:_temp_.url_query}) (?:-|%{NUMBER:destination.port:long}) (?:-|%{NOTSPACE:user.name})
(?:-|%{IPORHOST:source.address}) (?:-|%{NOTSPACE:user_agent.original}) (?:-|%{NOTSPACE:http.request.referrer})
(?:-|%{NUMBER:http.response.status_code:long}) (?:-|%{NUMBER:iis.access.sub_status:long}) (?:-|%{NUMBER:iis.access.win32_status:long})
(?:-|%{NUMBER:_temp_.duration:long})( (?:-|%{IPORHOST:network.forwarded_ip}))?'
- uri_parts:
field: _temp_.url_path
ignore_failure: true
Expand Down
2 changes: 2 additions & 0 deletions packages/iis/docs/README.md
View file
Open in desktop
Expand Up @@ -457,6 +457,8 @@ IIS integration offers certain field combinations shipped automatically into Ela

- Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(cookie) cs(Referer) sc-status sc-substatus sc-win32-status sc-bytes, cs-bytes time-taken

- Fields: date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes, cs-bytes time-taken

An example event for `access` looks as following:

```json
Expand Down
2 changes: 1 addition & 1 deletion packages/iis/manifest.yml
View file
Open in desktop
@@ -1,6 +1,6 @@
name: iis
title: IIS
version: "1.15.1"
version: "1.16.0"
description: Collect logs and metrics from Internet Information Services (IIS) servers with Elastic Agent.
type: integration
icons:
Expand Down