Skip to content

aws/data_stream/route53_public_logs: Handle non-%HOSTNAME entries #9249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jul 22, 2024
Merged

aws/data_stream/route53_public_logs: Handle non-%HOSTNAME entries #9249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b68ade9
Expand the set of allowed characters in the question.name part of the…
cafuego Feb 28, 2024
cac5cc4
Add a test case for _spf (previously failed to match) to avoid regres…
cafuego Feb 28, 2024
d49c066
Bump the changelog and manifest files.
cafuego Feb 28, 2024
e48a7ec
Merge branch 'main' into route53-log-hostnames
cafuego Feb 28, 2024
95ac269
Bump the manifest file after resolving the changelog merge conflict.
cafuego Feb 28, 2024
e34c87a
Merge branch 'main' into route53-log-hostnames
cafuego Mar 29, 2024
98ab6a1
Merge branch 'main' into route53-log-hostnames
cafuego Apr 29, 2024
1a134c1
Merge branch 'main' into route53-log-hostnames
cafuego May 29, 2024
f7c9392
Update packages/aws/changelog.yml
cafuego Jul 22, 2024
fe9303c
Merge branch 'main' into route53-log-hostnames
cafuego Jul 22, 2024
f93d5e4
Update packages/aws/data_stream/route53_public_logs/_dev/test/pipelin…
cafuego Jul 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/aws/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.21.0"
changes:
- description: Fix route53 public logs grok pattern.
type: enhancement
link: https://github.com/elastic/integrations/pull/9249
- version: "2.20.0"
changes:
- description: Add S3 polling option to data streams use aws-s3 input
Expand Down
3 changes: 2 additions & 1 deletion packages/aws/data_stream/route53_public_logs/_dev/test/pipeline/test-route53.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@
1.0 2017-12-13T08:16:03.983Z Z123412341234 example.com ANY NOERROR UDP FRA6 2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6 2001:db8:abcd::/48
1.0 2017-12-13T08:15:50.342Z Z123412341234 bad.example.com A NXDOMAIN UDP IAD12 89.160.20.112 192.168.111.0/24
1.0 2017-12-13T08:16:05.744Z Z123412341234 txt.example.com TXT NOERROR UDP JFK5 89.160.20.112 -
1.0 2017-12-13T08:16:05.744Z Z123412341234 txt.example.com TXT NOERROR UDP GRU1-C1 89.160.20.112 -
1.0 2017-12-13T08:16:05.744Z Z123412341234 txt.example.com TXT NOERROR UDP GRU1-C1 89.160.20.112 -
1.0 2017-12-13T08:17:05.744Z Z123412341234 _spf.example.com SPF NOERROR UDP DEN50-C1 89.160.20.112 -
75 changes: 75 additions & 0 deletions ...ges/aws/data_stream/route53_public_logs/_dev/test/pipeline/test-route53.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -440,6 +440,81 @@
"tags": [
"preserve_original_event"
]
},
{
"@timestamp": "2017-12-13T08:17:05.744Z",
"aws": {
"route53": {
"edge_location": "DEN50-C1",
"hosted_zone_id": "Z123412341234"
}
},
"cloud": {
"provider": "aws"
},
"dns": {
"question": {
"name": "_spf.example.com",
"registered_domain": "example.com",
"subdomain": "_spf",
"top_level_domain": "com",
"type": "SPF"
},
"response_code": "NOERROR"
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"network"
],
"kind": "event",
"original": "1.0 2017-12-13T08:17:05.744Z Z123412341234 _spf.example.com SPF NOERROR UDP DEN50-C1 89.160.20.112 -",
"outcome": "success",
"type": [
"protocol"
]
},
"network": {
"iana_number": "17",
"protocol": "dns",
"transport": "udp",
"type": "ipv4"
},
"related": {
"hosts": [
"_spf.example.com"
],
"ip": [
"89.160.20.112"
]
},
"source": {
"address": "89.160.20.112",
"as": {
"number": 29518,
"organization": {
"name": "Bredband2 AB"
}
},
"geo": {
"city_name": "Linköping",
"continent_name": "Europe",
"country_iso_code": "SE",
"country_name": "Sweden",
"location": {
"lat": 58.4167,
"lon": 15.6167
},
"region_iso_code": "SE-E",
"region_name": "Östergötland County"
},
"ip": "89.160.20.112"
},
"tags": [
"preserve_original_event"
]
}
]
}
2 changes: 1 addition & 1 deletion packages/aws/data_stream/route53_public_logs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ processors:
- grok:
field: event.original
patterns:
- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{HOSTNAME:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'
- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DATA:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'
pattern_definitions:
Comment on lines +34 to 35
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alternative would be to provided an augmented hostname pattern that includes all the codepoints that occur in DNS names. This only requires that we add _ to the possible codepoints.

Suggested change
- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DATA:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'
pattern_definitions:
- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DNS_QUESTION:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'
pattern_definitions:
DNS_QUESTION: \b(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62}))*(\.?|\b)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the very least you also need to add an escaped backslash, or you'll still miss any encoded UTF-8 data.

DNS entries can also contain any binary rubbish you'd care to put in (although Route53 may not allow you to add that) so from a log analysis perspective it's important to grab those logs and allow them to be analysed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DNS entries can also contain any binary rubbish you'd care to put in

That's horrifying, but point well made.

EDGE_LOCATION: '[A-Z]{3}\d+(-[A-Z]+\d+)?'
SUBNET: '%{IP}/[0-9]+'
Expand Down
2 changes: 1 addition & 1 deletion packages/aws/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.0
name: aws
title: AWS
version: 2.20.0
version: 2.21.0
description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
type: integration
categories:
Expand Down