Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[apache_tomcat.access] Restructure Grok pattern and ingest pipeline to improve performance #8615

Merged
merged 10 commits into from Dec 1, 2023
Merged

[apache_tomcat.access] Restructure Grok pattern and ingest pipeline to improve performance #8615

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
54ede5d
Change grok processing to improve perfomance
SimonKoetting Nov 30, 2023
49fc0b1
Add changelog details
SimonKoetting Nov 30, 2023
cb13817
fix destination bytes detection in grok
SimonKoetting Nov 30, 2023
1fed419
change version in manifest
SimonKoetting Nov 30, 2023
d034554
Update default.yml
SimonKoetting Nov 30, 2023
04d0abf
Fix _tmp_grok detection bug
SimonKoetting Nov 30, 2023
8ee422a
adjust version in manifest
SimonKoetting Nov 30, 2023
02142f8
adjust version in changelog
SimonKoetting Nov 30, 2023
8e77483
Update packages/apache_tomcat/changelog.yml
SimonKoetting Nov 30, 2023
6e6e891
rename to _tmp.grok
SimonKoetting Nov 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/apache_tomcat/changelog.yml
View file
Open in desktop
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.0.1"
changes:
- description: Improve apacht_tomcat.access pipeline performance
SimonKoetting marked this conversation as resolved.
Show resolved Hide resolved
type: enhancement
link: https://github.com/elastic/integrations/pull/8615
- version: "1.0.0"
changes:
- description: Make Apache Tomcat GA.
Expand Down
20 changes: 16 additions & 4 deletions packages/apache_tomcat/data_stream/access/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Expand Up @@ -25,15 +25,27 @@ processors:
field: event.original
tag: 'grok_parse_log'
patterns:
- '^(%{IP:source.ip}|%{DATA:source.user.name}) %{DATA:apache_tomcat.access.http.ident} %{DATA:apache_tomcat.access.http.useragent} \[%{DATA:_tmp.timestamp}\] \"%{DATA:http.request.method} %{DATA:url.original} HTTP\/%{DATA:http.version}\" %{NUMBER:http.response.status_code} %{DATA:destination.bytes} (?:%{IP:apache_tomcat.access.ip.local}%{SPACE})?(?:%{CONN_STATUS:apache_tomcat.access.connection_status}%{SPACE})?(?:%{NUMBER:apache_tomcat.access.response_time}%{SPACE})?\"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" X-Forwarded-For=\"%{DATA:_tmp.header_forwarder}(\")*$'
- '^(%{IP:source.ip}|%{DATA:source.user.name}) %{DATA:apache_tomcat.access.http.ident} %{DATA:apache_tomcat.access.http.useragent} \[%{DATA:_tmp.timestamp}\] \"%{DATA:http.request.method} %{DATA:url.original} HTTP\/%{DATA:http.version}\" %{NUMBER:http.response.status_code} %{DATA:destination.bytes} \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}(\")*$'
- '^(%{IP:source.ip}|%{DATA:source.user.name}) %{DATA:apache_tomcat.access.http.ident} %{DATA:apache_tomcat.access.http.useragent} \[%{DATA:_tmp.timestamp}\] \"%{DATA:http.request.method} %{DATA:url.original} HTTP\/%{DATA:http.version}\" %{NUMBER:http.response.status_code} %{DATA:destination.bytes}$'
- '^(%{IP:source.ip}|%{DATA:source.user.name}) %{DATA:apache_tomcat.access.http.ident} %{DATA:apache_tomcat.access.http.useragent} \[%{DATA:_tmp.timestamp}\] \"%{DATA:http.request.method} %{DATA:url.original} HTTP\/%{DATA:http.version}\" %{NUMBER:http.response.status_code} %{POSINT:destination.bytes}( %{GREEDYDATA:_tmp_grok})?$'
SimonKoetting marked this conversation as resolved.
Show resolved Hide resolved
on_failure:
- append:
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag fail-{{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- grok:
field: _tmp_grok
SimonKoetting marked this conversation as resolved.
Show resolved Hide resolved
tag: 'grok_parse_log2'
patterns:
- '(?:%{IP:apache_tomcat.access.ip.local}%{SPACE})?(?:%{CONN_STATUS:apache_tomcat.access.connection_status}%{SPACE})?(?:%{NUMBER:apache_tomcat.access.response_time}%{SPACE})?\"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\" X-Forwarded-For=\"%{DATA:_tmp.header_forwarder}(\")*$'
SimonKoetting marked this conversation as resolved.
Show resolved Hide resolved
- '\"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}(\")*$'
pattern_definitions:
CONN_STATUS: "[X+-]"
ignore_missing: true
on_failure:
- append:
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag fail-{{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- remove:
field: _tmp_grok
ignore_missing: true
SimonKoetting marked this conversation as resolved.
Show resolved Hide resolved
- append:
field: related.ip
value: '{{{source.ip}}}'
Expand Down Expand Up @@ -181,4 +193,4 @@ on_failure:
- append:
field: event.kind
value: pipeline_error
allow_duplicates: false
allow_duplicates: false
2 changes: 1 addition & 1 deletion packages/apache_tomcat/manifest.yml
View file
Open in desktop
@@ -1,7 +1,7 @@
format_version: "3.0.0"
name: apache_tomcat
title: Apache Tomcat
version: "1.0.0"
version: "1.0.1"
SimonKoetting marked this conversation as resolved.
Show resolved Hide resolved
description: Collect and parse logs and metrics from Apache Tomcat servers with Elastic Agent.
categories: ["web", "observability"]
type: integration
Expand Down