Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(logs): correctly handle newlines #2805

Merged
merged 2 commits into from
Jan 19, 2023
Merged

fix(logs): correctly handle newlines #2805

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f77a81b
docs: fix description of merge-cri-lines operator
andrzej-stencel Jan 18, 2023
6a5ff8a
fix(logs): correctly handle newlines
andrzej-stencel Jan 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- fix(logs): make `excludeHostRegex` consistent between Otelcol and Fluentd [#2771]
- The `sumologic.logs.container.excludeHostRegex` should filter on the Kubernetes node name,
to be consistent with Fluentd and chart v2.
- fix(logs): correctly handle newlines [#2805]
- Fixes [#2802], [#2803]

[#2724]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2724
[#2745]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2745
Expand All @@ -50,6 +52,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
[#2791]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2791
[#2773]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2773
[#2790]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2790
[#2802]: https://github.com/SumoLogic/sumologic-kubernetes-collection/issues/2802
[#2803]: https://github.com/SumoLogic/sumologic-kubernetes-collection/issues/2803
[#2805]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2805
[v1.15.3-sumo-0]: https://github.com/SumoLogic/sumologic-kubernetes-fluentd/releases/tag/v1.15.3-sumo-0
[Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v3.0.0-beta.0...main

Expand Down
16 changes: 13 additions & 3 deletions deploy/helm/sumologic/conf/logs/collector/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,15 +101,15 @@ receivers:
- id: merge-docker-lines
type: recombine
source_identifier: attributes["log.file.path"]
output: {{ .Values.sumologic.logs.multiline.enabled | ternary "merge-multiline-logs" "extract-metadata-from-filepath" }}
combine_field: body.log
combine_with: ""
is_last_entry: body.log matches "\n$"
output: strip-trailing-newline

## merge-cri-lines stitches back together log lines split by CRI logging drivers.
## Input Body (JSON): { "log": "2001-02-03 04:05:06 very long li", "logtag": "P" }
## Input Body (JSON): { "log": "ne that was split by the logging driver", "logtag": "F" }
## Output Body (JSON): { "log": "2001-02-03 04:05:06 very long line that was split by the logging driver\n", "stream": "stdout" }
## Output Body (JSON): { "log": "2001-02-03 04:05:06 very long line that was split by the logging driver", "logtag": "F" }
- id: merge-cri-lines
type: recombine
source_identifier: attributes["log.file.path"]
Expand All @@ -119,6 +119,16 @@ receivers:
is_last_entry: body.logtag == "F"
overwrite_with: newest

## strip-trailing-newline removes the trailing "\n" from the `log` key. This is required for logs coming from Docker container runtime.
## Input Body (JSON): { "log": "2001-02-03 04:05:06 very long line that was split by the logging driver\n", "stream": "stdout" }
## Output Body (JSON): { "log": "2001-02-03 04:05:06 very long line that was split by the logging driver", "stream": "stdout" }
- id: strip-trailing-newline
type: regex_parser
regex: "^(?P<log>.*)\n$"
parse_from: body.log
parse_to: body
output: {{ .Values.sumologic.logs.multiline.enabled | ternary "merge-multiline-logs" "extract-metadata-from-filepath" }}

## merge-multiline-logs merges incoming log records into multiline logs.
## Input Body (JSON): { "log": "2001-02-03 04:05:06 first line\n", "stream": "stdout" }
## Input Body (JSON): { "log": " second line\n", "stream": "stdout" }
Expand All @@ -130,7 +140,7 @@ receivers:
output: extract-metadata-from-filepath
source_identifier: attributes["log.file.path"]
combine_field: body.log
combine_with: ""
combine_with: "\n"
is_first_entry: body.log matches {{ .Values.sumologic.logs.multiline.first_line_regex | quote }}
{{- end }}

Expand Down
13 changes: 11 additions & 2 deletions tests/helm/logs_otc/static/basic.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ data:
combine_with: ""
id: merge-docker-lines
is_last_entry: body.log matches "\n$"
output: merge-multiline-logs
output: strip-trailing-newline
source_identifier: attributes["log.file.path"]
type: recombine
- combine_field: body.log
Expand All @@ -105,8 +105,17 @@ data:
overwrite_with: newest
source_identifier: attributes["log.file.path"]
type: recombine
- id: strip-trailing-newline
output: merge-multiline-logs
parse_from: body.log
parse_to: body
regex: |-
^(?P<log>.*)
$
type: regex_parser
- combine_field: body.log
combine_with: ""
combine_with: |2+

Comment on lines +117 to +118
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty weird, why does it get interpreted as a special character?

Copy link
Contributor Author

@andrzej-stencel andrzej-stencel Jan 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just regular Yaml 😉

I looked it up and it seems to be a combination of the "keep" chomping indicator + and the "explicit" indentation indicator 2.

It's actually a bit suspicious, because the next line has no indentation at all, so the 2 seems incorrect 🤔 But hey, the tests pass 🤷

id: merge-multiline-logs
is_first_entry: body.log matches "^\\[?\\d{4}-\\d{1,2}-\\d{1,2}.\\d{2}:\\d{2}:\\d{2}"
output: extract-metadata-from-filepath
Expand Down