fluentd: Fix bug that caused lines to be dropped when containing non utf-8 characters #5107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a log line in hash format contains non UTF-8 characters fluentd would drop the complete line because it failed to convert the line in key-value format. By forcing UTF-8 encoding and replacing non UTF-8 characters with empty strings the log line will not be dropped but only contain the valid UTF-8 characters. Fixes #5099 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum
changed the title
|
@cyriltovena is bumping the fluentd plugin version here the right thing to do? 7fd4c40 |
|
Correct ! |
cyriltovena
reviewed
Jan 12, 2022
| @@ -278,6 +278,8 @@ def record_to_line(record) | |||
| when :key_value | |||
| formatted_labels = [] | |||
| record.each do |k, v| | |||
| # Remove non UTF-8 characters by force-encoding the string and replacing said chars with empty string | |||
| v = v.encode('utf-8', invalid: :replace, replace: '') | |||
There was a problem hiding this comment.
should we replace with another special character may be ? Not sure if we can use empty string , it will not tell users there was an issue there ? WDYT ?
There was a problem hiding this comment.
You can let Ruby decide how to replace the invalid characters, like so:
v = v.encode('utf-8', invalid: :replace)
cyriltovena
reviewed
Jan 12, 2022
| @@ -4,7 +4,7 @@ $LOAD_PATH.push File.expand_path('lib', __dir__) | |||
|
|
|||
| Gem::Specification.new do |spec| | |||
| spec.name = 'fluent-plugin-grafana-loki' | |||
| spec.version = '1.2.16' | |||
| spec.version = '1.2.17' | |||
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum
commented
Jan 12, 2022
| @@ -122,7 +139,7 @@ | |||
| expect(body[:streams][0]['stream'].empty?).to eq true | |||
| expect(body[:streams][0]['values'].count).to eq 1 | |||
| expect(body[:streams][0]['values'][0][0]).to eq '1546270458000000000' | |||
| expect(body[:streams][0]['values'][0][1]).to eq 'message="' + content[0] + '" stream="stdout"' | |||
| expect(body[:streams][0]['values'][0][1]).to eq 'message="' + content[0] + '" stream=stdout' | |||
There was a problem hiding this comment.
This test was failing since c298514793ddf49ec8d15972b35cdae5f225b5d2
|
Thank you so much @chaudum |
cyriltovena
pushed a commit
that referenced
this pull request
Jan 17, 2022
* Fix encoding error in fluentd client This change fixes a bug that is caused by incorrectly trying to encode any values other than string. This bug was a regression introduced with #5107 and caused `NoMethodError` errors. Fixes #5099 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Bump fluent-plugin-grafana-loki to version 1.2.18 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
When a log line in hash format contains non UTF-8 characters fluentd
would drop the complete line because it failed to convert the line in
key-value format.
By forcing UTF-8 encoding and replacing non UTF-8 characters with empty
strings the log line will not be dropped but only contain the valid
UTF-8 characters.
Which issue(s) this PR fixes:
Fixes #5099
Special notes for your reviewer:
Checklist
CHANGELOG.mdabout the changes.