fluent-plugin-grafana-loki: Add fluentd_thread label when `flush_thread_count' > 1
#1514
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
|
Thank you for this !! Really love that you're contributing back. Can you bump the gem version ? And we should be good to go. |
srstsavage
force-pushed
the
fluentd-buffer-thread-id
branch
from
196343d
to
61fc4a5
Compare
|
@cyriltovena Done. Set it to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Currently fluent-plugin-grafana-loki can't be used with multiple buffer flush threads (
num_threadsorflush_thread_count> 1) because logs with the same label sets are flushed in parallel, resulting inentry out of order for streamerrors in Loki.This PR changes
fluent-plugin-grafana-lokito automatically add a 'fluentd_threadlog label when the fluentd loki output is configured withflush_thread_count` > 1, which allows logs flushed in parallel to have unique label sets and avoid out-of-order errors.In our deployment (~300 fluentd agents forwarding to a single loki), using this change and setting
flush_thread_countto 4 has allowed us to achieve sub-500ms log latecy times (measured with loki-canary). Previously we were seeing 32s latency, even when using multi-worker fluentd configs.Which issue(s) this PR fixes:
No issue currently, should I create one?
Special notes for your reviewer:
Thanks for reviewing!
Checklist