-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
out_cloudwatch_logs: support log_group_class option #8832
Conversation
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
147dda285fd4:bottlerocket-eks wppttt$ kubectl -n amazon-cloudwatch logs fluent-bit-fbf8584f8-vvsnl
[2024/05/17 03:47:44] [ info] [fluent bit] version=1.9.10, commit=7eb6b397eb, pid=1
|
OperationAbortedException with log_group_class: Race Condition between Fluent Bit instancesBelow is normal even if the log_group_class was successfully set.
A k8s cluster usually has > 1 node and Fluent Bit is run as a daemonset. On startup, every daemon will try to set the logGroupClass, but only one can succeed and create the group. Thus, the above messages will be printed by individual Fluent Bit instances that failed. The succeeding instance would print:
Search CloudWatch Log Insights for the above to determine if one instance succeeded in setting the log group class because it created the group. Workaround: Create Log Group before deploymentThe best practice workaround is to create log groups before deployment; generally at cluster creation time. Each Fluent Bit instance will not need to create the group. You can disable group creation with |
|
@PettitWesley thank you, can you pls add here the PR for docs ? (cc: @lecaros) |
ping @PettitWesley
|
Hi, When I updated 3.0.6 -> 3.0.7 I noticed my previously working config:
Required me to add
before fluent-bit would start. Could it be related to this PR? |
Echoing @michael-stevens , I believe this has introduced a regression. I do not want fluent bit to create my log groups on my behalf, so I do not want to enable |
Enter
[N/A]
in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-test
label to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.