Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log group class Support => aws distro cherry pick #31

Open
wants to merge 1 commit into
base: 2.32.2-all-cherry-picks
Choose a base branch
from

Conversation

PettitWesley
Copy link
Owner


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
@PettitWesley
Copy link
Owner Author

147dda285fd4:bottlerocket-eks wppttt$ kubectl -n amazon-cloudwatch logs fluent-bit-fbf8584f8-vvsnl
Fluent Bit v1.9.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2024/05/17 03:47:44] [ info] [fluent bit] version=1.9.10, commit=7eb6b397eb, pid=1
[2024/05/17 03:47:44] [ info] [storage] version=1.4.0, type=memory+filesystem, sync=normal, checksum=disabled, max_chunks_up=128
[2024/05/17 03:47:44] [ info] [storage] backlog input plugin: storage_backlog.8
[2024/05/17 03:47:44] [ info] [cmetrics] version=0.3.7
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] multiline core started
[2024/05/17 03:47:44] [ info] [input:tail:tail.1] multiline core started
[2024/05/17 03:47:44] [ info] [input:tail:tail.2] multiline core started
[2024/05/17 03:47:44] [ info] [input:systemd:systemd.3] seek_cursor=s=fd652eba798a44cc9ecbea19cdd2ac3e;i=1af... OK
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] multiline core started
[2024/05/17 03:47:44] [ info] [input:storage_backlog:storage_backlog.8] queue memory limit: 4.8M
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] https=1 host=127.0.0.1 port=10250
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with Kubelet...
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2024/05/17 03:47:44] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Configured `log_group_class INFREQUENT_ACCESS` will only apply to log groups created by Fluent Bit. Look for the `Created log group` info level message emitted when a group does not already exist and is created.
[2024/05/17 03:47:44] [ warn] [output:cloudwatch_logs:cloudwatch_logs.1] Configured `log_group_class INFREQUENT_ACCESS` will only apply to log groups created by Fluent Bit. Look for the `Created log group` info level message emitted when a group does not already exist and is created.
[2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started
[2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] worker #0 started
[2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] worker #0 started
[2024/05/17 03:47:44] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/05/17 03:47:44] [ info] [sp] stream processor started
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=42307 watch_fd=1 name=/var/log/containers/coredns-68c6b7b454-r6t8q_kube-system_coredns-1709ec5295314385997364635ee603e6d6a559f6cfb70bf3f3e2c431355833c5.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=167812073 watch_fd=2 name=/var/log/containers/coredns-68c6b7b454-tc5mh_kube-system_coredns-2cd2d04c3f2bc1460ac0e86dbd1c3b3b3f8ba0828fc97cc122b491ce897b457e.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=201708930 watch_fd=3 name=/var/log/containers/eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-init-809c5a47b01cca41b44dc18b54b81ab3f3753d1caae2c0c0ca1f6439b5fa4ae3.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=218165991 watch_fd=1 name=/var/log/containers/aws-node-jhslt_kube-system_aws-eks-nodeagent-f083e3e56bc143254ca23011138731a632725afa821c9fc1f442f4e0901eef29.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=16777403 watch_fd=2 name=/var/log/containers/aws-node-jhslt_kube-system_aws-node-7cd53aacd33b6193b69e8085093fe6876e3d0dcda16f4003813528a7c378059e.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=117531748 watch_fd=3 name=/var/log/containers/aws-node-jhslt_kube-system_aws-vpc-cni-init-e5fe7f223d165f4b1c55d1bcc3b8e9f47aa1a475f45d8b3e83c2a1070e82f941.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=185075710 watch_fd=4 name=/var/log/containers/eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.1] inotify_fs_add(): inode=218429523 watch_fd=1 name=/var/log/containers/fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=100789792 watch_fd=4 name=/var/log/containers/kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.'
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events
[2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.974275925.flb', retry in 10 seconds: task_id=4, input=tail.4 > output=cloudwatch_logs.1 (out_id=1)
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.'
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events
[2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.948458095.flb', retry in 10 seconds: task_id=3, input=systemd.3 > output=cloudwatch_logs.1 (out_id=1)
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane with storage class INFREQUENT_ACCESS
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application with storage class INFREQUENT_ACCESS
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.940218327.flb', retry in 7 seconds: task_id=1, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.929877801.flb', retry in 6 seconds: task_id=0, input=tail.0 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:54] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2024/05/17 03:47:54] [ warn] [engine] failed to flush chunk '1-1715917669.177582045.flb', retry in 10 seconds: task_id=2, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:55] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:55] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log
[2024/05/17 03:47:55] [ info] [engine] flush chunk '1-1715917664.929877801.flb' succeeded at retry 1: task_id=0, input=tail.0 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log
[2024/05/17 03:47:56] [ info] [engine] flush chunk '1-1715917664.940218327.flb' succeeded at retry 1: task_id=1, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service
[2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log
[2024/05/17 03:47:59] [ info] [engine] flush chunk '1-1715917664.948458095.flb' succeeded at retry 1: task_id=3, input=systemd.3 > output=cloudwatch_logs.1 (out_id=1)
[2024/05/17 03:47:59] [ info] [engine] flush chunk '1-1715917664.974275925.flb' succeeded at retry 1: task_id=4, input=tail.4 > output=cloudwatch_logs.1 (out_id=1)
[2024/05/17 03:48:04] [ info] [engine] flush chunk '1-1715917669.177582045.flb' succeeded at retry 1: task_id=2, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)

@PettitWesley
Copy link
Owner Author

OperationAbortedException with log_group_class: Race Condition between Fluent Bit instances

Below is normal even if the log_group_class was successfully set.

[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.'
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events
[2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.

A k8s cluster usually has > 1 node and Fluent Bit is run as a daemonset. On startup, every daemon will try to set the logGroupClass, but only one can succeed and create the group. Thus, the above messages will be printed by individual Fluent Bit instances that failed.

The succeeding instance would print:

[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application with storage class INFREQUENT_ACCESS

Search CloudWatch Log Insights for the above to determine if one instance succeeded in setting the log group class because it created the group.

Workaround: Create Log Group before deployment

The best practice workaround is to create log groups before deployment; generally at cluster creation time. Each Fluent Bit instance will not need to create the group. You can disable group creation with auto_create_group Off in your Fluent Bit config.

@PettitWesley
Copy link
Owner Author

Upstream: fluent#8832

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant