-
Notifications
You must be signed in to change notification settings - Fork 135
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] Cannot register file error #796
Comments
same issue: |
Same issue: [2024/07/17 14:03:56] [error] [plugins/in_tail/tail_fs_inotify.c:147 errno=2] No such file or directory EKS: 1.24 |
Source code for this "No such file or directory" error: NOTE that the The |
Describe the question/issue
When the rate at which logs are ingested per second is elevated, the system produce an error stating "cannot register file."
Configuration
The customer tried to have fluent retrieve about 3000 logs per second. It is unable to retrieve all the logs.
3000 is the number of logs that fluent retrieved, but actually more than 3000 logs are sent by application.
Fluent-bit was deployed by built-in feature(aws-logging configmap) of EKS Fargate.
The logs were set to send to both kinesis firehose and cloudwatch logs, and the number of logs matched.
Fluent Bit Log Output
Fluent Bit Version Info
Fluent Bit v1.9.10(eks on fargate built-in fluentbit)
EKS version 1.27, 1.28, 1.29
Cluster Details
VPC is unlimited outbound, inbound is focused on specific ip and sg.
Use Appmesh
Using EKS with Fargate
Incorporate Fluent Bit into Fargate
Application Details
Logs are no longer recoverable past 3000 per second.
Roughly 6 MB per second.
Related Issues
Not sure if this issue related to EKS AMI update with 1024 NOFILE
awslabs/amazon-eks-ami#1535
The text was updated successfully, but these errors were encountered: