fluent-bit > 5.0.6 retry_exhausted_action delete creating underflow exception

[smiley-ci]

s3 output Buffer Full Even with buffer path is empty too. Any workaround to get the chunks deleted post retry limit in fluentbit versions 4

[2026/07/23 17:35:18.363] [ warn] [output:s3:s3.0] Could not buffer chunk. Data order preservation will be compromised
[2026/07/23 17:35:18.363] [error] [output:s3:s3.0] Buffer is full: current_buffer_size=18446744073684963328, new_data=9272, store_dir_limit_size=2000000000 bytes

[OUTPUT]
Name s3
Match_regex ^(kubernetes.logs.*)$
Bucket ${S3_BUCKET}
Region us-east-1
endpoint ${S3_ENDPOINT}
compression gzip
s3_key_format /logs/v1_%Y%m%d%H%M%S_$UUID.gz
Upload_chunk_size 50M
static_file_path On
use_put_object Off
store_dir /var/log/fluentbit/log/s3/
store_dir_limit_size 2GB
retry_limit 5
retry_exhausted_action delete
[OUTPUT]
Name prometheus_exporter
match fluentbit_metrics
host ::
port 2022

BufferPath:

/var/log/fluentbit/log/s3$ ls -lhtr
var/log/fluentbit/log/s3$ ls -lhtr
total 16K
drwxr-xr-x 2 root root 4.0K Jul 17 17:07 quarantine
drwxr-xr-x 2 root root 4.0K Jul 17 17:07 multipart_upload_metadata
drwxr-xr-x 2 root root 4.0K Jul 20 15:09 2026-07-20T15:09:42
drwxr-xr-x 2 root root 4.0K Jul 20 15:09 2026-07-17T17:07:11

[Maxaubert]

This is a bug in the S3 buffer accounting, not a real "buffer full" and not a problem with retry_exhausted_action delete. The fix is in v5.1.1 and later.

current_buffer_size=18446744073684963328 is 2^64 minus about 24.6 MB, so the counter wrapped around below zero. Once it wraps, every new chunk fails the store_dir_limit_size check and you get "Could not buffer chunk" / "Buffer is full" even though the directory is nearly empty.

Cause: before the fix, the plugin added to the counter when buffering a chunk and subtracted from it unconditionally when a chunk file was deleted. Chunks left in store_dir from a previous run (your 2026-07-17T17:07:11 and 2026-07-20T15:09:42 directories) were not credited to the counter on startup. Deleting them later, for example after retries are exhausted, then subtracted more than had ever been added, and the unsigned counter wrapped.

The fix is #12266 ("out_s3: Address underflow of buffer size on startup"):

• Restored files are now counted at startup.
• Limit check and reservation are one atomic operation.
• Subtraction clamps to zero instead of wrapping.

Commits: 0c87b56 and 02baae40e

It first shipped in https://github.com/fluent/fluent-bit/releases/tag/v5.1.1 (also in v5.1.2). I checked the tags: it is not in any v5.0.x release (latest 5.0.10) or v4.2.x (latest 4.2.8), so those still have the bug.

So the fix is to upgrade to 5.1.1 or newer. On an older version, the only way out I know of is to restart with a clean store_dir so no leftover chunks exist at startup. That loses the buffered data in it, and it can reoccur after any restart that leaves chunks behind.