Occasional Corruption of JSON Logs when Uploading to S3 with Fluent Bit

[jayong-sv]

Bug Report

Describe the bug
I'm using Fluent Bit versions 1.8.1 and 1.9.10. There are some corrupted logs in S3 when uploading JSON logs. While it doesn't happen often, it seems to occur rarely, especially when the network conditions are not great. I'm currently trying to figure out whether Fluent Bit or network problems are causing this. Any help in resolving this would be much appreciated.

The corrupted logs have this kind of structure, and either the file can't be opened, or the JSON is messed up, causing parsing issues.

To Reproduce
conf file

[SERVICE]
    Flush         5.0
    Log_File      /home/ubuntu/our-log-dir/fluent-bit.log
    Log_Level     info
    Parsers_file  parsers.conf
    Daemon        off
    storage.path  /home/ubuntu/fluent-bit/storage/flb-storage/
    storage.sync  full
    storage.checksum on
    storage.backlog.mem_limit 200M

[INPUT]
    Name              tail
    Tag               our-log-name.history.s3
    Path              /home/vv-team/our-log-dir/our-log-name/history/history.log
    Mem_Buf_Limit     10MB
    Skip_Long_Lines   On
    Refresh_Interval  5.0
    storage.type      filesystem

[FILTER]
    Name parser
    Match *
    Key_name log
    Parser JSON_PARSE

[OUTPUT]
    Name    s3
    Match   *.s3
    region  ap-northeast-2
    bucket  our-log-dir
    store_dir   /home/ubuntu/fluent-bit/storage/s3
    total_file_size 10M
    storage.total_limit_size 30M
    upload_timeout  5m
    s3_key_format   /$TAG[0]/$TAG[1]/${INVENTORY_ID}/%Y-%m-%d/%Y%m%d%H%M%S_${INVENTORY_ID}_${HOSTNAME}_$TAG[1]_$UUID.json
    s3_key_format_tag_delimiters .
    Retry_Limit False


[OUTPUT]
    Name    s3
    Match   our-log-name.history.s3
    region  ap-northeast-2
    bucket  our-log-dir
    store_dir   /home/ubuntu/fluent-bit/storage/s3_log_name
    total_file_size 10M
    storage.total_limit_size 30M
    upload_timeout  5m
    s3_key_format   /our-log-name-history-v3/part_date=%Y-%m-%d/%Y%m%d%H%M%S_${INVENTORY_ID}_${HOSTNAME}_$TAG[1]_$UUID.json
    s3_key_format_tag_delimiters .
    Retry_Limit False

parser

[PARSER]
    Name        JSON_PARSE
    Format      json
    Time_Offset +0900

service

[Unit]
Description=TD Agent Bit
Requires=network.target
After=network.target

[Service]
Type=simple
ExecStart=/opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
Restart=always

[Install]
WantedBy=multi-user.target

Your Environment

• Version used: 1.8.1, 1.9.10
• Environment name and version: plain pc
• Operating System and version: ubuntu 20.04

[patrick-stephens]

Neither 1.8 or 1.9 are supported so I would encourage you to step up to the latest 2.2.0 version to confirm if it is already resolved.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.