S3 Output Plugin Config Race Condition

[miztiik]

Based on FluenBit v1.6 Doc, - https://docs.fluentbit.io/manual/pipeline/outputs/s3

The default total_file_size to default of 1M, and the default upload_chunk_size is 5M. This leads to a configuration mismatch for multi-part uploads. And FluentBit service does not get started because of the mismatch in configuration.

The user has to reconfigure the total_file_size to something greater than 5M to get it work.

As the defaults does not work as-is, I feel the documentation should be updated

• Steps to reproduce the problem:
  Version used: 1.6

[OUTPUT]
     Name               s3
     Match              *
     bucket             BUCKET_NAME
     region             us-east-1
     total_file_size    1M

error log messages,

Dec 08 15:06:41 ip-10-10-0-33.ec2.internal td-agent-bit[3461]: [2020/12/08 15:06:41] [ info] [output:s3:s3.0] Using upload size 1000000 bytes
Dec 08 15:06:41 ip-10-10-0-33.ec2.internal td-agent-bit[3461]: [2020/12/08 15:06:41] [error] [output:s3:s3.0] upload_chunk_size can not be larger than total_f
Dec 08 15:06:41 ip-10-10-0-33.ec2.internal td-agent-bit[3461]: [2020/12/08 15:06:41] [error] [output] Failed to initialize 's3' plugin
...
Dec 08 15:06:41 ip-10-10-0-33.ec2.internal systemd[1]: Unit td-agent-bit.service entered failed state.
Dec 08 15:06:41 ip-10-10-0-33.ec2.internal systemd[1]: td-agent-bit.service failed.
...
Dec 08 15:06:42 ip-10-10-0-33.ec2.internal systemd[1]: Failed to start TD Agent Bit.
-- Subject: Unit td-agent-bit.service has failed

[PettitWesley]

@miztiik You can fix this be adding: use_put_object true in your config.

I do think its a bad user experience though- if the total file size is below the min size for multipart, the plugin should auto-set use_put_object to be true and then ignore the default value for upload_chunk_size.

[PettitWesley]

@zhonghui12 Can you add this on your todo list, it should be very simple to fix.