Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The value of total_file_size in example config for S3 output plugin is invalid. #411

Closed
tam0ri opened this issue Oct 28, 2020 · 1 comment

Comments

@tam0ri
Copy link

tam0ri commented Oct 28, 2020

https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/s3.md#using-s3-without-persisted-disk

[OUTPUT]
     Name s3
     Match *
     bucket your-bucket
     region us-east-1
     total_file_size 1M
     upload_timeout 1m

In this example config, total_file_size is specified as 1M. However, default value of upload_chunk_size is 5,242,880 bytes and upload_chunk_size can not be larger than total_file_size.
https://github.com/fluent/fluent-bit/blob/44190c2a1c4b939dc9ecb2908148d38c82a40831/plugins/out_s3/s3.c#L353-L357

        if (ctx->upload_chunk_size > ctx->file_size) {
            flb_plg_error(ctx->ins,
                          "upload_chunk_size can not be larger than total_file_size");
            return -1;
        }

Also, upload_chunk_size must be at least 5,242,880 bytes.
https://github.com/fluent/fluent-bit/blob/44190c2a1c4b939dc9ecb2908148d38c82a40831/plugins/out_s3/s3.c#L358-L361

        if (ctx->upload_chunk_size < MIN_CHUNKED_UPLOAD_SIZE) {
            flb_plg_error(ctx->ins, "upload_chunk_size must be at least 5,242,880 bytes");
            return -1;
        }

Proposal:
Would you modify the example as below?

[OUTPUT]
     Name s3
     Match *
     bucket your-bucket
     region us-east-1
     total_file_size 6M
     upload_timeout 1m
@PettitWesley
Copy link
Contributor

So upload_chunk_size is for multipart uploads only. But the no persistent disk example is supposed to use the PutObject API, since that is best when you don't have disk to store the upload state file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants