s3 output dropping metrics without informative logs

[jmcarp]

Bug Report

Describe the bug

I'm testing out the s3 output plugin to write logs to gcs using the s3 compatibility layer. Some logs arrive in s3, but many are rejected with errors that are, at least to me, uninformative.

To Reproduce

• Steps to reproduce the problem:

Configure log export to google cloud storage using the s3 output plugin:

    [OUTPUT]
        Name s3
        Match *
        compression gzip
        content_type application/gzip
        use_put_object true
        total_file_size 1M
        upload_timeout 1m
        endpoint https://storage.googleapis.com
        bucket my-bucket
        region my-region
        s3_key_format /$TAG/%Y/%m/%d/%H/$UUID.log.gz

Some uploads succeed, but many fail with logs like this:

fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [error] [http_client] broken connection to storage.googleapis.com:443 ?
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [debug] [aws_client] storage.googleapis.com: http_do=-1, HTTP Status: 0
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [debug] [upstream] KA connection #49 to storage.googleapis.com:443 is now available
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [error] [output:s3:s3] PutObject request failed
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [error] [output:s3:s3] Could not send chunk with tag redacted.cockroach.health.
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [debug] [output:s3:s3] Running upload timer callback (upload_queue)..
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [debug] [output:s3:s3] No files found in upload_queue. Scanning for timed out chunks
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [debug] [output:s3:s3] Running upload timer callback (cb_s3_upload)..
fluent-bit-vr8cw fluent-bit [2023/08/23 21:38:42] [debug] [upstream] KA connection #49 is in a failed state to: storage.googleapis.com:443, cleaning up

Expected behavior

Output to s3 either works or prints an informative error message on failure.

Screenshots

Your Environment

• Version used: 2.1.8

[flecno]

I have the same problem with GCS when using the S3 output. For some files the upload works, but for many uploads I get the same TLS errors:

[2023/09/10 19:10:28] [error] [output:s3:s3.0] PutObject request failed
[2023/09/10 19:10:28] [error] [output:s3:s3.0] Could not send chunk with tag xxx
[2023/09/10 19:10:28] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/09/10 19:10:28] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/09/10 19:10:28] [error] [http_client] broken connection to storage.googleapis.com:443 ?

I use this basic configuration:

      [OUTPUT]
          endpoint https://storage.googleapis.com
          name s3
          match *
          bucket ${FLUENTBIT_OUTPUT_BUCKET}
          s3_key_format /$TAG/%Y/%m/%d/%H_%M_%S
          json_date_key false
          compression gzip

[Lord-Y]

@jmcarp @flecno Did you guys finally have been able to send logs to GCS?. I tried with fluent-bit version 2.2 but it doesn't work.

[Lord-Y]

Alright, I made it worked. I needed to do this https://cloud.google.com/storage/docs/authentication/managing-hmackeys#command-line_1 as specified here #4165 (comment)

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.