out_stackdriver: failed to flush chunk

[franciscovalentecastro]

Bug Report

Describe the bug
When executing fluent-bit using a configuration that uses the out_stackdriver plugin, I don't get any output in the Google Cloud Logging interface and getting warnings of the form :

[ warn] [engine] failed to flush chunk '15752-1649349347.320582218.flb', retry in 10 seconds: task_id=0, input=dummy.0 > output=stackdriver.1 (out_id=1)

Tried a couple of previous commits (e.g. bc905f6 ) where the issue doesn't appear. Probably the following commits 0cd3519 , dfc840b related to the out_stackdriver plugin may be causing the issue :

To Reproduce

When executing fluent-bit from current master branch, version v1.9.2, the following way :

fluent-bit/build/bin/fluent-bit   --config ./fluent_bit_main_dummy.conf   --parser ./fluent_bit_parser_dummy.conf

With the following configuration files (and a couple other different configurations that use the out_stackdriver output plugin):

fluent_bit_main_dummy.conf

[INPUT]
    Name dummy
    Tag  dummy.data
    Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2", "INT":"150"}

[FILTER]
    Name parser
    Match dummy.*
    Key_Name data
    Parser dummy_test
    Reserve_Data On

[OUTPUT]
    Name  stdout
    Match *

[OUTPUT]
    Match                         *
    Name                          stackdriver
    Retry_Limit                   3
    net.connect_timeout_log_error False
    resource                      gce_instance
    stackdriver_agent             Google-Cloud-Ops-Agent-Logging/2.11.0 (BuildDistro=buster;Platform=linux;ShortName=debian;ShortVersion=10.11)
    tls                           On
    tls.verify                    Off
    workers                       8

fluent_bit_main_dummy.conf

[PARSER]
    Name dummy_test
    Format regex
    Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$

I get the following warnings and no output to the LogExplorer (Google Cloud Logging):

Fluent Bit v1.9.2
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/04/07 16:35:46] [ info] [fluent bit] version=1.9.2, commit=2ecaa7b3b8, pid=15752
[2022/04/07 16:35:46] [ info] [storage] version=1.1.6, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/04/07 16:35:46] [ info] [cmetrics] version=0.3.0
[2022/04/07 16:35:46] [ info] [output:stdout:stdout.0] worker #0 started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] metadata_server set to http://metadata.google.internal
[2022/04/07 16:35:46] [ warn] [output:stackdriver:stackdriver.1] client_email is not defined, using a default one
[2022/04/07 16:35:46] [ warn] [output:stackdriver:stackdriver.1] private_key is not defined, fetching it from metadata server
[2022/04/07 16:35:46] [ info] [sp] stream processor started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] worker #6 started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] worker #5 started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] worker #4 started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] worker #3 started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] worker #7 started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] worker #2 started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] worker #1 started
[2022/04/07 16:35:46] [ info] [output:stackdriver:stackdriver.1] worker #0 started
[0] dummy.data: [1649349347.320567991, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2", "INT"=>"150"}]
[2022/04/07 16:35:48] [ warn] [engine] failed to flush chunk '15752-1649349347.320582218.flb', retry in 10 seconds: task_id=0, input=dummy.0 > output=stackdriver.1 (out_id=1)
[0] dummy.data: [1649349348.320772430, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2", "INT"=>"150"}]
[2022/04/07 16:35:49] [ warn] [engine] failed to flush chunk '15752-1649349348.320781563.flb', retry in 10 seconds: task_id=1, input=dummy.0 > output=stackdriver.1 (out_id=1)
[0] dummy.data: [1649349349.320777109, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2", "INT"=>"150"}]
[2022/04/07 16:35:50] [ warn] [engine] failed to flush chunk '15752-1649349349.320786075.flb', retry in 11 seconds: task_id=2, input=dummy.0 > output=stackdriver.1 (out_id=1)

Details

• When using Ctrl-C to close the execution, it shows the following and stalls :

[2022/04/07 17:20:02] [ info] [task]   task_id=9 still running on route(s): stdout/stdout.0 stackdriver/stackdriver.1 
[2022/04/07 17:20:02] [ info] [engine] service has stopped (7 pending tasks)
[2022/04/07 17:20:02] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2022/04/07 17:20:02] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2022/04/07 17:20:02] [ info] [output:stackdriver:stackdriver.1] thread worker #0 stopping...

Expected behavior
Expected to see the observed records from stdout in the LogExplorer.

Your Environment

• Version used: current master v1.9.2
• Server type and version: Google Cloud Engine Virtual Machine
• Operating System and version: Debian GNU/Linux 10 (buster)

Additional context

Ran through this issue while working on developing updates to the out_stackdriver plugin.

[franciscovalentecastro]

Fixed with #5262 .