Missing logs in LogzIO

[omerlh]

I noticed that some of our logs are missing, so I've enabled debug logging in fluentd. After enabling the debug logs, I noticed the following errors:

2017-12-25 05:54:40 +0000 [debug]: Successfuly sent bulk
2017-12-25 05:54:50 +0000 [debug]: Got flush timeout, containing 25 chunks
2017-12-25 05:54:50 +0000 [debug]: Successfuly sent bulk
2017-12-25 05:55:00 +0000 [debug]: Got flush timeout, containing 14 chunks
2017-12-25 05:55:00 +0000 [debug]: Successfuly sent bulk
2017-12-25 05:55:10 +0000 [debug]: Got flush timeout, containing 27 chunks
2017-12-25 05:55:10 +0000 [debug]: Successfuly sent bulk
2017-12-25 05:55:20 +0000 [debug]: Got flush timeout, containing 14 chunks
2017-12-25 05:55:20 +0000 [debug]: Successfuly sent bulk
2017-12-25 05:55:30 +0000 [debug]: Got flush timeout, containing 14 chunks
2017-12-25 05:55:30 +0000 [debug]: Successfuly sent bulk
2017-12-25 05:55:40 +0000 [debug]: Got flush timeout, containing 15 chunks
2017-12-25 05:55:40 +0000 [debug]: Successfuly sent bulk
2017-12-25 05:55:50 +0000 [debug]: Got flush timeout, containing 25 chunks
2017-12-25 05:55:50 +0000 [debug]: Successfuly sent bulk

After quick google search for Got flush timeout, I've encounter a similar isse (fluent-plugins-nursery/fluent-plugin-concat#4) and that made me think that it's a potential bug. Could you help me figure this out?

[roiravhon]

Hey @omerlh!
Well, this line just indicates that fluentd asked us to ship a bulk, containing X chunks (basically logs).
This is the normal operation of the plugin, look here:
https://github.com/logzio/fluent-plugin-logzio/blob/master/lib/fluent/plugin/out_logzio_buffered.rb#L52

Which logs are you missing? What is your reference point?

[omerlh]

I am asking about the Got flush timeout, containing 14 chunks, it sounds like an error to me?

[roiravhon]

As i said - not an error. timeout on the fluentd buffering (i.e. buffer drain to logz.io)

[omerlh]

ok, thanks