Metrics (buffer_queue_length and buffer_total_queued_size) not available with detached processes

[comtef]

Fluentd metrics are useful to check for contention in output plugins buffers.
buffer_queue_length and buffer_total_queued_size can be checked against maximum values to ensure that no buffer chunk is about to be dropped.

When using detach_process, the metrics are no more available. Is it possible to agregate metrics for all detached processes?

Detaching processes may be the only way to ensure high throughput because increasing only the number of threads is not enough : 1 process with 50 threads is slower than 5 processes with 10 threads each.

[wryun]

Depending on your use case, you could try https://github.com/atlassian/fluent-plugin-kinesis-aggregation . You would still have to have a single process, but you may find you get more throughput, particularly if you've been having issues with retries. If you do, I'd be interested in any feedback.

[riywo]

This is caused by fluentd itself. I've opend an issue on fluentd GitHub.

So far, it is hard to be fixed by this plugin...

[riywo]

Since fluentd has stopped to support detach_multi_process, I'd like to close this issue. They introduced new worker system and 2.0.0 of this plugin will support it.