Metrics (buffer_queue_length and buffer_total_queued_size) not available with detached processes

[riywo]

When detach_process is used in BufferedOutput, children have its own buffer in my understanding. So, monitor_agent can't find buffer_queue_length and buffer_total_queued_size for that plugin.

Do you have any plan to support buffer metrics for DetachProcessMixin and DetachMultiProcessMixin by monitor_agent?

One of my suggestion is here:

    {
      "plugin_id": "object:3fea11e14868",
      "plugin_category": "output",
      "type": "kinesis_streams",
      "config": ...,
      "output_plugin": true,
      "detach_processes": [
        {
          "buffer_queue_length": 0,
          "buffer_total_queued_size": 0
        }
      ],
      "retry_count": 0
    }

See also awslabs/aws-fluent-plugin-kinesis#45

[repeatedly]

Hmm... I see.
In v0.12, fixing this issue seems hard.

@tagomoris Can we fix this problem in new plugin API since v0.14?

[tagomoris]

I have no idea about it, but I think I can consider about it after implementing whole new APIs.
But I have a question: Why detach_process is needed for output plugins? num_threads is good enough to make I/O requests concurrent.

[riywo]

I think multiple buffers and no giant lock by detach_process will help to gain high throughput since output plugin needs some computation in some case.

[tagomoris]

This issue is related with #858