Plugin specific metric collection framework

[ph]

I think the easiest part of the code we can instrument is probably the pipeline itself and this will give us a good idea how a plugin perform in real world. One way to do it is to instrument the bounderies of the plugins which are the method interactings with the SizeQueue, the filter and the receive methods.
This is options require the least invasive changes and can be done in a backward compatible way.

Most the interaction or useful metrics concerning plugin are related to throughput of a specific plugin.

To correctly instrument a plugin we must be sure that every plugins has a unique identifier. Ref: #3892, #3893

input

I suggest we change how we are sending the queue to the plugin, instead of sending directly the SizeQueue to #run method we can send an instrumented SizeQueue and the code would look like something like this:

queue = InstrumentedSizeQueue.new(queue, :id => plugin.unique_id)
plugin.run(queue)

We can easily instrument push of events and probably the SizeQueue's contention lock?
Since the current SizeQueue is only working with Events we could track when a specific event enter or leave a specific queue.

Filter

The majority of filter don't do any buffering, beside the multiline filter but since we are dealing with current time when we yield a metrics we can also track the flush.

ifilter = InstrumentedFilter.new(filter)
ifilter.filter(event)
ifilter.flush() # This logic is similar to the `batch_acks` below

output

The problem with the output is we don't give any feedback up to the pipeline if all the events were correctly written to disk or send to ES.
If even worse if the plugin is doing some kind of buffering, the output base class need to provide a way to record the metrics when we correctly send a event.

# The pipeline knows when an events get in.
receive(event)

# the pipeline need to know if we successfully send a event.
ack(event)
batch_acks(events)

The output also need a way to track dropped events and/or failures with a specific message so we can track different error scenario.

[ph]

I think metrics tracing in the conditionals should be done when doing the refactor of the ast #3774

[jsvd]

I'd probably include a nak in the outputs, we can often enough know that we gave up on an event or discarded it for some reason

[ph]

The nak is more like an error count, I think this should be a separate concept no?

[jsvd]

not sure what you mean by separate concept, but at least it should likely be reported with a similar interface

[ph]

By separate concept, I should have said we need more information for theses kind of stats.

# not actual syntax.
drop_event(event, :timeout_error)
drop_event(event, :invalid_document)

[ph]

updated with your comment.