[Feature Request] Buffer Fill Percentage

[cmanning09]

Is your feature request related to a problem? Please describe.
Buffer fill percentage is the primary indicator for horizontal scaling for data prepper. However, the metrics reported by micrometer are with a gauge and with no concept of upper bounds as defined in the pipeline configuration. Determining when to scale and reviewing metrics in a dashboards requires knowledge of the blockingBuffer pipeline configuration buffer size values in relation to the current blockingBuffer.recordsInBuffer metric value.

Describe the solution you'd like
A new metrics for blockingBuffer.bufferUsage` which tracks the utilization rate of the buffer based on the number of records in the buffer and the buffer size.

Describe alternatives you've considered (Optional)

• publishing the buffer size would be an alternative. Users could then perform calculations on their dashboards.

[dlvenable]

I think this is a useful approach and it follows other metrics such as CPU usage. I take it that this going to be a per-buffer metric. Do you think there is value for supporting a more aggregate metric in the future? Perhaps maximum bufferUsage?

[cmanning09]

I was initially thinking per buffer is sufficient enough for now. There may be value in a buffer usage across pipelines however we would need to create a standard metric name across the service. Ideally if we support a more persistent buffer solution we may be able to generalize the buffer across pipelines. Again all for the future though.

[cmanning09]

Should all buffer plugins support this metric? This would require standardizing buffer plugins to have max buffer size or capacity attribute (bufferCapacity). This attribute would allow the AbstractBuffer to calculate the usage percentage based on the recordsInBuffer and bufferCapacity. I am thinking about extending the buffer interface to include a public Integer getBufferCapacity() method. For Example:

private final AtomicLong bufferUsage;
...

     public void write(T record, int timeoutInMillis) throws TimeoutException {

            ...
            final long currentRecords = recordsInBuffer.incrementAndGet();
            bufferUsage.set(currentRecords / this.getBufferCapacity());
            ...

Alternatively I could create this metric just for the blockingBuffer and we can decide if want to standardize later.

[dlvenable]

I think we should start with blockingBuffer for now. If we do extend this behavior, it could be part of AbstractBuffer, but I'd be hesitant to add a buffer capacity to any Buffer. Some buffers might be so large as to have no specific capacity if they used an external system.