AsyncDisruptorAppender not releasing events "fast enough" when receiving batches of events

[brenuart]

Consider the following scenario:

1. A RingBuffer with a size of 4 elements
2. Append 4 events in a raw to fill the buffer - these 4 events will form a batch
3. At this point, the buffer is full and new events will be dropped
4. Unlock the EventHandler and let it process the first of the 4 events out of the queue.
5. We now expect the buffer to contain only 3 events and there should be enough room to accept another. However, any attempt to append a new event will result in a drop (as indicated by the warn status message logged by the appender).

This behaviour due to the Disruptor BatchingEventProcessor updating the sequence (and releasing space in the buffer) only after the entire batch is processed. See https://github.com/LMAX-Exchange/disruptor/blob/3.4.4/src/main/java/com/lmax/disruptor/BatchEventProcessor.java#L165-L172, especially the last line...

This means that space is reserved in the RingBuffer longer that needed. The retained space is proportional to the batch size, and therefore the logging rate of the application.

The Disruptor library comes with the SequenceReportingEventHandler interface for this exact purpose. It allows event handlers to notify the batch processor when it has finished to process an event by updating the sequence accordingly.

Another interesting comment found around the net:

The Disruptor will only update the processed sequence for that event handler until the batch is complete (the default is a batch that is the size of all of the available messages). This reduces the number of updates that it needs to make to the sequence variable used by the publisher to determine if space is available. If you need to make space available earlier than the default then you can do that using a SequenceReportingEventHandler.

--> see how updating the sequence after each event effectively affects performance. If needed, only update the sequence after x events.