F/merge proto #165
Merged
F/merge proto #165
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
it calls log_source_flow_control_adjust() which doing - flow control rate adjustment and - flow control window size adjustment Signed-off-by: Budai Laszlo <Laszlo.Budai@balabit.com>
late ack mechanism: - late ack is for position tracking - position should be saved only after the LogMessage (corresponding to the logline read from the position we want to save) has ACKed in order to do that: - proto queued() method removed [before the late-ack era queued() has called after the LogMessage forwarded to the LogWriter::queue and this method saved the position immediately -> this means that if the message ACK has aborted we lost that message] - fetch() knows the position of the stream where logline read, we just set a bookmark for that position - if the fetched data is a valid log message, a new LogMessage object is created and the bookmark is bound to that object - when a positive ACK is received for the LogMessage, bookmark is saved Decoupling position save from queued() generates a new problem: it is not guaranteed that the ACKs received in-order (eg.: msg1 -> unmatching filter -> the message is processed -> ACKed msg0 -> destination -> logqueue -> transport -> ACKed ). For solving this problem we have to use a Ringbuffer storing AckRecords. AckRecord = (Bookmark, ack status). (Number of slots in Ringbuffer is proportional to window size; when window is full, no new message is received, thus no new Bookmark is needed) For every incoming message an AckRecord is bound when it is queued to logsource(log_source_queue() called, this is the point when we know that the fetched line is a valid logmessage, Bookmark is filled, it is ready for forwarding to the consumer's queue). When a continuous range of positive ACK is formed at the head of the RingBuffer, Bookmark of the AckRecord with the highest position in the range is saved and the range is dropped from the RingBuffer (-> the ringbuffer head is reset by range length, window size also increased by range-length). Tracking ACKs: AckTracker Using AckTracker for reliable cases is OK but what about "unreliable" cases where no position tracking supported? There are two kind of AckTrackers: a reliable and a non-reliable. Non-reliable AckTracker just not use a RingBuffer, not saving a Bookmark, of course, and window size increased by 1. Signed-off-by: Budai Laszlo <Laszlo.Budai@balabit.com>
Signed-off-by: Budai Laszlo <Laszlo.Budai@balabit.com>
…ta pointer LogMsg held a reference to a LogSource instance. This is not needed when AckTracker is used. Signed-off-by: Budai Laszlo <Laszlo.Budai@balabit.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.