internal source + trace level causes message storm #2425
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Build SUCCESS |
|
We discussed the topic IRL, and the conclusion is: there will be an |
When syslog-ng started with trace enabled and internal source, a single message in the internal source triggers infinite messages because the loop detection started later than first triggering message. Signed-off-by: Antal Nemes <antal.nemes@balabit.com>
furiel
force-pushed
the
furiel-msg-infinite-loop
branch
from
2705eb7
to
90e03b7
Compare
Signed-off-by: Antal Nemes <antal.nemes@balabit.com>
When internal source is used and trace level enabled, some messages generate infinite other messages. For example the triggering message adds a new message to the internal message, which is reported in log_source_queue, that will trigger a new message etc. The infinite loop is broken by not generating messages into the internal queue but written to stderr. Signed-off-by: Antal Nemes <antal.nemes@balabit.com>
furiel
force-pushed
the
furiel-msg-infinite-loop
branch
from
90e03b7
to
b71bad9
Compare
furiel
changed the title
|
Build SUCCESS |
|
I still don’t really like this, but I do understand why this is the solution. |
lbudai
approved these changes
Nov 27, 2018
|
@furiel: and thanks for the investigation, also for the detailed commit messages! |
szemere
approved these changes
Nov 28, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem: when internal source is enabled and trace level set, infinite message is generated in an exponential manner.
There are two independent root cause for this, both of these need to be addressed.
There is a loop detection that was not working, because the order msg_set_context and the triggering messages were swapped. With that change, the
Source side message processing beginmessages are suppressed.With that
internal() messages are looping back, preventing loop by ...messages occur, so the detection kicks in, avoiding the infinite loop, but still causing minor inconvenience.There is a message about window size adjustment, that also triggers a message each time a message arrives into the internal queue. However, the loop detection does not work on this, and should not, the window size change is completely valid event in this case.
For now I just deleted the msg_trace, but this is probably suboptimal solution.
I was thinking that maybe trace level messages could be written to stderr unconditionally. To my undrestanding, that would also stop the infinite generation of messages, because the trace messages are not put back into the internal queue.