Flow fixes/v9 #6438
Merged
Flow fixes/v9 #6438
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In a scenario where there was suddenly no more traffic flowing, flows in a threads `flow_queue` would not be processed. The easiest way to see this would be in a traffic replay scenario. After the replay is done no more packets come in and these evicted flows got stuck. In workers mode, the capture part handles timeout this was updated to take the `ThreadVars::flow_queue` into account. The autofp mode the logic that puts a flow into a threads `flow_queue` would already wake a thread up, but the `flow_queue` was then ignored. This has been updated to take the `flow_queue` into account. In both cases a "capture timeout" packet is pushed through the pipeline to "flush" the queues. Bug: OISF#4722.
Track availability of break loop callback to avoid overhead.
Codecov Report
@@ Coverage Diff @@
## master #6438 +/- ##
==========================================
+ Coverage 76.88% 76.97% +0.09%
==========================================
Files 613 613
Lines 186684 186712 +28
==========================================
+ Hits 143536 143730 +194
+ Misses 43148 42982 -166
Flags with carried forward coverage won't be shown. Click here to find out more. |
The flows exceeding the spare pools config setting would be freed per at max 100 flows a second. After a high speed test this would lead to excessive memory use for a long time. This patch updates the logic to free 10% of the excess flows per run, freeing multiple blocks of flows as needed. Bug: OISF#4731.
victorjulien
force-pushed
the
flow-fixes/v9
branch
from
34d7b42
to
fa72a5a
Compare
jasonish
added a commit
to jasonish/suricata
that referenced
this pull request
Nov 2, 2023
With the change to the hash table for tracking threaded loggers, this call is now called once per thread, so should be changed to the ThreadDeinit, as that is not longer being called. Then call Deinit for the primary logger. In threaded mode this would be the parent, its just the logger in non-threaded mode. Bug: OISF#6438
jasonish
added a commit
to jasonish/suricata
that referenced
this pull request
Nov 2, 2023
With the change to the hash table for tracking threaded loggers, this call is now called once per thread, so should be changed to the ThreadDeinit, as that is not longer being called. Then call Deinit for the primary logger. In threaded mode this would be the parent, its just the logger in non-threaded mode. Bug: OISF#6438
jasonish
added a commit
to jasonish/suricata
that referenced
this pull request
Nov 3, 2023
With the change to the hash table for tracking threaded loggers, this call is now called once per thread, so should be changed to the ThreadDeinit, as that is not longer being called. Then call Deinit for the primary logger. In threaded mode this would be the parent, its just the logger in non-threaded mode. Bug: OISF#6438
jasonish
added a commit
to jasonish/suricata
that referenced
this pull request
Nov 11, 2023
With the change to the hash table for tracking threaded loggers, this call is now called once per thread, so should be changed to the ThreadDeinit, as that is not longer being called. Then call Deinit for the primary logger. In threaded mode this would be the parent, its just the logger in non-threaded mode. Bug: OISF#6438
jasonish
added a commit
to jasonish/suricata
that referenced
this pull request
Nov 11, 2023
With the change to the hash table for tracking threaded loggers, this call is now called once per thread, so should be changed to the ThreadDeinit, as that is not longer being called. Then call Deinit for the primary logger. In threaded mode this would be the parent, its just the logger in non-threaded mode. Bug: OISF#6438
victorjulien
pushed a commit
to victorjulien/suricata
that referenced
this pull request
Nov 14, 2023
With the change to the hash table for tracking threaded loggers, this call is now called once per thread, so should be changed to the ThreadDeinit, as that is not longer being called. Then call Deinit for the primary logger. In threaded mode this would be the parent, its just the logger in non-threaded mode. Bug: OISF#6438
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Misc fixes for freeing flow memory more aggressively after a T-Rex test.