New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
flow worker v39 #2089
Merged
Merged
flow worker v39 #2089
Commits on May 20, 2016
-
detect: split detect entry into flow/noflow
This is a preparation for flow locking updates.
Configuration menu - View commit details
-
Copy full SHA for a81766c - Browse repository at this point
Copy the full SHA a81766cView commit details -
flow: get flow reference during lookup
Update Flow lookup functions to get a flow reference during lookup. This reference is set under the FlowBucket lock. This paves the way to not getting a flow lock during lookups.
Configuration menu - View commit details
-
Copy full SHA for ae7aae8 - Browse repository at this point
Copy the full SHA ae7aae8View commit details -
flow: move flow handling into worker threads
Instead of handling the packet update during flow lookup, handle it in the stream/detect threads. This lowers the load of the capture thread(s) in autofp mode. The decoders now set a flag in the packet if the packet needs a flow lookup. Then the workers will take care of this. The decoders also already calculate the raw flow hash value. This is so that this value can be used in flow balancing in autofp. Because the flow lookup/creation is now done in the worker threads, the flow balancing can no longer use the flow. It's not yet available. Autofp load balancing uses raw hash values instead. In the same line, move UDP AppLayer out of the DecodeUDP module, and also into the stream/detect threads. Handle TCP session reuse inside the flow engine itself. If a looked up flow matches the packet, but is a TCP stream starter, check if the ssn needs to be reused. If that is the case handle it within the lookup function. Simplies the locking and removes potential race conditions.
Configuration menu - View commit details
-
Copy full SHA for 2f0e0f1 - Browse repository at this point
Copy the full SHA 2f0e0f1View commit details -
time: improve offline time handling
When we run on live traffic, time handling is simple. Packets have a timestamp set by the capture method. Management threads can simply use 'gettimeofday' to know the current time. There should never be any serious gap between the two or major differnces between the threads. In offline mode, things are dramatically different. Here we try to keep the time from the pcap, which means that if the packets are recorded in 2011 the log output should also reflect this. Multiple issues: 1. merged pcaps might have huge time jumps or time going backward 2. slowly recorded pcaps may be processed much faster than their 'realtime' 3. management threads need a concept of what the 'current' time is for enforcing timeouts 4. due to (1) individual threads may have very different views on what the current time is. E.g. T1 processed packet 1 with TS X, while T2 at the very same time processes packet 2 with TS X+100000s. The changes in flow handling make the problems worse. The capture thread no longer handles the flow lookup, while it did set the global 'time'. This meant that a thread may be working on Packet 1 with TS 1, while the capture thread already saw packet 2 with TS 10000. Management threads would take TS 10000 as the 'current time', considering a flow created by the first thread as timed out immediately. This was less of a problem before the flow changes as the capture thread would also create a flow reference for a packet, meaning the flow couldn't time out as easily. Packets in the queues between capture thread and workers would all hold such references. The patch updates the time handling to be as follows. In offline mode we keep the timestamp per thread. If a management thread needs current time, it will get the minimum of the threads' values. This is to avoid the problem that T2s time value might already trigger a flow timeout as the flow lastts + 100000s is almost certainly meaning the flow would be considered timed out.
Configuration menu - View commit details
-
Copy full SHA for 6f56014 - Browse repository at this point
Copy the full SHA 6f56014View commit details -
To simplify locking, move all locking out of the individual detect code. Instead at the start of detection lock the flow, and at the end of detection unlock it. The lua code can be called without a lock still (from the output code paths), so still pass around a lock hint to take care of this.
Configuration menu - View commit details
-
Copy full SHA for 4089488 - Browse repository at this point
Copy the full SHA 4089488View commit details -
Initial version of the 'FlowWorker' thread module. This module combines Flow handling, TCP handling, App layer handling and Detection in a single module. It does all flow related processing under a single flow lock.
Configuration menu - View commit details
-
Copy full SHA for 52d500c - Browse repository at this point
Copy the full SHA 52d500cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 61ce05e - Browse repository at this point
Copy the full SHA 61ce05eView commit details -
Now that the flow lookup is done in the worker threads the flow queue handlers running after the capture thread(s) no longer have access to the flow. This limits the options of how flow balancing can be done. This patch removes all code that is now useless. The only 2 methods that still make sense are 'hash' and 'ippair'.
Configuration menu - View commit details
-
Copy full SHA for 78ecfe8 - Browse repository at this point
Copy the full SHA 78ecfe8View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.