Next 20160520 v6 #2091
Merged
Next 20160520 v6 #2091
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a preparation for flow locking updates.
Update Flow lookup functions to get a flow reference during lookup. This reference is set under the FlowBucket lock. This paves the way to not getting a flow lock during lookups.
Instead of handling the packet update during flow lookup, handle it in the stream/detect threads. This lowers the load of the capture thread(s) in autofp mode. The decoders now set a flag in the packet if the packet needs a flow lookup. Then the workers will take care of this. The decoders also already calculate the raw flow hash value. This is so that this value can be used in flow balancing in autofp. Because the flow lookup/creation is now done in the worker threads, the flow balancing can no longer use the flow. It's not yet available. Autofp load balancing uses raw hash values instead. In the same line, move UDP AppLayer out of the DecodeUDP module, and also into the stream/detect threads. Handle TCP session reuse inside the flow engine itself. If a looked up flow matches the packet, but is a TCP stream starter, check if the ssn needs to be reused. If that is the case handle it within the lookup function. Simplies the locking and removes potential race conditions.
When we run on live traffic, time handling is simple. Packets have a
timestamp set by the capture method. Management threads can simply
use 'gettimeofday' to know the current time. There should never be
any serious gap between the two or major differnces between the
threads.
In offline mode, things are dramatically different. Here we try to keep
the time from the pcap, which means that if the packets are recorded in
2011 the log output should also reflect this. Multiple issues:
1. merged pcaps might have huge time jumps or time going backward
2. slowly recorded pcaps may be processed much faster than their
'realtime'
3. management threads need a concept of what the 'current' time is for
enforcing timeouts
4. due to (1) individual threads may have very different views on what
the current time is. E.g. T1 processed packet 1 with TS X, while T2
at the very same time processes packet 2 with TS X+100000s.
The changes in flow handling make the problems worse. The capture thread
no longer handles the flow lookup, while it did set the global 'time'.
This meant that a thread may be working on Packet 1 with TS 1, while the
capture thread already saw packet 2 with TS 10000. Management threads
would take TS 10000 as the 'current time', considering a flow created by
the first thread as timed out immediately.
This was less of a problem before the flow changes as the capture thread
would also create a flow reference for a packet, meaning the flow
couldn't time out as easily. Packets in the queues between capture
thread and workers would all hold such references.
The patch updates the time handling to be as follows.
In offline mode we keep the timestamp per thread. If a management thread
needs current time, it will get the minimum of the threads' values. This
is to avoid the problem that T2s time value might already trigger a flow
timeout as the flow lastts + 100000s is almost certainly meaning the
flow would be considered timed out.
To simplify locking, move all locking out of the individual detect code. Instead at the start of detection lock the flow, and at the end of detection unlock it. The lua code can be called without a lock still (from the output code paths), so still pass around a lock hint to take care of this.
Initial version of the 'FlowWorker' thread module. This module combines Flow handling, TCP handling, App layer handling and Detection in a single module. It does all flow related processing under a single flow lock.
Now that the flow lookup is done in the worker threads the flow queue handlers running after the capture thread(s) no longer have access to the flow. This limits the options of how flow balancing can be done. This patch removes all code that is now useless. The only 2 methods that still make sense are 'hash' and 'ippair'.
Add a new API to store data from streaming sources, like HTTP body processing or TCP data. Currently most of the code uses a pattern of list of data chunks (e.g. TcpSegment) that is reassembled into a large buffer on-demand. The Streaming Buffer API changes the logic to store the data in reassembled form from the start, with the segments/chunks pointing to the reassembled data. The main buffer storing the data slides forward, automatically or manually. The *NoTrack calls allows for a segmentless mode of operation. This approach has two main advantages: 1. accessing the reassembled data is virtually cost-free 2. reduction of allocations and memory management
Convert HTTP body handling to use the Streaming Buffer API. This means the HtpBodyChunks no longer maintain their own data segments, but instead add their data to the StreamingBuffer instance in the HtpBody structure. In case the HtpBodyChunk needs to access it's data it can do so still through the Streaming Buffer API. Updates & simplifies the various users of the reassembled bodies: multipart parsing and the detection engine.
The HTPCfgDir structure is meant to contain config for per direction body parsing parameters. This patch stores the streaming API config.
Enforce inspect window also in IDS mode. Try always to get at least 'inspect win' worth of data. In case there is more new data, take some of the old data as well to make sure there is always some overlap. This unifies IDS and IPS modes, the only difference left is the start of inspection. IDS waits until min_size is available, IPS starts right away.
Make the file storage use the streaming buffer API. As the individual file chunks were not needed by themselves, this approach uses a chunkless implementation.
This will handle minimal DetectEngineCtx structures (used in delayed detect mode) safely, since they don't get SPM global contexts allocated. Also added BUG_ON checks for valid spm_table entries.
No need for cooked header in the case of mmap capture.
This patch adds a basic implementation of AF_PACKET tpacket v3. It is basic in the way it is only working for 'workers' runnning mode. If not in 'workers' mode there is a fallback to tpacket_v2. Feature is activated via tpacket-v3 option in the af-packet section of Suricata YAML.
Reorder fields in AFPThreadVars and suppress some that were not used elsewhere than in the initialization.
Suppress useless fields in AFPThreadVars. This patch also get rid of bytes counter as it was only used to display a message at exit. Information on livedev and on packet counters are enough.
Error handling was not done. The implementation is making the choice to consider we must detroy the socket in case of parsing error. The same was done for tpacket_v2.
It is used to set the block size in tpacket_v3. It will allow user to tune the capture depending on his bandwidth. Default block size value has been updated to a bigger value to allow more efficient wlak on block.
Block timeout defines the maximum filling duration of a block.
If TPACKET_V3 is not defined then it is not available and we should not build anything related to tpacket_v3. This will allow us to activate it dy default and fallback to v2 if not available.
Update the code to use mmap capture by default even in unset in configuration file. mmap capture is now be turned off by using explicitely 'use-mmap: no' in configuration.
We can loose packets during setup because we are reading nothing. So it is logical to discard the counter at start of capture to start from a clean state. This means we don't need to account the drop at start. But the stats call that will reset the drop counts will also return and reset the packets count. So we need to know how many packets we really have. This is in fact the number of packets coming from the stats call minus the number of discarded packets and the drop count. All the other packets will have to be read.
Only parse them if mmap is activated.
As we only use the second we don't need GetTime() which is slower and get us milliseconds.
This was referenced May 20, 2016
Closed
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Contains:
Prscript: