[preview] detect grouping v174 #1811
Conversation
Introduce 'ac-ks' or the Kenneth Steele AC implementation. It's actually 'ac-tile' written by Ken for the Tilera platform. This patch adds support for it on other architectures as well. Enable ac-tile for other archs as 'ac-ks'. Fix a bunch of OOB reads in the loops that triggered ASAN.
Use sgh-mpm-context single is it is set to 'auto' when ac-ks is used.
Remove: ac-gfbs, wumanber, b2g, b3g.
Add funcs to see if a rule needs a SYN flag in the packet.
Each SGH has a unique ipproto and direction.
Replace tree based approach for rule grouping with a per port (tcp/udp)
and per protocol approach.
Grouping now looks like:
+----+
|icmp+--->
+----+
|gre +--->
+----+
|esp +--->
+----+
other|... |
+----->-----+
| |N +--->
| +----+
|
| tcp +----+ +----+
+----->+ 80 +-->+ 139+-->
| +----+ +----+
|
| udp +----+ +----+
+---+----->+ 53 +-->+ 135+-->
| +----+ +----+
|toserver
+--->
|toclient
|
+--->
So the first 'split' in the rules is the direction: toserver or toclient.
Rules that don't have a direction, are in both branches.
Then the split is between tcp/udp and the other protocols. For tcp and
udp port lists are used. For the other protocols, grouping is simply per
protocol.
The ports used are the destination ports for toserver sigs and source
ports for toclient sigs.
Allow multi-proto, multi-direction sgh's.
Update port grouping logic. Previously it would create one consistent list w/o overlap. It largely still does this, except for the 'catch all' port group at the end of the list. This port group contains all the sigs that didn't fit into the other groups.
Per rule group tracking of checks, use of lists, mpm matches, post filter counts. Logs SGH id so it can be compared with the rule_group.json output. TODO json output
Dump a json record containing all sigs that need to be inspected after prefilter. Part of profiling. Only dump if threshold is met, which is currently set by: --set detect.profile-match-logging-threshold=200 TODO: threshold setting, yaml TODO: log also IP tuple, to make it more useful in case of online mode
For all mpm wrapper functions, check minlen vs the input buffer to see if we can bypass the mpm search. Next to this, make all the function inline. Also constify the input and do other minor cleanups.
SGH's for tcp and udp are now always only per proto and per direction. This means we can simply reuse the packet and stream mpm pointers. The SGH's for the other protocols already used a directionless catch all mpm pointer.
Create a hash table of unique DetectPort objects before trying to create a unique list of these objects. This safes a lot of cycles in the creation of the list.
Allow setting of shared or unique setting per app buffer type: e.g. detect.mpm.http_uri.shared=true
Instead of detect-engine which used a list for no good reason, use a
simple map now.
detect:
profile: medium
custom-values:
toclient-groups: 3
toserver-groups: 25
sgh-mpm-context: auto
inspection-recursion-limit: 3000
# If set to yes, the loading of signatures will be made after the capture
# is started. This will limit the downtime in IPS mode.
#delayed-detect: yes
|
I have the following set up - |
|
This works as expected. to resulting in:
Sorry for the confusion. |
|
Replaced by #1852 |
Fix yaml parsing segv.
Replaces #1804
PRscript: