prefilter engines v99g #2310

Add SigMatchGetLastSM which simply returns the very last SM added to the signature. Minor cleanups.

It's unused by all of the implementations.

In preparation of the introduction of more general purpose prefilter engines, rename PatternMatcherQueue to PrefilterRuleStore. The new engines will fill this structure a similar way to the current mpm prefilters.

Rename to non_pf: non prefilter.

New name is SignatureNonPrefilterStore to reflect that it's not just about MPM anymore.

Introduce abstraction layer for prefilter engines.

Inspect partial request line as well.

Register for both regular headers and trailers.

Register for both regular headers and trailer.

Introduce prefilter keyword to force a keyword to be used as prefilter. e.g. alert tcp any any -> any any (content:"A"; flags:R; prefilter; sid:1;) alert tcp any any -> any any (content:"A"; flags:R; sid:2;) alert tcp any any -> any any (content:"A"; dsize:1; prefilter; sid:3;) alert tcp any any -> any any (content:"A"; dsize:1; sid:4;) In sid 2 and 4 the content keyword is used in the MPM engine. In sid 1 and 3 the flags and dsize keywords will be used.

If there are many rules for TCP flags these rules would be inspected against each TCP packet. Even though the flags check is not expensive, the combined cost of inspecting multiple rules against each and every packet is high. This patch implements a prefilter engine for flags. If a rule group has rules looking for specific flags and engine for that flag or flags combination is set up. This way those rules are only inspected if the flag is actually present in the packet.

Rules for the 'ack' keyword are uncommon, but if used inspected against almost every packet.

Enable prefilter support for the dsize keyword.

Many of the packet engines are very generic. Rules are generally more limited. A rule like 'alert tcp any any -> any 888 (flags:S; sid:1;)' would still be inspected against every SYN packet in most cases (it depends a bit on rule grouping though). This extra match logic adds an additional check to these packet engines. It can add a check based on alproto, source port and dest port. It uses only one of these 3. Priority order is src port > alproto > dst port. For the ports only 'single' ports are used at this time.

Instead of a type per buffer type, pass just 3 possible types: packet, stream, state. The individual types weren't used. State is just there to be not packet and not stream.

Register keywords globally at start up. Create a map of the registery per detection engine. This we need because the sgh_mpm_context value is set per detect engine. Remove APP_MPMS_MAX.

Allow for duplicate registrations for the same list. After the first registration new calls will be ignored.

Implemented as 'stickybuffer'. Move all logic into the keyword file and remove bad tests that tested URI instead of request line.

Implemented as 'stickybuffer'.

List the common non-buffer specific flags on top.

Inspect engines are called per signature per sigmatch list. Most wrap around DetectEngineContentInspection, but it's more generic. Until now, the inspect engines were setup in a large per ipproto, per alproto, per direction table. For stateful inspection each engine needed a global flag. This approach had a number of issues: 1. inefficient: each inspection round walked the table and then checked if the inspect engine was even needed for the current rule. 2. clumsy registration with global flag registration. 3. global flag space was approaching the need for 64 bits 4. duplicate registration for alprotos supporting both TCP and TCP (DNS). This patch introduces a new approach. First, it does away with the per ipproto engines. This wasn't used. Second, it adds a per signature list of inspect engine containing only those engines that actually apply to the rule. Third, it gets rid of the global flags and replaces it with flags assigned per rule per engine.

Make it more in line with MPM registration.

Move engine and registration into the keyword file. Register as 'ALPROTO_UNKNOWN' instead of per alproto. The registration will only apply it to those rules that have events set.

The order of keyword registration currently affects inspect engine registration order and ultimately the order of inspect engines per rule. Which in turn affects state keeping. This patch makes sure the ordering is the same as with older releases.

Instead of the linked list of engines setup an array with the engines. This should provide better locality. Also shrink the engine structure so that we can fit 2 on a cacheline. Remove the FreeFunc from the runtime engines. Engines now have a 'gid' (global id) that can be used to look up the registered Free function.

Currently the regular 'Header' inspection code will run each time after the HTTP progress moved beyond 'headers'. This will include the trailers if there are any. Leave the code in place as this model will change in the not too distant future.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prefilter engines v99g #2310

prefilter engines v99g #2310

Commits on Sep 29, 2016