Preview/variables/v18#2468
Closed
inliniac wants to merge 19 commits into
Closed
Conversation
Use engine version based on global detect engine master. This is incremented between reloads.
Until now variable names, such as flowbit names, were local to a detect engine. This made sense as they were only ever used in that context. For the purpose of logging these names, this needs a different approach. The loggers live outside of the detect engine. Also, in the case of reloads and multi-tenancy, there are even multiple detect engines, so it would be even more tricky to access them from the outside. This patch brings a new approach. A any time, there is a single active hash table mapping the variable names and their id's. For multiple tenants the table is shared between tenants. The table is set up in a 'staging' area, where locking makes sure that multiple loading threads don't mess things up. Then when the preparing of a detection engine is ready, but before the detect threads are made aware of the new detect engine, the active varname hash is swapped with the staging instance. For this to work, all the mappings from the 'current' or active mapping are added to the staging table. After the threads have reloaded and the new detection engine is active, the old table can be freed. For multi tenancy things are similar. The staging area is used for setting up until the new detection engines / tenants are applied to the system. This patch also changes the variable 'id'/'idx' field to uint32_t. Due to data structure padding and alignment, this should have no practical drawback while allowing for a lot more vars.
Support up to 8 substring captures into pkt or flow vars.
Flowvars were already using a temporary store in the detect thread ctx. Use the same facility for pktvars. The reasons are: 1. packet is not always available, e.g. when running pcre on http buffers. 2. setting of vars should be done post match. Until now it was also possible that it is done on a partial match.
Logs into alerts and flows. TODO - currently unconditional. Needs to be optional.
Until now the way to specify a var name in pcre substring capture into pkt and flow vars was to use the pcre named substring support: e.g. /(?P<pkt_somename>.*)/ This had 2 drawbacks: 1. limitations of the name. The name could be max 32 chars, only have alphanumeric and the underscore characters. This imposed limitations that are not present in flowbits/ints. 2. we didn't actually use the named substrings in pcre through the API. We parsed the names separately. So putting the names in pcre would actually be wasteful. This patch introduces a new way of mapping captures with names: pcre:"/(.*)/, pkt:somename"; pcre:"/([A-z]+) ([0-9]+)/, pkt:somename,flow:anothername"; The order of the captures and the order of the names are mapped 1 on 1. This method is no longer limited by the pcre API's naming limits. The 'flow:' and 'pkt:' prefixes indicate what the type of variable is. It's mandatory to specify one. The old method is still supported as well.
EVE addition called 'vars' that logs pkt/flow vars for each packet/flow.
Author
|
Replaced by #2470 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preview of the work I presented at SuriCon DC.
Logging of flowbits and other vars. PCRE extraction of key/value pairs into pkt/flowvars.
Additionally: add-hostbit and remove-hostbit unix-socket commands.
Prscript: