RFC Feature/60x/decoder data store/v8 #4977
Conversation
A fixed size static store is part of the Packet structure. If that space isn't sufficient the store can dynamically expand. It uses a list of storage blocks for this. The primary goals are: - don't use space in the packet when a feature/proto isn't in use - reduce (or avoid) the need for new protocols to manually add to the Packet structure In general the storage is a TLV (type length value) store, but some types get special support to reduce the space overhead of the TLV header. Implemented so far: - VLAN (2 bytes) - MPLS (4 bytes) - TLV (3 bytes + the data to be stored)
I think its important to have this same information in flow records as well. Not sure I like how generic this is to get at this fields under the decoder_store. I also feel like that "decoder_store" name should be an internal implementation detail and not be exposed in the JSON. Is there a better name we can use here? |
| } | ||
|
|
||
|
|
||
| typedef union DecodeStoreBuiltinVLAN { |
There was a problem hiding this comment.
Not liking the "Builtin" in the name too much. "DecoderStoreTypeVlan", "DecodeStoreTypeGeneric", etc.
| { | ||
| assert(size >= 1); | ||
|
|
||
| *type = *data & 0x0f; |
There was a problem hiding this comment.
Will 16 enough for types ?
There was a problem hiding this comment.
the idea was that there are some built-in types and then a flexible general purpose TLV type that allows 256 subtypes. The builtins should be extra compact/efficient.
|
What is the status on this ? |
Inspect individual chunks in lossy traffic. Don't use the frame idx as the inspection buffer idx. Engines are running per frame, so multi inspection can be used for stream chunks instead. Ticket: OISF#4977.
Inspect individual chunks in lossy traffic. Don't use the frame idx as the inspection buffer idx. Engines are running per frame, so multi inspection can be used for stream chunks instead. Ticket: OISF#4977.
Inspect individual chunks in lossy traffic. Don't use the frame idx as the inspection buffer idx. Engines are running per frame, so multi inspection can be used for stream chunks instead. Ticket: OISF#4977.
|
Bump -- this would be interesting/useful to me as well as I'm looking at MPLS logging atm and I already felt guilty for adding more fields to the packet and flow structs in my branch (https://github.com/satta/suricata/tree/mpls) ;) I agree that the output should be different, more in line with current EVE structure and not expose how things are stored. |
|
I am also still curious what the status is here. |
|
@satta, not sure. I think it needs a rebase and I also really dislike the macro hack (TOP/BOT) think I did... feel free to play around with it. |
|
Yeah, quite a big rebase as quite a bit seems to have changed in the packet department. I'll see when i can find the time to do the rebase (and hopefully get a better idea of the change in the process). |
Initial work on 'decoder store', where the packet fields stored in the
Packetare more dynamic.From efaf3bc
A fixed size static store is part of the Packet structure. If that space
isn't sufficient the store can dynamically expand. It uses a list of
storage blocks for this.
The primary goals are:
Packet structure
In general the storage is a TLV (type length value) store, but some types
get special support to reduce the space overhead of the TLV header.
Implemented so far:
Right now the only 'consumer' is EVE, where the current PR adds
and
Looking for general feedback. Also wondering if this should be extended to the
Flow, and perhaps somehow even be taken into account for flow hash lookups.