Stream inspection

[rcbarnett-zz]

MODSEC-18: Go beyond the discrete inspection model we currently have implemented, and toward streaming inspection. The idea is that the code would generate a number of streams, each streaming transaction data but in a slightly different way. Examples include:

• Request body (after dechunking and decompression)
• Response body (before dechunking and decompression)
• File content (one stream per uploaded file)

Implementation:

• SecStreamingInspection On|Off - on by default.
• SecStreamMatch TARGETS PATTERNS ACTIONS
• We would reuse variable names for TARGETS.
• PATTERNS is a list of @pm patterns.
• The ACTIONS part would support a limited set of actions (e.g. no flow control ones, no phases).
• We would initially support inspection of raw streams, but, at a later date, we can implement streaming transformation operators to allow for streaming transformation pipelines.
• All instances of SecStreamMatch targeting the same stream (and, later, using the same tfn operators) would be combined into a single matching tree.

Stream inspection would occur in real-time, as the content is being processed. There are two advantages of this approach:

1. We can block on streaming-only mode (MODSEC-17).
2. Makes pre-qualification easier (and faster).

[rcbarnett-zz]

Original reporter: ivanr

[rcbarnett-zz]

ivanr: A more generic implementation would probably be a better choice. For example:

SecStreamInspect REQUEST_BODY "@pm one two" log,pass,t:none

[victorhora]

Already implemented:

https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#REQUEST_BODY
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#RESPONSE_BODY
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#STREAM_INPUT_BODY
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#STREAM_OUTPUT_BODY

[csanders-git]

You are not understanding what stream inspection means

[victorhora]

Thank you @csanders-git. Could you please clarify the meaning of stream inspection on this issue? And please let us know if this a feature that the community is missing.

[csanders-git]

@zimmerle could also easily explain what this means, this means treating HTTP requests incoming as a stream instead of assembling them into a buffered request.

[victorhora]

Nice. Thanks for clarifying @csanders-git. Is this something that the CRS and / or the community are currently missing? If yes, I'll happily tag it for libModSecurity and see if we get some traction, otherwise it should be treated as #304 as it had no update / interest or for the past 5 years.

Thanks for letting us know.