Implement content inspection that does not require buffering

[rcbarnett-zz]

MODSEC-17: In order to block 100% reliably we need to buffer; there's no question about that. However, buffering is not always the best, or even desired, approach. Some mechanisms (such as file upload progress bars) rely on having a free flow of data, and there also may be latency issues with very large request and response sizes. Furthermore, turning buffering off when in DetectionOnly mode will make ModSecurity more transparent and reduce memory consumption. This ticket outlines a proposal how to improve on our current handling on content inspection in this respect:

1. Add SecRequestBodyBuffering, which can have two settings: On and Off. Have it On by default to preserve backward compatibility.
2. Also add ctl:requestBodyBuffering, to allow rules to inspect request/response headers and make a decision whether to buffer.
3. On the inbound, do not read request bodies in the fixups phase. Instead, install a different input filter that will be invoked as content becomes available.
4. Add SecResponseBodyBuffering, with two settings: On and Off. Have it On by default.
5. Add ctl:responseBodyBuffering.
6. On the outbound, since the current filter is content driven already I think it can be modified to support operation without buffering.
7. Do not block in phases 2 (request buffering disabled) and phase 4 (response buffering disabled).
8. Automatically disable buffering in DetectionOnly mode.
9. Implement a streaming urlencoded parser.
10. Determine when to populate REQUEST_BODY, and how to allow users to control that feature (and if to allow them to control it at all).

[rcbarnett-zz]

Original reporter: ivanr

[zimmerle]

This was an issue back on the 2.x architecture. No longer an issue in 3.0. On 3.0 the phases are independent and the content hit the inspection/delivered to the client whenever it is ready by the webserver. Indeed, for response and request body we are not inspecting on the buffer arrival but when it is fully delivered. This could be a further improvement, as for the other items, i think we have it all covered.