Minimize get byte multipart and fix buffer reuse

[fredericBregier]

Fix HttpPostMultipartRequestDecoder buffer usages and allocations

Motivation:

Method getByte(position) is too often called within the current implementation
of the HttpPostMultipartRequestDecoder.
This implies too much activities which is visible when PARANOID mode is active.
This is also true in standard mode.

Apply the same fix on buffer from HttpPostMultipartRequestDecoder to HttpPostStandardRequestDecoder
made previously.

Finally in order to ensure we do not rewrite already decoded HttpData when decoding
next ones within multipart, we must ensure the buffers are copied and not a retained slice.

Modifications:

Use the bytesBefore(...) method instead of getByte(pos) in order to limit the external
access to the underlying buffer by retrieving iteratively the beginning of a correct start
position.
It is used to find both LF/CRLF and delimiter.
2 methods in HttpPostBodyUtil were created for that.

The undecodedChunk is copied when adding a chunk to a DataMultipart is loaded.
The same buffer is also rewritten in order to release the copied memory part.

Result:

Just for note, for both Memory or Disk or Mixed mode factories, the release has to be done as:

for (InterfaceHttpData httpData: decoder.getBodyHttpDatas()) {
    httpData.release();
    factory.removeHttpDataFromClean(request, httpData);
}
factory.cleanAllHttpData();
decoder.destroy();

The memory used is minimal in Disk or Mixed mode. In Memory mode, a big file is still
in memory but not more in the undecodedChunk but its own buffer (copied).

In terms of benchmarking, the results are:

Original code Benchmark                                                             Mode  Cnt  Score    Error   Units
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigAdvancedLevel   thrpt    6  0,152 ±  0,100  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigDisabledLevel   thrpt    6  0,543 ±  0,218  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigParanoidLevel   thrpt    6  0,001 ±  0,001  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigSimpleLevel     thrpt    6  0,615 ±  0,070  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighAdvancedLevel  thrpt    6  0,114 ±  0,063  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighDisabledLevel  thrpt    6  0,664 ±  0,034  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighParanoidLevel  thrpt    6  0,001 ±  0,001  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighSimpleLevel    thrpt    6  0,620 ±  0,140  ops/ms

New code Benchmark                                                                  Mode  Cnt  Score   Error   Units
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigAdvancedLevel   thrpt    6  4,253 ± 0,333  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigDisabledLevel   thrpt    6  4,422 ± 0,250  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigParanoidLevel   thrpt    6  0,877 ± 0,014  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigSimpleLevel     thrpt    6  4,151 ± 0,481  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighAdvancedLevel  thrpt    6  2,167 ± 0,098  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighDisabledLevel  thrpt    6  2,520 ± 0,043  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighParanoidLevel  thrpt    6  0,177 ± 0,003  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighSimpleLevel    thrpt    6  2,419 ± 0,061  ops/ms

In short, using big file transfers, this is about 7 times faster with new code, while
using high number of HttpData, this is about 4 times faster with new code when using Simple Level.
When using Paranoid Level, using big file transfers, this is about 800 times faster with new code, while
using high number of HttpData, this is about 170 times faster with new code.

Small improvements are also added, concerning buffer allocation and unsueful check within MixedAttribute and charset extended usage.

Add some tests to check consistency in Http Codec Multipart

Motivation:

Underlying buffer usages might be erroneous when releasing them internaly
in HttpPostMultipartRequestDecoder.

2 bugs occurs:

1. Final File upload seems not to be of the right size
2. Memory, even in Disk mode, is increasing continuously, while it shouldn't

Modification:

Add some tests to check consistency for HttpPostMultipartRequestDecoder.
Add a package protected method for testing purpose only.

Results:

Without fixes, those tests failed. With the fixes, they passed.

[fredericBregier]

I try to rewrite from the beginning the previous proposal.
I believe it is clearer and also I found some bugs still there (decoding in error, memory allocation in error) in the current actual implementation.

[franz1981]

I see that I should have used this PR benchmark to test the effectiveness of #10737 in a more realistic use case too :)

[fredericBregier]

I see that I should have used this PR benchmark to test the effectiveness of #10737 in a more realistic use case too :)

@franz1981
I'm not sure to what point you think it is relevant but if it is using getByte when iterating on a position to find a value is quite consuming (mainly due to access being partially or totally logged within Netty to find bad releasing, but not only limited to that point): using bytesBefore seems more efficient.
If this is this point, yes, trying to not iterate on getByte is quite a good way (as it is in "almost" every language on file for instance to not read byte after byte).

In addition, we could implement a specific method within ByteBuf that allows to search for a "serie of bytes", but I think it might be too specific in my case so I decide to let this within HttpCodec as a static helper method.

[franz1981]

I'm not sure to what point you think it is relevant

If you will use bytesBefore or indexOf you will likely get an additional speedup thanks to the PR I have linked ;)

[normanmaurer]

@fredericBregier does this replace #10982 ?

[fredericBregier]

@normanmaurer yes, I forgot to close the previous one.
It is a rewritten for both original #10623 and following ones.

[chrisvest]

I had some comments.

[chrisvest]

I forgot to mention: the benchmarks look great, and I like how many parts of the code became simpler. Nice work!

[chrisvest]

Had another comment, but otherwise I think this looks good.

[fredericBregier]

@franz1981 Done, thank you !!

[chrisvest]

@fredericBregier Thanks! (sorry about the delay)

[jameskleeh]

This change broke the semantics of the decoder API. Will file an issue