Compression middleware

[tpaschalis]

Which problem is this PR solving?

This PR resolves #223 Add optional gzip compression for HTTP responses. I'm using the work of @KenanBek on #232 as a base.

The idea is to support the gzip and deflate compression schemes, as defined in section 3.5 of the HTTP/1.1 RFC with a clean up the public API, while also limiting the potential for errors on the user side.

Short description of the changes

• Define a unified compression middleware
• Use the builder pattern for creating and defining the middleware properties
• Add tests both for the builder and the functionality
• Introduce some new constants in the encoding package

Notes

• Be ruthless in your review :) A fresh look might give new ideas on how to make things better
• Add example
• Add more extensive tests, for the other compression algorithms as well

[codecov]

Codecov Report

Merging #270 (f681aa5) into master (587a2c1) will decrease coverage by 0.03%.
The diff coverage is 73.77%.

@@            Coverage Diff             @@
##           master     #270      +/-   ##
==========================================
- Coverage   72.17%   72.14%   -0.04%     
==========================================
  Files          61       61              
  Lines        3217     3278      +61     
==========================================
+ Hits         2322     2365      +43     
- Misses        804      820      +16     
- Partials       91       93       +2     

Impacted Files	Coverage Δ
component/http/component.go	94.07% <64.28%> (-3.45%)	⬇️
component/http/middleware.go	83.76% <75.00%> (-2.68%)	⬇️
client/http/http.go	92.85% <81.81%> (-2.70%)	⬇️
component/async/kafka/simple/duration_client.go	92.72% <0.00%> (-3.64%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 587a2c1...f681aa5. Read the comment docs.

[mantzas]

Do we need to work on the client also for this meaning uncompress the payload?

[tpaschalis]

Implementing the HTTP client reading of compressed data is quite simple; a POC can be seen here.

The only issue is that the LZW algorithm requires knowing the parameters that were used when compressing (order and litWidth) for decompressing correctly. In practice, these are inferred by the filetype, but this doesn't fit our use-case.

We could utilize two additional headers for LZW-compressed requests, so the receiving end knows what to do with that information. What do you think?

[mantzas]

Implementing the HTTP client reading of compressed data is quite simple; a POC can be seen here.

The only issue is that the LZW algorithm requires knowing the parameters that were used when compressing (order and litWidth) for decompressing correctly. In practice, these are inferred by the filetype, but this doesn't fit our use-case.

We could utilize two additional headers for LZW-compressed requests, so the receiving end knows what to do with that information. What do you think?

I would suggest to keep the scope small and not add LZW which also needs soem new headers. This way we add 2 basic compression algos, which are the most common ones, and keep LZW for a future addition. What do you think?

[tpaschalis]

I would suggest to keep the scope small and not add LZW which also needs soem new headers. This way we add 2 basic compression algos, which are the most common ones, and keep LZW for a future addition. What do you think?

Well, it's either this or hardcoding the LZW parameters which might be confusing for other clients interacting with Patron.

So yes, I'd say I agree, let's remove LZW for now, and see how other OSS servers and clients handle this (eg. if they rely on file type).

@drakos74 what's your view on this?

[mantzas]

I propose the compression middleware to be always created inside patron HTTP component, much like the other middlewares already present e.g. recovery, caching etc.
The only thing that we need is a builder function for the ignore routes in the service builder.
This way the API is minimal and there is no need to pipe into patron the middleware at all...
No need for a builder etc... much simpler and less code.

[tpaschalis]

@mantzas Hm, I think you're right, I'll do that.

Do you think we should also hardcode the compression level for 'deflate' to the maximum value to keep the API minimal? I'd think towards that.

For modern hardware and usual sizes of HTTP transports, the differences are miniscule.
For larger transport sizes, the speed/size trade-off is more apparent.

Here's some benchmarks I've run

size \ deflate |   lvl 1         lvl 9
------------------------------------------
1KiB data      |  0.18ms        0.17ms
1MiB data      |   3.5ms          23ms
1GiB data      |    4.5s           24s

[mantzas]

@tpaschalis could we provide this parameter as a Method in the service builder? This way we can have a sane default and also support configuring the value...