[MRG+1] [httpcompression] add support for br - brotli content encoding

[pawelmhm]

brotli encoding is used by multiple websites, e.g. amazon started to use it recently. Support in Scrapy is added via python library brotlipy https://github.com/python-hyper/brotlipy/

[redapple]

Does the library require anything special/additional to get installed properly?
Maybe importing brotlipy can be tested at inittime/runtime and support for br only advertized in headers if succesful?

[pawelmhm]

Does the library require anything special/additional to get installed properly?

I didnt have to install anything, but it seems like some environments are failing to install, so most likely some systems are missing required libraries. Testing for support at runtime may be much safer yes, I'll add that @redapple

[codecov-io]

Codecov Report

Merging #2535 into master will increase coverage by -0.01%.

@@            Coverage Diff             @@
##           master    #2535      +/-   ##
==========================================
- Coverage   83.49%   83.49%   -0.01%     
==========================================
  Files         161      161              
  Lines        8787     8795       +8     
  Branches     1289     1290       +1     
==========================================
+ Hits         7337     7343       +6     
- Misses       1203     1205       +2     
  Partials      247      247

Impacted Files	Coverage Δ
scrapy/downloadermiddlewares/httpcompression.py	88.88% <77.77%> (-3.01%)	❌

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b8e6d4...fb4ef21. Read the comment docs.

[redapple]

@pawelmhm , I do not see brotlipy being installed for Python 3.x tests on Travis.

[pawelmhm]

@redapple sorry missed that updated in af802ba

[kmike]

pep8 says 1 line between import groups :)

[pawelmhm]

good point, we could add flake8 style checks for PR. Flake8 could run on diff and point out all things like this.

[kmike]

I'd prefer not to pin brotlipy version here: users will be installing it with 'pip install brotlipy', and we won't be able to detect if a brotlipy upgrade broke scrapy if we pin a version in tests.

[pawelmhm]

updated fb4ef21

[kmike]

The PR looks good, besides a couple of minor points 👍 A nice feature.
Have you compared gzip and brotli for amazon? How's the download size and CPU usage?

[pawelmhm]

@kmike re performance there are good benchmarks and charts here: https://quixdb.github.io/squash-benchmark/#ratio-vs-compression

[kmike]

@pawelmhm I was more worried about Python wrappers speed