[MRG+1] [httpcompression] add support for br - brotli content encoding #2535
Conversation
Does the library require anything special/additional to get installed properly? |
I didnt have to install anything, but it seems like some environments are failing to install, so most likely some systems are missing required libraries. Testing for support at runtime may be much safer yes, I'll add that @redapple |
Codecov Report@@ Coverage Diff @@
## master #2535 +/- ##
==========================================
- Coverage 83.49% 83.49% -0.01%
==========================================
Files 161 161
Lines 8787 8795 +8
Branches 1289 1290 +1
==========================================
+ Hits 7337 7343 +6
- Misses 1203 1205 +2
Partials 247 247
Continue to review full report at Codecov.
|
@pawelmhm , I do not see brotlipy being installed for Python 3.x tests on Travis. |
@@ -1,23 +1,33 @@ | |||
import zlib | |||
|
|||
|
kmike
Feb 6, 2017
Member
pep8 says 1 line between import groups :)
pep8 says 1 line between import groups :)
pawelmhm
Feb 7, 2017
Author
Contributor
good point, we could add flake8 style checks for PR. Flake8 could run on diff and point out all things like this.
good point, we could add flake8 style checks for PR. Flake8 could run on diff and point out all things like this.
@@ -6,6 +6,7 @@ pytest==2.9.2 | |||
pytest-twisted | |||
pytest-cov==2.2.1 | |||
jmespath | |||
brotlipy==0.6 |
kmike
Feb 6, 2017
Member
I'd prefer not to pin brotlipy version here: users will be installing it with 'pip install brotlipy', and we won't be able to detect if a brotlipy upgrade broke scrapy if we pin a version in tests.
I'd prefer not to pin brotlipy version here: users will be installing it with 'pip install brotlipy', and we won't be able to detect if a brotlipy upgrade broke scrapy if we pin a version in tests.
The PR looks good, besides a couple of minor points |
@kmike re performance there are good benchmarks and charts here: https://quixdb.github.io/squash-benchmark/#ratio-vs-compression |
@pawelmhm I was more worried about Python wrappers speed |
brotli encoding is used by multiple websites, e.g. amazon started to use it recently. Support in Scrapy is added via python library brotlipy https://github.com/python-hyper/brotlipy/