Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Place Content-encoding header into response's flags is desired #5290

Open
wants to merge 42 commits into
base: master
Choose a base branch
from
Open

Place Content-encoding header into response's flags is desired #5290

wants to merge 42 commits into from

Commits on Sep 14, 2019

  1. Add HEADERS_KEEP variable to settings and to HttpCompressionMiddlewar… 

    …e class (scrapy#1988)
    @cwen13
    cwen13 committed Sep 14, 2019
    Configuration menu
    Copy the full SHA
    5f068dc View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2019

  1. fixed type

    @cwen13
    cwen13 committed Sep 15, 2019
    Configuration menu
    Copy the full SHA
    6a725ca View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2019

  1. Added init to HttpCompMiddleware scrapy#1988

    @cwen13
    cwen13 committed Sep 16, 2019
    Configuration menu
    Copy the full SHA
    2b6fb1b View commit details
    Browse the repository at this point in the history

  2. Added init to HttpCompMiddleware scrapy#1988 Typeo

    @cwen13
    cwen13 committed Sep 16, 2019
    Configuration menu
    Copy the full SHA
    96a8dfc View commit details
    Browse the repository at this point in the history

  3. Trying with passing init crawler scrapy#1988

    @cwen13
    cwen13 committed Sep 16, 2019
    Configuration menu
    Copy the full SHA
    9010d9f View commit details
    Browse the repository at this point in the history

  4. Adding settings to HttpCompressionMiddleware instantiation in test

    @cwen13
    cwen13 committed Sep 16, 2019
    Configuration menu
    Copy the full SHA
    608b342 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2019

  1. Slowed down and tried to take things slower

    @cwen13
    cwen13 committed Sep 17, 2019
    Configuration menu
    Copy the full SHA
    3350d0f View commit details
    Browse the repository at this point in the history

  2. I mispelled init -_-

    @VorBoto
    VorBoto committed Sep 17, 2019
    Configuration menu
    Copy the full SHA
    cb09ef5 View commit details
    Browse the repository at this point in the history

  3. changed the how its intialized in test

    @VorBoto
    VorBoto committed Sep 17, 2019
    Configuration menu
    Copy the full SHA
    f949126 View commit details
    Browse the repository at this point in the history

  4. Added a crawler to HttpCompressionTest using the Spider it created

    @VorBoto
    VorBoto committed Sep 17, 2019
    Configuration menu
    Copy the full SHA
    c94f35b View commit details
    Browse the repository at this point in the history

  5. Added from scrapy.utils.test import get_crawler

    @VorBoto
    VorBoto committed Sep 17, 2019
    Configuration menu
    Copy the full SHA
    490fb3c View commit details
    Browse the repository at this point in the history

  6. Passed tox but still not right

    @VorBoto
    VorBoto committed Sep 17, 2019
    Configuration menu
    Copy the full SHA
    e22c851 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2019

  1. Trying to add flag for encoding

    @VorBoto
    VorBoto committed Sep 18, 2019
    Configuration menu
    Copy the full SHA
    e479f88 View commit details
    Browse the repository at this point in the history

  2. Trying to add flag for encoding

    @VorBoto
    VorBoto committed Sep 18, 2019
    Configuration menu
    Copy the full SHA
    25d4760 View commit details
    Browse the repository at this point in the history

  3. Place encodings in response's flags issue scrapy#1988

    @VorBoto
    VorBoto committed Sep 18, 2019
    Configuration menu
    Copy the full SHA
    ccbb524 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2019

  1. Did some recomended changes

    @VorBoto
    VorBoto committed Sep 21, 2019
    Configuration menu
    Copy the full SHA
    8814265 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2019

  1. updated tests to function with no paramters sent to setUp()

    @VorBoto
    VorBoto committed Sep 25, 2019
    Configuration menu
    Copy the full SHA
    4588368 View commit details
    Browse the repository at this point in the history

  2. Switched dict['k'] to a dict.get('k') so no KeyError

    @VorBoto
    VorBoto committed Sep 25, 2019
    Configuration menu
    Copy the full SHA
    0a0b40d View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2019

  1. Actually pull the flags before trying to add decoded

    @VorBoto
    VorBoto committed Sep 26, 2019
    Configuration menu
    Copy the full SHA
    30ead53 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2019

  1. Removed unecessary checks for flag entry of kwargs

    @VorBoto
    VorBoto committed Sep 27, 2019
    Configuration menu
    Copy the full SHA
    fd11144 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2019

  1. Added/copied a few test as a way to show some continued functionality

    @VorBoto
    VorBoto committed Oct 5, 2019
    Configuration menu
    Copy the full SHA
    091a8a4 View commit details
    Browse the repository at this point in the history

  2. Fix an oversight

    @VorBoto
    VorBoto committed Oct 5, 2019
    Configuration menu
    Copy the full SHA
    d2fbd10 View commit details
    Browse the repository at this point in the history

  3. Fix an oversight on casting a str to binary

    @VorBoto
    VorBoto committed Oct 5, 2019
    Configuration menu
    Copy the full SHA
    a944117 View commit details
    Browse the repository at this point in the history

  4. Fix an oversight on casting a str to binary again

    @VorBoto
    VorBoto committed Oct 5, 2019
    Configuration menu
    Copy the full SHA
    8164e31 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2019

  1. fix some sytanx and formating as well as variable usage

    @VorBoto
    VorBoto committed Oct 17, 2019
    Configuration menu
    Copy the full SHA
    11af9bb View commit details
    Browse the repository at this point in the history

  2. Ther was a ( on the loose

    @VorBoto
    VorBoto committed Oct 17, 2019
    Configuration menu
    Copy the full SHA
    64597cd View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2019

  1. Syntax clean up

    @VorBoto
    VorBoto committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    0a03b45 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2021

  1. Merge branch 'master' into vorboto_local_master 

    # Conflicts:
#	scrapy/downloadermiddlewares/httpcompression.py
#	tests/test_downloadermiddleware_httpcompression.py
    @T0shik
    T0shik committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    ed06e4c View commit details
    Browse the repository at this point in the history

  2. fix post merge tests

    @T0shik
    T0shik committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    52b435a View commit details
    Browse the repository at this point in the history

  3. conftest 'tests/ignores.txt' to rely on native paths

    @T0shik
    T0shik committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    624a1ff View commit details
    Browse the repository at this point in the history

  4. revert conftest.py

    @T0shik
    T0shik committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    fa417f5 View commit details
    Browse the repository at this point in the history

  5. HttpCompressionMiddleware COMPRESSION_KEEP_ENCODING_HEADERS updates 

    - HttpCompressionMiddleware constructor deprecation warning
- refactor process_response
- put COMPRESSION_KEEP_ENCODING_HEADER in settings.py.tmpl
- test HttpCompressionMiddleware init
- test COMPRESSION_KEEP_ENCODING_HEADERS presence
- ensure b'decoded' flag is set if decoded
    @T0shik
    T0shik committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    0d095c4 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2021

  1. improve wording & deprecation warning test

    @T0shik
    T0shik committed Oct 25, 2021
    Configuration menu
    Copy the full SHA
    6556495 View commit details
    Browse the repository at this point in the history

  2. add keep_encoding_header setting to test_engine.py

    @T0shik
    T0shik committed Oct 25, 2021
    Configuration menu
    Copy the full SHA
    38e23ff View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2021

  1. remove un-used import

    @T0shik
    T0shik committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    3320993 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2021

  1. actually revert conftest.py

    @T0shik
    T0shik committed Oct 30, 2021
    Configuration menu
    Copy the full SHA
    b811b81 View commit details
    Browse the repository at this point in the history

  2. set COMPRESSION_KEEP_ENCODING_HEADER to True as default and revert ch… 

    …anges in test_engine.py
    @T0shik
    T0shik committed Oct 30, 2021
    Configuration menu
    Copy the full SHA
    35b48ab View commit details
    Browse the repository at this point in the history

  3. do NOT mutate original response content_encoding list. 

    - amend tests & logic to follow the new default of COMPRESSION_KEEP_ENCODING_HEADER=True
- remove redundant condition
    @T0shik
    T0shik committed Oct 30, 2021
    Configuration menu
    Copy the full SHA
    87998af View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2021

  1. Merge remote-tracking branch 'upstream/master'

    @T0shik
    T0shik committed Nov 6, 2021
    Configuration menu
    Copy the full SHA
    533f69a View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2021

  1. Merge branch 'scrapy:master' into master

    @T0shik
    T0shik committed Nov 8, 2021
    Configuration menu
    Copy the full SHA
    3aea4a8 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2021

  1. Merge branch 'scrapy:master' into master

    @T0shik
    T0shik committed Nov 18, 2021
    Configuration menu
    Copy the full SHA
    cc491e7 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2022

  1. Merge branch 'scrapy:master' into master

    @T0shik
    T0shik committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    ed157d5 View commit details
    Browse the repository at this point in the history