Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare for 0.7.0 release #61

Merged
merged 4 commits into from Dec 9, 2022
Merged

Prepare for 0.7.0 release #61

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
23ddbc0
Bump version: 0.6.0 → 0.7.0
BurnzZ Dec 9, 2022
523631f
prepare CHANGES.rst for 0.7.0 release
BurnzZ Dec 9, 2022
152a8f0
fix failing tests on new tox >= 4.0 update
BurnzZ Dec 9, 2022
b5cfb5f
polish 0.7.0 updates and add release date
BurnzZ Dec 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bumpversion.cfg
View file
Open in desktop
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.6.0
current_version = 0.7.0
commit = True
tag = True
tag_name = {new_version}
Expand Down
40 changes: 40 additions & 0 deletions CHANGES.rst
View file
Open in desktop
@@ -1,6 +1,46 @@
Changes
=======

0.7.0 (2022-12-09)
------------------

When upgrading, you should set the following in your Scrapy settings:

.. code-block:: python

DOWNLOADER_MIDDLEWARES = {
"scrapy_zyte_api.ScrapyZyteAPIDownloaderMiddleware": 1000,
}
# only applicable for Scrapy 2.7+
REQUEST_FINGERPRINTER_CLASS = "scrapy_zyte_api.ScrapyZyteAPIRequestFingerprinter"

* Fixes the issue where scrapy-zyte-api is slow when Scrapy Cloud has Autothrottle
Addon enabled. The new ``ScrapyZyteAPIDownloaderMiddleware`` fixes this.

* It now supports Scrapy 2.7's new ``REQUEST_FINGERPRINTER_CLASS`` which ensures
that Zyte API requests are properly fingerprinted. This addresses the issue
where Scrapy marks POST requests as duplicate if they point to the same URL
despite having different request bodies. As a workaround, users were marking
their requests with ``dont_filter=True`` to prevent such dupe filtering.

For users having ``scrapy >= 2.7``, you can simply update your Scrapy settings
to have ``REQUEST_FINGERPRINTER_CLASS = "scrapy_zyte_api.ScrapyZyteAPIRequestFingerprinter"``.

If your Scrapy project performs other requests aside from Zyte API, you can set
``ZYTE_API_FALLBACK_REQUEST_FINGERPRINTER_CLASS = "custom.RequestFingerprinter"``
to allow custom fingerprinting. By default, the default Scrapy request
fingerprinter is used for non-Zyte API requests.

For users having ``scrapy < 2.7``, check the following link to see different
ways on handling the duplicate request issue:
https://github.com/scrapy-plugins/scrapy-zyte-api#request-fingerprinting-before-scrapy-27.

More information about the request fingerprinting topic can be found in
https://github.com/scrapy-plugins/scrapy-zyte-api#request-fingerprinting.

* Various improvements to docs and tests.


0.6.0 (2022-10-20)
------------------

Expand Down
2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Expand Up @@ -2,7 +2,7 @@

setuptools.setup(
name="scrapy-zyte-api",
version="0.6.0",
version="0.7.0",
description="Client library to process URLs through Zyte API",
long_description=open("README.rst").read(),
long_description_content_type="text/x-rst",
Expand Down
12 changes: 6 additions & 6 deletions tox.ini
View file
Open in desktop
Expand Up @@ -28,43 +28,43 @@ deps =


# Earliest supported Scrapy version.
[testenv:pinned-scrapy-2.0]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new update on tox >= 4.0 had a lot of backward incompatible changes.

It's now strict on the testenv:<env_name> format which renders this old line to have this error:

pinned-26-scrapy: failed with env name pinned-scrapy-2.0 conflicting with base python python3.7
pinned-scrapy-2.0: FAIL code 1 (0.01 seconds)
evaluation failed :( (1.24 seconds)

Setting ignore_basepython_conflict=True is supposed to fix this but it doesn't work. Seems like they're still releasing a lot of regression fixes after the 4.0 release.

They're also quite aggressive on the name inference. If they see any two digits in the env name, they'd treat it as a Python version. So other workarounds like 20, 2-0, etc doesn't work.

[testenv:pinned-scrapy-2x0]
basepython=python3.7
deps =
{[pinned]deps}
scrapy==2.0.1

# Scrapy version introducing Response.ip_address.
[testenv:pinned-scrapy-2.1]
[testenv:pinned-scrapy-2x1]
basepython=python3.7
deps =
{[pinned]deps}
scrapy==2.1.0

# Latest Scrapy version since 2.0.1 not requiring to install the reactor early.
[testenv:pinned-scrapy-2.3]
[testenv:pinned-scrapy-2x3]
basepython=python3.7
deps =
{[pinned]deps}
scrapy==2.3.0

# First Scrapy version requiring to install the reactor early.
[testenv:pinned-scrapy-2.4]
[testenv:pinned-scrapy-2x4]
basepython=python3.7
deps =
{[pinned]deps}
scrapy==2.4.0

# Scrapy version introducing Response.protocol.
[testenv:pinned-scrapy-2.5]
[testenv:pinned-scrapy-2x5]
basepython=python3.7
deps =
{[pinned]deps}
scrapy==2.5.0

# First Scrapy version since 2.4.0 where installing the reactor earlier is not
# necessary.
[testenv:pinned-scrapy-2.6]
[testenv:pinned-scrapy-2x6]
basepython=python3.7
deps =
{[pinned]deps}
Expand Down