New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) #4504
Merged
Merged
Changes from 26 commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
f361899
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)
Gallaecio de1ffb8
Merge branch 'master' into pypy
Gallaecio 6ef0ed9
Merge branch 'master' into pypy
Gallaecio 63091fb
Merge branch 'master' into pypy
Gallaecio a3eb069
Log a detailed error message to discover why MockServer is not working
Gallaecio 477b9e2
Go for all lines!
Gallaecio 3bf84af
Disable tests based on mitmproxy while running on PyPy
Gallaecio 35d23a9
Fix test_get_func_args for PyPy 3.6+
Gallaecio 1bf4e26
Make testPayloadDefaultCiphers work regardless of OpenSSL default cip…
Gallaecio 63939cc
Merge remote-tracking branch 'upstream/master' into pypy
Gallaecio a613667
Crossing fingers…
Gallaecio f95f924
Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher
Gallaecio 99f0271
Test the PyPy version currently documented as the minimum required ve…
Gallaecio 47f6e0e
Fix the PYPY_VERSION tag
Gallaecio 1dff643
Update the documentation about supported PyPy versions
Gallaecio e33d459
Also test the latest 3.5 Python version with PyPy
Gallaecio b671086
Fix the PYPY_VERSION value for the latest 3.5 version
Gallaecio 6fe6aec
Use pinned dependencies for asyncio and PyPy tests against oldest sup…
Gallaecio 6b648e0
Fix PyPy installation for the pypy3-pinned Tox environment
Gallaecio edf08a2
Try installing Cython
Gallaecio 75dc93d
Maybe PyPy requires lxml 3.6.0?
Gallaecio 7d3dc04
install.rst: minor clarification
Gallaecio 9279af9
lxml 4.0.0 is required on PyPy
Gallaecio 017ec33
Require setuptools 18.5+
Gallaecio 2a47ac7
Revert "Require setuptools 18.5+"
Gallaecio a221144
Maintain lxml as a dependency if setuptools < 18.5 is used
Gallaecio 04981de
Merge branch 'master' into pypy
Gallaecio 38ae506
Merge branch 'master' into pypy
Gallaecio File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,12 +18,37 @@ def has_environment_marker_platform_impl_support(): | |
return parse_version(setuptools_version) >= parse_version('18.5') | ||
|
||
|
||
install_requires = [ | ||
'Twisted>=17.9.0', | ||
'cryptography>=2.0', | ||
'cssselect>=0.9.1', | ||
'parsel>=1.5.0', | ||
'PyDispatcher>=2.0.5', | ||
'pyOpenSSL>=16.2.0', | ||
'queuelib>=1.4.2', | ||
'service_identity>=16.0.0', | ||
'w3lib>=1.17.0', | ||
'zope.interface>=4.1.3', | ||
'protego>=0.1.15', | ||
'itemadapter>=0.1.0', | ||
] | ||
extras_require = {} | ||
|
||
if has_environment_marker_platform_impl_support(): | ||
extras_require[':platform_python_implementation == "CPython"'] = [ | ||
'lxml>=3.5.0', | ||
] | ||
extras_require[':platform_python_implementation == "PyPy"'] = [ | ||
# Earlier lxml versions are affected by | ||
# https://bitbucket.org/pypy/pypy/issues/2498/cython-on-pypy-3-dict-object-has-no, | ||
# which was fixed in Cython 0.26, released on 2017-06-19, and used to | ||
# generate the C headers of lxml release tarballs published since then, the | ||
# first of which was: | ||
'lxml>=4.0.0', | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is technically possible for someone to install an older version if they re-generate the C headers with Cython 0.26+. If they are willing to do that, however, they can also change this number here locally for a Scrapy version supporting that. |
||
'PyPyDispatcher>=2.1.0', | ||
] | ||
else: | ||
install_requires.append('lxml>=3.5.0') | ||
|
||
|
||
setup( | ||
|
@@ -67,20 +92,6 @@ def has_environment_marker_platform_impl_support(): | |
'Topic :: Software Development :: Libraries :: Python Modules', | ||
], | ||
python_requires='>=3.5.2', | ||
install_requires=[ | ||
'Twisted>=17.9.0', | ||
'cryptography>=2.0', | ||
'cssselect>=0.9.1', | ||
'lxml>=3.5.0', | ||
'parsel>=1.5.0', | ||
'PyDispatcher>=2.0.5', | ||
'pyOpenSSL>=16.2.0', | ||
'queuelib>=1.4.2', | ||
'service_identity>=16.0.0', | ||
'w3lib>=1.17.0', | ||
'zope.interface>=4.1.3', | ||
'protego>=0.1.15', | ||
'itemadapter>=0.1.0', | ||
], | ||
install_requires=install_requires, | ||
extras_require=extras_require, | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think "Installing Scrapy" section should be updated, with the new minimal pypy version?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense.
Once we fix the remaining issue, though, we should probably try and find the lowest version where tests pass.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we should care too much about supporting old pypy versions; I think it is for advanced users to run Scrapy on pypy, and they could install a recent version anyways, to get best performance & compatibility. That should be fine to pick some version which is fresh enough, and doesn't cause us problems. Of course, knowing a specific pypy version is better, but I'd not put too much effort in figuring it out - if that's easy, let's do it, otherwise - don't bother.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PyPy 5.9 continues to work.
I’ve also moved the FAQ entry about supported Python versions to
install.rst
, to avoid duplication. The previous information was accurate, though, so if you have any second thoughts about the documentation changes, we can revert that commit and leave the documentation as is.