Skip to content

Add PyPy3 support #3050

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 25, 2017
Merged

Add PyPy3 support #3050

merged 3 commits into from
Dec 25, 2017

Conversation

lopuhin
Copy link
Member

@lopuhin lopuhin commented Dec 25, 2017

Just checking how they run on Travis. Turns out that lxml on PyPy3 is no longer a problem (PyPy 5.9.0, Cython==0.27.3 and lxml==4.1.1).

Related to #2213

@codecov
Copy link

codecov bot commented Dec 25, 2017

Codecov Report

Merging #3050 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3050   +/-   ##
=======================================
  Coverage   84.51%   84.51%           
=======================================
  Files         164      164           
  Lines        9270     9270           
  Branches     1380     1380           
=======================================
  Hits         7835     7835           
  Misses       1177     1177           
  Partials      258      258

@lopuhin
Copy link
Member Author

lopuhin commented Dec 25, 2017

Totally unscientific speed comparison (on a slow Xeon, running scrapy bench -s CLOSESPIDER_TIMEOUT=30):

  rps
CPython 2.7 65
PyPy2-5.9.0 112
CPython 3.6 29
PyPy3-5.9.0 47

Would be good to deploy https://github.com/scrapy/scrapy-bench, benchmarks there are more representative I think.

These built-in functions are exposed as methods in PyPy3.
For scrapy this does not matter as:
1) they do not work for CPython at all
2) get_func_args is checked for presense of an argument in scrapy,
   extra "self" does not matter.
But it still makes sense to leave these tests so that we know we
shouldn't use get_func_args for built-in functions/methods.
@lopuhin
Copy link
Member Author

lopuhin commented Dec 25, 2017

This is almost ready, only waiting for the second pypy3 build to finish to check if it still takes 22 minutes 😱

Right now only Linux binaries are available for PyPy3, so I tested only Linux installation.

@lopuhin
Copy link
Member Author

lopuhin commented Dec 25, 2017

pypy3 took 23 minutes again, although the second longest py35 took 15 minutes. Locally, pypy3 took 16 minutes, and I don't have any ideas for quick speedups of the tests.

@lopuhin lopuhin changed the title [WIP] Run tests for PyPy3 Add PyPy3 support Dec 25, 2017
@lopuhin lopuhin requested a review from kmike December 25, 2017 13:46
Copy link
Member

@kmike kmike left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I don't see a big difference between waiting for 15 minutes and 23 minutes, it is a long time anyways. If we are to support pypy3, we should test it.

@kmike kmike merged commit 930f6ed into scrapy:master Dec 25, 2017
@kmike kmike added this to the v1.5 milestone Dec 25, 2017
@lopuhin lopuhin deleted the pypy3 branch December 25, 2017 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants