Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PyPy3 support #3050

Merged
merged 3 commits into from Dec 25, 2017
Merged

Add PyPy3 support #3050

merged 3 commits into from Dec 25, 2017

Conversation

@lopuhin
Copy link
Member

@lopuhin lopuhin commented Dec 25, 2017

Just checking how they run on Travis. Turns out that lxml on PyPy3 is no longer a problem (PyPy 5.9.0, Cython==0.27.3 and lxml==4.1.1).

Related to #2213

@lopuhin lopuhin force-pushed the lopuhin:pypy3 branch from cda69db to 58d686c Dec 25, 2017
@codecov
Copy link

@codecov codecov bot commented Dec 25, 2017

Codecov Report

Merging #3050 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3050   +/-   ##
=======================================
  Coverage   84.51%   84.51%           
=======================================
  Files         164      164           
  Lines        9270     9270           
  Branches     1380     1380           
=======================================
  Hits         7835     7835           
  Misses       1177     1177           
  Partials      258      258
@lopuhin
Copy link
Member Author

@lopuhin lopuhin commented Dec 25, 2017

Totally unscientific speed comparison (on a slow Xeon, running scrapy bench -s CLOSESPIDER_TIMEOUT=30):

  rps
CPython 2.7 65
PyPy2-5.9.0 112
CPython 3.6 29
PyPy3-5.9.0 47

Would be good to deploy https://github.com/scrapy/scrapy-bench, benchmarks there are more representative I think.

lopuhin added 2 commits Dec 25, 2017
These built-in functions are exposed as methods in PyPy3.
For scrapy this does not matter as:
1) they do not work for CPython at all
2) get_func_args is checked for presense of an argument in scrapy,
   extra "self" does not matter.
But it still makes sense to leave these tests so that we know we
shouldn't use get_func_args for built-in functions/methods.
@lopuhin lopuhin force-pushed the lopuhin:pypy3 branch from 4d1eec8 to 041308a Dec 25, 2017
@lopuhin
Copy link
Member Author

@lopuhin lopuhin commented Dec 25, 2017

This is almost ready, only waiting for the second pypy3 build to finish to check if it still takes 22 minutes 😱

Right now only Linux binaries are available for PyPy3, so I tested only Linux installation.

@lopuhin
Copy link
Member Author

@lopuhin lopuhin commented Dec 25, 2017

pypy3 took 23 minutes again, although the second longest py35 took 15 minutes. Locally, pypy3 took 16 minutes, and I don't have any ideas for quick speedups of the tests.

@lopuhin lopuhin changed the title [WIP] Run tests for PyPy3 Add PyPy3 support Dec 25, 2017
@lopuhin lopuhin requested a review from kmike Dec 25, 2017
@kmike
kmike approved these changes Dec 25, 2017
Copy link
Member

@kmike kmike left a comment

Looks good to me. I don't see a big difference between waiting for 15 minutes and 23 minutes, it is a long time anyways. If we are to support pypy3, we should test it.

@kmike kmike merged commit 930f6ed into scrapy:master Dec 25, 2017
2 checks passed
2 checks passed
codecov/patch Coverage not affected when comparing 632f1cc...bb1f311
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@kmike kmike added this to the v1.5 milestone Dec 25, 2017
@lopuhin lopuhin deleted the lopuhin:pypy3 branch Dec 25, 2017
@Prabhanjana

This comment has been minimized.

Copy link

@Prabhanjana Prabhanjana commented on bb1f311 Dec 7, 2018

Please let me know does scrapy supports browser-less crawling?

This comment has been minimized.

Copy link
Member Author

@lopuhin lopuhin replied Dec 7, 2018

@Prabhanjana actually browser-less crawling the the main thing scrapy does :) Maybe you meant something else?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants