Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.7 support #3326

Merged
merged 14 commits into from Jul 11, 2018
Merged

Python 3.7 support #3326

merged 14 commits into from Jul 11, 2018

Conversation

@lopuhin
Copy link
Member

@lopuhin lopuhin commented Jul 9, 2018

Continue @patiences work on python 3.7 compatibility from #3150, fixes #3143. Thanks a lot @patiences!

It's likely that we'll release scrapy 1.6 before new twisted version with twisted/twisted#966 is released (sorry for making the PR stall), so in this PR I work around this issue, so that scrapy at least works, although without telnet console support yet.

Also pytest is updated, so that tox -e py37 can collect the tests (2.9 was released more than 2 years ago).

WIP as I want to check how tests work on travis early (there are still a few failures due to PEP 479 left which didn't show up in #3150).

patiences and others added 11 commits Mar 2, 2018
2.9 gives collection errors on python 3.7 due to PEP 479.
Also give a more detailed reason why telnet is not enabled (for the
future).
PEP 479 does not allow for StopIteration in generators.  Instead,
handle it explicitly, also use a for loop which looks simpler.
@codecov
Copy link

@codecov codecov bot commented Jul 9, 2018

Codecov Report

Merging #3326 into master will increase coverage by 0.08%.
The diff coverage is 81.81%.

@@            Coverage Diff             @@
##           master    #3326      +/-   ##
==========================================
+ Coverage   84.24%   84.33%   +0.08%     
==========================================
  Files         167      167              
  Lines        9365     9369       +4     
  Branches     1387     1388       +1     
==========================================
+ Hits         7890     7901      +11     
+ Misses       1218     1214       -4     
+ Partials      257      254       -3
Impacted Files Coverage Δ
scrapy/extensions/telnet.py 87.5% <100%> (+12.5%) ⬆️
scrapy/utils/iterators.py 97.82% <77.77%> (-2.18%) ⬇️
scrapy/middleware.py 96.29% <0%> (+5.55%) ⬆️
@lopuhin lopuhin force-pushed the lopuhin:patiences-master branch from 70c5ee5 to b3cd12d Jul 9, 2018
Disable telnet console if it's not available, else we'll get an extra
warning about failure to enable it, and tests will fail.
@lopuhin lopuhin changed the title [WIP] Python 3.7 support Python 3.7 support Jul 9, 2018
@lopuhin
Copy link
Member Author

@lopuhin lopuhin commented Jul 9, 2018

Travis build is green now, this is ready for review.

@@ -40,7 +40,8 @@ def __init__(self, crawler):
if not crawler.settings.getbool('TELNETCONSOLE_ENABLED'):
raise NotConfigured
if not TWISTED_CONCH_AVAILABLE:
raise NotConfigured
raise NotConfigured('TelnetConsole not enabled: failed to import '
'required twisted modules.')

This comment has been minimized.

@kmike

kmike Jul 9, 2018
Member

I wonder if we should log the traceback here, and maybe explain that to remove an error TELNETCONSOLE_ENABLED can be set to False explicitly, if telnet is not needed.

This comment has been minimized.

@lopuhin

lopuhin Jul 9, 2018
Author Member

Thanks for suggestion @kmike , I think it gives more visibility to the issue but does not harm anyone. This is how it looks for me after 9428a4a:

2018-07-09 21:06:20 [scrapy.middleware] WARNING: Disabled TelnetConsole: TELNETCONSOLE_ENABLED setting is True but required twisted modules failed to import:
Traceback (most recent call last):
  File "/Users/kostia/shub/scrapy/scrapy/extensions/telnet.py", line 13, in <module>
    from twisted.conch import manhole, telnet
  File "/Users/kostia/shub/scrapy/py37/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
    def write(self, data, async=False):
                              ^
SyntaxError: invalid syntax
Capture traceback when trying to import required twisted modules,
print it in case telnet is enabled, and mention settings variable
that can be used to supress the message.
Thanks @kmike!
@lopuhin
Copy link
Member Author

@lopuhin lopuhin commented Jul 9, 2018

Btw, I did a quick check of speed via scrapy bench and Python 3.7 is a bit faster than Python 3.6

@kmike kmike added this to the v1.6 milestone Jul 11, 2018
@kmike kmike merged commit 8d5320d into scrapy:master Jul 11, 2018
1 of 2 checks passed
1 of 2 checks passed
continuous-integration/appveyor/pr AppVeyor build failed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@kmike
Copy link
Member

@kmike kmike commented Jul 11, 2018

Thanks @lopuhin and @patiences!

@lopuhin lopuhin deleted the lopuhin:patiences-master branch Aug 15, 2018
@rasendubi rasendubi mentioned this pull request Dec 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants