Python 3.7 support #3326
Merged
Python 3.7 support #3326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2.9 gives collection errors on python 3.7 due to PEP 479.
Also give a more detailed reason why telnet is not enabled (for the future).
PEP 479 does not allow for StopIteration in generators. Instead, handle it explicitly, also use a for loop which looks simpler.
Codecov Report
@@ Coverage Diff @@
## master #3326 +/- ##
==========================================
+ Coverage 84.24% 84.33% +0.08%
==========================================
Files 167 167
Lines 9365 9369 +4
Branches 1387 1388 +1
==========================================
+ Hits 7890 7901 +11
+ Misses 1218 1214 -4
+ Partials 257 254 -3
|
Disable telnet console if it's not available, else we'll get an extra warning about failure to enable it, and tests will fail.
|
Travis build is green now, this is ready for review. |
kmike
reviewed
Jul 9, 2018
scrapy/extensions/telnet.py
Outdated
| @@ -40,7 +40,8 @@ def __init__(self, crawler): | |||
| if not crawler.settings.getbool('TELNETCONSOLE_ENABLED'): | |||
| raise NotConfigured | |||
| if not TWISTED_CONCH_AVAILABLE: | |||
| raise NotConfigured | |||
| raise NotConfigured('TelnetConsole not enabled: failed to import ' | |||
| 'required twisted modules.') | |||
There was a problem hiding this comment.
I wonder if we should log the traceback here, and maybe explain that to remove an error TELNETCONSOLE_ENABLED can be set to False explicitly, if telnet is not needed.
There was a problem hiding this comment.
Thanks for suggestion @kmike , I think it gives more visibility to the issue but does not harm anyone. This is how it looks for me after 9428a4a:
2018-07-09 21:06:20 [scrapy.middleware] WARNING: Disabled TelnetConsole: TELNETCONSOLE_ENABLED setting is True but required twisted modules failed to import:
Traceback (most recent call last):
File "/Users/kostia/shub/scrapy/scrapy/extensions/telnet.py", line 13, in <module>
from twisted.conch import manhole, telnet
File "/Users/kostia/shub/scrapy/py37/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
def write(self, data, async=False):
^
SyntaxError: invalid syntax
Capture traceback when trying to import required twisted modules, print it in case telnet is enabled, and mention settings variable that can be used to supress the message. Thanks @kmike!
|
Btw, I did a quick check of speed via |
|
Thanks @lopuhin and @patiences! |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Continue @patiences work on python 3.7 compatibility from #3150, fixes #3143. Thanks a lot @patiences!
It's likely that we'll release scrapy 1.6 before new twisted version with twisted/twisted#966 is released (sorry for making the PR stall), so in this PR I work around this issue, so that scrapy at least works, although without telnet console support yet.
Also pytest is updated, so that
tox -e py37can collect the tests (2.9 was released more than 2 years ago).WIP as I want to check how tests work on travis early (there are still a few failures due to PEP 479 left which didn't show up in #3150).