Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.7 support #3143

Closed
lopuhin opened this issue Feb 26, 2018 · 39 comments · Fixed by #3326
Closed

Python 3.7 support #3143

lopuhin opened this issue Feb 26, 2018 · 39 comments · Fixed by #3326

Comments

@lopuhin
Copy link
Member

lopuhin commented Feb 26, 2018

The goal is to add python 3.7 to travis and pass all tests, the first beta was already released at the end of January.

@lopuhin
Copy link
Member Author

lopuhin commented Feb 26, 2018

One issue was reported at https://stackoverflow.com/questions/48861287/why-am-i-getting-this-error-in-scrapy-python3-7-invalid-syntax

2018-02-26 19:11:02 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-02-26 19:11:02 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.3, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 3.7.0b1 (default, Feb 26 2018, 19:04:22) - [GCC 5.4.0 20160609], pyOpenSSL 17.5.0 (OpenSSL 1.0.2g  1 Mar 2016), cryptography 2.1.4, Platform Linux-4.4.0-116-generic-x86_64-with-debian-stretch-sid
2018-02-26 19:11:02 [scrapy.crawler] INFO: Overridden settings: {'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'EDITOR': 'vim', 'LOGSTATS_INTERVAL': 0, 'TELNETCONSOLE_ENABLED': '0'}
Traceback (most recent call last):
  File "/home/kostia/tmp/py37/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/cmdline.py", line 150, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
    func(*a, **kw)
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/cmdline.py", line 157, in _run_command
    cmd.run(args, opts)
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/commands/shell.py", line 65, in run
    crawler = self.crawler_process._create_crawler(spidercls)
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/crawler.py", line 203, in _create_crawler
    return Crawler(spidercls, self.settings)
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/crawler.py", line 55, in __init__
    self.extensions = ExtensionManager.from_crawler(self)
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
    mod = import_module(module)
  File "/home/kostia/.pyenv/versions/3.7.0b1/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 723, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/scrapy/extensions/telnet.py", line 12, in <module>
    from twisted.conch import manhole, telnet
  File "/home/kostia/tmp/py37/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
    def write(self, data, async=False):
                              ^
SyntaxError: invalid syntax

This also happens if -s TELNETCONSOLE_ENABLED=0 is passed, because the module is still imported.

Twisted issue: https://twistedmatrix.com/trac/ticket/9384#ticket and PR twisted/twisted#966

@patiences
Copy link
Contributor

Is this an issue that needs someone to work on? I would love to give it a shot. :-) Looks like the PR to fix the above issue should be merged soon.

@lopuhin
Copy link
Member Author

lopuhin commented Mar 2, 2018

@patiences yes, I'm not aware of anyone working on it, so it would be awesome if you give it a shot! And feel free to submit a work in progress PR. I think first step would be adding python 3.7 to tox and travis. There might be other issues besides this syntax error in twisted.

@patiences
Copy link
Contributor

@lopuhin Great!! The WIP PR is here: #3150 :-)

@qianyinghuanmie
Copy link

I once thought it supported Python3.7,in other words, was it support Python3.6-?

@grammy-jiang
Copy link
Contributor

@qianyinghuanmie

Scrapy supports Python 3.6, and Travis has related tests.

@illgitthat
Copy link
Contributor

It seems like this issue is dependent on twisted/twisted#966 to be merged in first.

@jstnms123
Copy link

I have reopened this issue. The problem persists.
<<<<<<<
jA$ scrapy shell
2018-07-08 14:42:44 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-07-08 14:42:44 [scrapy.utils.log] INFO: Versions: lxml 4.2.3.0, libxml2 2.9.2, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.4.0, Python 3.7.0 (default, Jun 29 2018, 20:13:53) - [Clang 8.0.0 (clang-800.0.42.1)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Darwin-15.6.0-x86_64-i386-64bit
2018-07-08 14:42:44 [scrapy.crawler] INFO: Overridden settings: {'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS_INTERVAL': 0}
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 11, in
sys.exit(execute())
File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 150, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
func(*a, **kw)
File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 157, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python3.7/site-packages/scrapy/commands/shell.py", line 65, in run
crawler = self.crawler_process._create_crawler(spidercls)
File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 203, in _create_crawler
return Crawler(spidercls, self.settings)
File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 55, in init
self.extensions = ExtensionManager.from_crawler(self)
File "/usr/local/lib/python3.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/usr/local/lib/python3.7/site-packages/scrapy/middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "/usr/local/lib/python3.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
mod = import_module(module)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/scrapy/extensions/telnet.py", line 12, in
from twisted.conch import manhole, telnet
File "/usr/local/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
def write(self, data, async=False):
^
SyntaxError: invalid syntax

@illgitthat
Copy link
Contributor

@jstnms123 I don't think this issue was ever closed, just the referenced issue since it was a duplicate.

@lopuhin do you know of a current workaround to install the branch you created in twisted so scrapy can run with 3.7? (twisted/twisted#966)

@lopuhin
Copy link
Member Author

lopuhin commented Jul 9, 2018

@illgitthat you can install the branch with pip install git+https://github.com/lopuhin/twisted.git@9384-remove-async-param. I hope we'll be able to also provide a work-around in 1.6, and then will hopefully finish the twisted fix (it affects modules that are very rarely used, but scrapy imports then unconditionally).

@illgitthat
Copy link
Contributor

Thank you @lopuhin! I googled around but I must have been just off on the syntax.

@jstnms123
Copy link

@lopuhin -- I used the work around to appropriate effect. Thanks for the info.

jA

@threezhang
Copy link

thanks @lopuhin

pip install git+https://github.com/lopuhin/twisted.git@9384-remove-async-param

@illgitthat
Copy link
Contributor

pip install git+https://github.com/lopuhin/twisted.git@9384-remove-async-param fails for me on Windows but worked on Linux. I think this is related to a twisted issue not scrapy, just leaving this here in case anyone else on Windows is stuck trying to run scrapy on 3.7.

@lopuhin
Copy link
Member Author

lopuhin commented Jul 20, 2018

FWIW, thanks to awesome Twisted maintainers, the fix in twisted/twisted#966 was finished and merged into Twisted, so it should be in the next Twisted release.

Also a workaround for this issue was merged into scrapy master (and will be in the 1.6 release), so another way to enable python 3.7 support right now, is to install scrapy master and use Twisted from PyPI.

@illgitthat
Copy link
Contributor

illgitthat commented Jul 21, 2018

Using pip install git+https://github.com/scrapy/scrapy@master --no-dependencies --upgrade is working for me.

Thanks for the help.

@johntiger1
Copy link

Yup @illgitthat works for me; I think @lopuhin branch might be merged in now, so it's gone

@ghost
Copy link

ghost commented Sep 4, 2018

Ummm just replace the word async with isAsync the other is deprecated....

@kmike
Copy link
Member

kmike commented Sep 4, 2018

@joshspivey async keyword is used in Twisted, not in Scrapy. @lopuhin worked with Twisted maintainers to fix it in Twisted, so Scrapy will work with Python 3.7 after Twisted release a new version with a fix. Also, we've worked around it in Scrapy itself, so that Scrapy works with the current Twisted release (disabling manhole), this will be available in a next Scrapy release. "Ummm just replace" is not how to fix this issue; such comments are not helpful.

@wertartem
Copy link

Go to Python37\Lib\site-packages\twisted\conch edit "manhole" file and replace the 'async' parameter by 'isAsync'.

@georgiana-gligor
Copy link

Because I was just trying this out in a new project, @illgitthat 's solution didn't work for me.
I had to follow all dependencies, so pip install git+https://github.com/scrapy/scrapy@master --no-dependencies --upgrade did the trick.

@cgironda
Copy link

Thank you to all of you guys, the command pip install git+https://github.com/scrapy/scrapy@master --no-dependencies --upgrade worked for me too.

@clockelliptic
Copy link

This also worked for me. pip install git+https://github.com/scrapy/scrapy@master --no-dependencies --upgrade

Thank you very much.

@ciehanski
Copy link

Also adding in that the current fix for this is running pip install git+https://github.com/scrapy/scrapy@master --no-dependencies --upgrade

Thank you everyone!

@AsthaSrivastava8
Copy link

AsthaSrivastava8 commented Oct 9, 2018

2018-10-10 01:56:15 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-10-10 01:56:15 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i  14 Aug 2018), cryptography 2.3.1, Platform Windows-10-10.0.17134-SP0
2018-10-10 01:56:15 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2018-10-10 01:56:15 [scrapy.middleware] WARNING: Disabled TelnetConsole: TELNETCONSOLE_ENABLED setting is True but required twisted modules failed to import:
Traceback (most recent call last):
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\extensions\telnet.py", line 13, in <module>
    from twisted.conch import manhole, telnet
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\twisted\conch\manhole.py", line 154
    def write(self, data, async=False):
                              ^
SyntaxError: invalid syntax

2018-10-10 01:56:16 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats']
Unhandled error in Deferred:
2018-10-10 01:56:16 [twisted] CRITICAL: Unhandled error in Deferred:

2018-10-10 01:56:16 [twisted] CRITICAL:
Traceback (most recent call last):
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\crawler.py", line 80, in crawl
    self.engine = self._create_engine()
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\crawler.py", line 105, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\core\engine.py", line 69, in __init__
    self.downloader = downloader_cls(crawler)
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\core\downloader\__init__.py", line 88, in __init__
    self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
    mod = import_module(module)
  File "c:\users\astha\appdata\local\programs\python\python37\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 20, in <module>
    from twisted.web.client import ResponseFailed
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\twisted\web\client.py", line 41, in <module>
    from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\twisted\internet\endpoints.py", line 41, in <module>
    from twisted.internet.stdio import StandardIO, PipeAddress
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
    from twisted.internet import _win32stdio
  File "c:\users\astha\appdata\local\programs\python\python37\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in <module>
    import win32api
ModuleNotFoundError: No module named 'win32api'

Getting error on crawl

@rogerlee6411
Copy link

same issue as AsthaSrivastava8 post. any solution now ?

@appcypher
Copy link

appcypher commented Oct 13, 2018

Installed scrapy via pipenv and it installed successfully but whenever I try the scrapy crawl command I get a similar error as the one above.

Python = 3.7.0
Pipenv = 2018.7.1
Scrapy = 1.5.1

Is there a temporary workaround?

@ghost
Copy link

ghost commented Oct 13, 2018

@appcypher As far as I understood the developers: Install from the repository directly, because it seems they implemented a workaround which was not released, yet.

See above: #3143 (comment)

@wmorgue
Copy link

wmorgue commented Oct 15, 2018

@appcypher via pipenv:

pipenv install scrapy==1.5.1
pipenv shell
pip install git+https://github.com/scrapy/scrapy@master --no-dependencies --upgrade

@xanderwang
Copy link

xanderwang commented Oct 24, 2018

pip install git+https://github.com/scrapy/scrapy@master --no-dependencies --upgrade

worked for me.

@kennblvnp
Copy link

just use python 2.7, problem solved

@ruchi3086
Copy link

Hi Lopuhin,
pip install git+https://github.com/lopuhin/twisted.git@9384-remove-async-param

it gives me path error, what i do?

@lopuhin
Copy link
Member Author

lopuhin commented Sep 25, 2019

hi @ruchi3086 since this was merged and released, now you can do pip install scrapy or pip3 install scrapy. If this still gives you an error, please paste it here.

@ruchi3086
Copy link

ruchi3086 commented Sep 25, 2019 via email

@ruchi3086
Copy link

Traceback (most recent call last):
File "C:/Users/rugupta/Documents/Project SL/import scrapy error.py", line 1, in
import scrapy
File "C:/Users/rugupta/Documents/Project SL\scrapy.py", line 5, in
from scrapy.crawler import CrawlerProcess
ModuleNotFoundError: No module named 'scrapy.crawler'; 'scrapy' is not a package

@lopuhin
Copy link
Member Author

lopuhin commented Sep 25, 2019

@ruchi3086 most likely you need to rename scrapy.py local file to some other name so that it does not conflict with scrapy package.

Note though that this is no longer related to python 3.7 support and most likely not a scrapy bug, please, use StackOverflow to ask this type of questions. See Getting Help.

@scrapy scrapy deleted a comment from ruchi3086 Sep 25, 2019
@scrapy scrapy deleted a comment from ruchi3086 Sep 25, 2019
@BHouwens
Copy link

I'm still having the original issue. None of the proposed workarounds seem to make any difference.

@Gallaecio
Copy link
Member

The original issue is Python 3.7 support. If you have a different issue, and you believe it is a Scrapy bug rather than an error in your end, please open a separate issue.

@kallesamuelsson
Copy link

The original issue is Python 3.7 support. If you have a different issue, and you believe it is a Scrapy bug rather than an error in your end, please open a separate issue.

I deleted my comment because I realized my misstake. Apologies for this

@scrapy scrapy deleted a comment from muhammedidris004 Jan 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.