Python 3 support #263
Comments
|
Scrapy uses Twisted in its core, so support python 3 at least depended on Twisted python 3 support. Twisted development team has a project to port Twisted on python 3 and it is in progress, so I think as soon as Twisted is ported to python 3 Scrapy will get good chances to be ported as well. |
|
mark |
|
we are waiting for http://www.python.org/dev/peps/pep-3156/ |
|
for python3 I am developing |
|
mark' |
|
@nramirezuy there's a reference implementation for pep 3156 here: https://code.google.com/p/tulip/ |
|
Is there a list of what parts of Twisted are used? |
|
Can scrapy not be made to work with python 3, now that asyncio is available? |
|
+1 for Python 3.4 support. After a year using Python 3 (mainly sklearn, numpy, Anaconda, matplotlib, networkx etc) this is the first blocker I've had forcing me to downgrade. The only other Python2.7-only project that I'm lightly using is Apache Spark and 3.4+ support is scheduled for their next release. In their issue tracker I posted some stats for Python 3 adoption - roughly speaking it is ">40%" (accepting the self-selected group of survey participants): |
|
@ianozsvald we are working on it, it is a priority :) Scrapy is the worst kind of project to port to Python 3 - it depends on Twisted (which is not ported to Python 3 yet - some subset of Twisted works though), and it works at outside world / python world boundary, so there are many questions about unicode. "Outer World" Scrapy works with is wild - there is no a well-defined encoding we can decode/encode data from/to. Encoding rules are sometimes crazy - e.g. browsers (which Scrapy aims to emulate) can use different charsets for different parts of a single URL, e.g. cp1251 for /path and utf-8 for GET parameter values. I've ported a lot of code to Python 3 (including most of NLTK and tens of other Python packages), but still getting porting details wrong for Scrapy (e.g. #837 is wrong). Some parts of Scrapy are already ported to Python 3. We're running tests for Python 3.3 on Travis to prevent regressions; ~240 tests pass in 3.3, out of ~1000. There is a GSoC project to port Scrapy to Python 3.x; I think we should make a good progress this summer. |
|
There is also https://github.com/mitmproxy/mitmproxy Scrapy dependency which doesn't have Python 3 support yet, but it is used only in tests. |
|
@kmike Hey Mikhail! You are a man of many projects :-) Glad to hear it is being worked on, I didn't get that impression from the early parts of this thread and couldn't see any other porting docs. I quite agree that this project (just like Flask et al.) is going to be hard, dealing with the interface to the outside world is horrid. I certainly didn't know that URLs themselves could have mixed encodings :-( |
|
+1 for Python 3 support! Thanks for the hard work you guys are putting into it, hope GSoC goes well. |
|
👍 as well, would really love to be able to use python 3 with scrapy! And many thanks your effort! |
|
You can use my patches with ported twisted.web.client.Agent and friends from my fork. |
|
Are there still outside blockers for porting to python3? (twisted libs, etc.?) Also, in the name of eventual portability (e.g. asyncio?) how do people feel about dropping dependencies on twisted for the web/downloader part? I recall there was a gsoc idea for this? |
|
There's a comprehensive status of the twisted dependencies in Berker's proposal. @berkerpeksag, would you mind if we put it up on our wiki for reference? |
|
Sure, but that list is a bit outdated. For example, |
|
Will do, thanks! |
|
Here's the updated list: https://github.com/scrapy/scrapy/wiki/PY3%3A-Twisted-Dependencies. |
|
Thank you both! |
|
That's great news, thanks for reporting! I just updated the wiki. |
|
Hello all. I've got a Lightning Talk on Python3.5 at my next PyDataLondon meet (200+ data scientists in the room). Someone is bound to ask about scrapy/twisted on Python 3.4+, could someone comment on the current state? It isn't clear to me from the links above if enough of |
|
Basic support is planned for v1.1 |
|
👍 |
|
Great ! Is there a document that estimates the rough timeline of these two milestones ? spring 2016 and summer 2016 ? |
|
@stonebig , we plan on releasing Scrapy 1.1 officially by the end of February 2016 (with a candidate release at least in the next few days) |
|
thanks a lot for this information, @redapple ! |
|
It has gone and past six days, @redapple! :) |
|
@KeremTubluk , we're not quite there yet: https://github.com/scrapy/scrapy/milestones/v1.1 |
|
aand its official now. http://doc.scrapy.org/en/stable/news.html#id1 |
|
it seems that the twisted already supports py3.3+ |
@ABSmiLT Yeah, AFAICT Twisted only recently supported 3 well enough for Scrapy. Hence all the discussion above and in the docs. |
|
@ABSmiLT we've released scrapy 1.1rc1 with alpha-level Python 3 support about a month ago. 1.1rc2 will be released soon; it fixes several Python 3 compatibility issues we've found while testing 1.1rc1. |
|
Scrapy 1.1.0 is on PyPI, with Python 3 support (finally).
See release notes. Have fun! |
|
Are we there yet? are we there yet are wethereyetarewethere — 💥 ...eh, WHAT? Congrats. Great job there keeping up the backports, @redapple. (Probably made a nice bang there, punching that "close"- button just then, right?) |
|
Congratulations!!!! |
|
@redapple FINALLY! btw, the "PY3 95%" label is now out of date? |
|
@ABSmiLT no, it means 95% tests are passing in Python 3. There is a couple of features not ported yet (e.g. telnet console), and some tests are skipped. If we'd have PY2 badge it also won't be 100% according to this metric because some tests are also skipped in Python 2 :) |
|
So glad to see Python 3 support. Currently writing an automation/web crawling system in Ruby but may switch to Python with the help of Scrapy! |
|
It has been a long road, great work! |
|
@kmike thanks a lot~ 💯 |
|
But what about Twisted? I can`t run my Scrapy project on python 3.5 cause of Twisted errors. |
|
@AverHLV Twisted can run on python 3 but not on windows because |
|
Yeah, there is an issue in Scrapy bug tracker about that: #1998 |
|
oops...am using python 3.5...so i have to continue waiting...loooooool |
|
i wish i can contribute to this project ASAP...still a newbie now... |
|
hey @ABSmiLT, try the |
|
@dangra okay, thanks a lot~ |
Update run-modes.rst
and others
Python 3 is several years old and most of packages now support it (even django!). It would be really nice to support it in scrapy as well.
The text was updated successfully, but these errors were encountered: