Python 3 support #263

Closed
extesy opened this Issue Mar 6, 2013 · 65 comments

Comments

Projects
None yet
@extesy

extesy commented Mar 6, 2013

Python 3 is several years old and most of packages now support it (even django!). It would be really nice to support it in scrapy as well.

@artemdevel

This comment has been minimized.

Show comment
Hide comment
@artemdevel

artemdevel Mar 7, 2013

Contributor

Scrapy uses Twisted in its core, so support python 3 at least depended on Twisted python 3 support. Twisted development team has a project to port Twisted on python 3 and it is in progress, so I think as soon as Twisted is ported to python 3 Scrapy will get good chances to be ported as well.

Contributor

artemdevel commented Mar 7, 2013

Scrapy uses Twisted in its core, so support python 3 at least depended on Twisted python 3 support. Twisted development team has a project to port Twisted on python 3 and it is in progress, so I think as soon as Twisted is ported to python 3 Scrapy will get good chances to be ported as well.

@todoit

This comment has been minimized.

Show comment
Hide comment

todoit commented Apr 25, 2013

mark

@nramirezuy

This comment has been minimized.

Show comment
Hide comment
@estin

This comment has been minimized.

Show comment
Hide comment
@estin

estin Apr 29, 2013

for python3 I am developing
https://bitbucket.org/estin/pomp
like scrapy but very small, unstable and without hard twisted dependency

estin commented Apr 29, 2013

for python3 I am developing
https://bitbucket.org/estin/pomp
like scrapy but very small, unstable and without hard twisted dependency

@coodoing

This comment has been minimized.

Show comment
Hide comment
@coodoing

coodoing May 7, 2013

mark'
the latest development branch 0.17 did not support py3

coodoing commented May 7, 2013

mark'
the latest development branch 0.17 did not support py3

@ariddell

This comment has been minimized.

Show comment
Hide comment
@ariddell

ariddell May 16, 2013

@nramirezuy there's a reference implementation for pep 3156 here: https://code.google.com/p/tulip/

@nramirezuy there's a reference implementation for pep 3156 here: https://code.google.com/p/tulip/

@muelli

This comment has been minimized.

Show comment
Hide comment
@muelli

muelli Jun 19, 2014

Is there a list of what parts of Twisted are used?
Twisted have a python3 migration plan here: http://twistedmatrix.com/trac/wiki/Plan/Python3
It might be worthwhile to investigate whether the used parts of Twisted are already ported.

muelli commented Jun 19, 2014

Is there a list of what parts of Twisted are used?
Twisted have a python3 migration plan here: http://twistedmatrix.com/trac/wiki/Plan/Python3
It might be worthwhile to investigate whether the used parts of Twisted are already ported.

@txtsd

This comment has been minimized.

Show comment
Hide comment
@txtsd

txtsd Oct 23, 2014

Can scrapy not be made to work with python 3, now that asyncio is available?

txtsd commented Oct 23, 2014

Can scrapy not be made to work with python 3, now that asyncio is available?

@curita curita removed the gsoc-candidate label Feb 11, 2015

@curita curita added this to the Scrapy 1.0 milestone Feb 11, 2015

@pablohoffman pablohoffman modified the milestones: Scrapy 1.1, Scrapy 1.0 Mar 5, 2015

@ianozsvald

This comment has been minimized.

Show comment
Hide comment
@ianozsvald

ianozsvald Mar 18, 2015

+1 for Python 3.4 support. After a year using Python 3 (mainly sklearn, numpy, Anaconda, matplotlib, networkx etc) this is the first blocker I've had forcing me to downgrade.

The only other Python2.7-only project that I'm lightly using is Apache Spark and 3.4+ support is scheduled for their next release. In their issue tracker I posted some stats for Python 3 adoption - roughly speaking it is ">40%" (accepting the self-selected group of survey participants):
https://issues.apache.org/jira/browse/SPARK-4897?focusedCommentId=14303154&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14303154

+1 for Python 3.4 support. After a year using Python 3 (mainly sklearn, numpy, Anaconda, matplotlib, networkx etc) this is the first blocker I've had forcing me to downgrade.

The only other Python2.7-only project that I'm lightly using is Apache Spark and 3.4+ support is scheduled for their next release. In their issue tracker I posted some stats for Python 3 adoption - roughly speaking it is ">40%" (accepting the self-selected group of survey participants):
https://issues.apache.org/jira/browse/SPARK-4897?focusedCommentId=14303154&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14303154

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Mar 18, 2015

Member

@ianozsvald we are working on it, it is a priority :)

Scrapy is the worst kind of project to port to Python 3 - it depends on Twisted (which is not ported to Python 3 yet - some subset of Twisted works though), and it works at outside world / python world boundary, so there are many questions about unicode. "Outer World" Scrapy works with is wild - there is no a well-defined encoding we can decode/encode data from/to. Encoding rules are sometimes crazy - e.g. browsers (which Scrapy aims to emulate) can use different charsets for different parts of a single URL, e.g. cp1251 for /path and utf-8 for GET parameter values. I've ported a lot of code to Python 3 (including most of NLTK and tens of other Python packages), but still getting porting details wrong for Scrapy (e.g. #837 is wrong).

Some parts of Scrapy are already ported to Python 3. We're running tests for Python 3.3 on Travis to prevent regressions; ~240 tests pass in 3.3, out of ~1000. There is a GSoC project to port Scrapy to Python 3.x; I think we should make a good progress this summer.

Member

kmike commented Mar 18, 2015

@ianozsvald we are working on it, it is a priority :)

Scrapy is the worst kind of project to port to Python 3 - it depends on Twisted (which is not ported to Python 3 yet - some subset of Twisted works though), and it works at outside world / python world boundary, so there are many questions about unicode. "Outer World" Scrapy works with is wild - there is no a well-defined encoding we can decode/encode data from/to. Encoding rules are sometimes crazy - e.g. browsers (which Scrapy aims to emulate) can use different charsets for different parts of a single URL, e.g. cp1251 for /path and utf-8 for GET parameter values. I've ported a lot of code to Python 3 (including most of NLTK and tens of other Python packages), but still getting porting details wrong for Scrapy (e.g. #837 is wrong).

Some parts of Scrapy are already ported to Python 3. We're running tests for Python 3.3 on Travis to prevent regressions; ~240 tests pass in 3.3, out of ~1000. There is a GSoC project to port Scrapy to Python 3.x; I think we should make a good progress this summer.

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Mar 18, 2015

Member

There is also https://github.com/mitmproxy/mitmproxy Scrapy dependency which doesn't have Python 3 support yet, but it is used only in tests.

Member

kmike commented Mar 18, 2015

There is also https://github.com/mitmproxy/mitmproxy Scrapy dependency which doesn't have Python 3 support yet, but it is used only in tests.

@ianozsvald

This comment has been minimized.

Show comment
Hide comment
@ianozsvald

ianozsvald Mar 19, 2015

@kmike Hey Mikhail! You are a man of many projects :-) Glad to hear it is being worked on, I didn't get that impression from the early parts of this thread and couldn't see any other porting docs. I quite agree that this project (just like Flask et al.) is going to be hard, dealing with the interface to the outside world is horrid. I certainly didn't know that URLs themselves could have mixed encodings :-(
Given the continual migration to Python 3 for personal projects (50/50 according to the survey I linked vs Python 2.7) and >40% for work, the need for scrapy's Py3 support is only going to get stronger. Bon chance!

@kmike Hey Mikhail! You are a man of many projects :-) Glad to hear it is being worked on, I didn't get that impression from the early parts of this thread and couldn't see any other porting docs. I quite agree that this project (just like Flask et al.) is going to be hard, dealing with the interface to the outside world is horrid. I certainly didn't know that URLs themselves could have mixed encodings :-(
Given the continual migration to Python 3 for personal projects (50/50 according to the survey I linked vs Python 2.7) and >40% for work, the need for scrapy's Py3 support is only going to get stronger. Bon chance!

@pbronez

This comment has been minimized.

Show comment
Hide comment
@pbronez

pbronez Mar 23, 2015

Contributor

+1 for Python 3 support! Thanks for the hard work you guys are putting into it, hope GSoC goes well.

Contributor

pbronez commented Mar 23, 2015

+1 for Python 3 support! Thanks for the hard work you guys are putting into it, hope GSoC goes well.

@nuschk

This comment has been minimized.

Show comment
Hide comment
@nuschk

nuschk Apr 20, 2015

👍 as well, would really love to be able to use python 3 with scrapy! And many thanks your effort!

nuschk commented Apr 20, 2015

👍 as well, would really love to be able to use python 3 with scrapy! And many thanks your effort!

@vmarkovtsev

This comment has been minimized.

Show comment
Hide comment
@vmarkovtsev

vmarkovtsev May 19, 2015

You can use my patches with ported twisted.web.client.Agent and friends from my fork.

You can use my patches with ported twisted.web.client.Agent and friends from my fork.

@nyov

This comment has been minimized.

Show comment
Hide comment
@nyov

nyov Jul 14, 2015

Contributor

Are there still outside blockers for porting to python3? (twisted libs, etc.?)
Would love to see a list of those, if one has been made.

Also, in the name of eventual portability (e.g. asyncio?) how do people feel about dropping dependencies on twisted for the web/downloader part? I recall there was a gsoc idea for this?
Would be interesting to see if a downloader using pycurl bindings might work with twisted here.
(Though pycurl has no cffi bindings at this time, so no pypy support.)

Contributor

nyov commented Jul 14, 2015

Are there still outside blockers for porting to python3? (twisted libs, etc.?)
Would love to see a list of those, if one has been made.

Also, in the name of eventual portability (e.g. asyncio?) how do people feel about dropping dependencies on twisted for the web/downloader part? I recall there was a gsoc idea for this?
Would be interesting to see if a downloader using pycurl bindings might work with twisted here.
(Though pycurl has no cffi bindings at this time, so no pypy support.)

@curita

This comment has been minimized.

Show comment
Hide comment
@curita

curita Jul 14, 2015

Member

There's a comprehensive status of the twisted dependencies in Berker's proposal. @berkerpeksag, would you mind if we put it up on our wiki for reference?

Member

curita commented Jul 14, 2015

There's a comprehensive status of the twisted dependencies in Berker's proposal. @berkerpeksag, would you mind if we put it up on our wiki for reference?

@berkerpeksag

This comment has been minimized.

Show comment
Hide comment
@berkerpeksag

berkerpeksag Jul 14, 2015

Contributor

Sure, but that list is a bit outdated. For example, twisted.web.static has already been ported to Python 3. You may want to check twisted/python/dist3.py first.

Contributor

berkerpeksag commented Jul 14, 2015

Sure, but that list is a bit outdated. For example, twisted.web.static has already been ported to Python 3. You may want to check twisted/python/dist3.py first.

@curita

This comment has been minimized.

Show comment
Hide comment
@curita

curita Jul 14, 2015

Member

Will do, thanks!

Member

curita commented Jul 14, 2015

Will do, thanks!

@curita

This comment has been minimized.

Show comment
Hide comment
@nyov

This comment has been minimized.

Show comment
Hide comment
@nyov

nyov Jul 14, 2015

Contributor

Thank you both!

Contributor

nyov commented Jul 14, 2015

Thank you both!

@tonal

This comment has been minimized.

Show comment
Hide comment
@curita

This comment has been minimized.

Show comment
Hide comment
@curita

curita Jul 15, 2015

Member

That's great news, thanks for reporting! I just updated the wiki.

Member

curita commented Jul 15, 2015

That's great news, thanks for reporting! I just updated the wiki.

@ianozsvald

This comment has been minimized.

Show comment
Hide comment
@ianozsvald

ianozsvald Jul 23, 2015

Hello all. I've got a Lightning Talk on Python3.5 at my next PyDataLondon meet (200+ data scientists in the room). Someone is bound to ask about scrapy/twisted on Python 3.4+, could someone comment on the current state? It isn't clear to me from the links above if enough of twisted has been ported for scrapy to run on Python 3 (or will soon)?

Hello all. I've got a Lightning Talk on Python3.5 at my next PyDataLondon meet (200+ data scientists in the room). Someone is bound to ask about scrapy/twisted on Python 3.4+, could someone comment on the current state? It isn't clear to me from the links above if enough of twisted has been ported for scrapy to run on Python 3 (or will soon)?

@curita

This comment has been minimized.

Show comment
Hide comment
@curita

curita Jul 25, 2015

Member

Hi @ianozsvald, glad to hear about the interest in python3 support!

Currently scrapy doesn't run in python3, not even a meaningful subset of it, but twisted support isn't the only issue holding us back. Most of the twisted modules used in scrapy are already ported, and in some cases the features that use them could be deactivated, like telnet or mail (well, extensions that use mail could be deactivated or changed to not use mail in python3 for instance). twisted.web.client.Agent is a problem anyhow, but this can be patched in our side.

We stopped the python3 integration for some time because we couldn't agree on the type we should use to represent urls but thankfully that matter was resolved, though it hasn't been coded yet.

So, there aren't any big stoppers (not that I know of), just the time to get around it.

We haven't defined a deadline yet but it's something we want to see before the end of the year. On top of that, this weekend some scrapinghubbers will hold a sprint to accelerate the support, so maybe there'll be news sooner than expected 😉

Member

curita commented Jul 25, 2015

Hi @ianozsvald, glad to hear about the interest in python3 support!

Currently scrapy doesn't run in python3, not even a meaningful subset of it, but twisted support isn't the only issue holding us back. Most of the twisted modules used in scrapy are already ported, and in some cases the features that use them could be deactivated, like telnet or mail (well, extensions that use mail could be deactivated or changed to not use mail in python3 for instance). twisted.web.client.Agent is a problem anyhow, but this can be patched in our side.

We stopped the python3 integration for some time because we couldn't agree on the type we should use to represent urls but thankfully that matter was resolved, though it hasn't been coded yet.

So, there aren't any big stoppers (not that I know of), just the time to get around it.

We haven't defined a deadline yet but it's something we want to see before the end of the year. On top of that, this weekend some scrapinghubbers will hold a sprint to accelerate the support, so maybe there'll be news sooner than expected 😉

@ianozsvald

This comment has been minimized.

Show comment
Hide comment
@ianozsvald

ianozsvald Jul 27, 2015

Hi @curita, thanks for the note. For my data science audience I think scrapy is the only non-python-3.4 package that matters, everything else that they (and I) use is already running with Python 3.4. I wish you all luck in the conversion, knowing the data science stack is almost fully 3.4 compliant really helps when planning larger-scale projects.

Hi @curita, thanks for the note. For my data science audience I think scrapy is the only non-python-3.4 package that matters, everything else that they (and I) use is already running with Python 3.4. I wish you all luck in the conversion, knowing the data science stack is almost fully 3.4 compliant really helps when planning larger-scale projects.

@rmax

This comment has been minimized.

Show comment
Hide comment
@rmax

rmax Jul 29, 2015

Contributor

Is there any plan to replace mitmproxy requirement for tests?

Contributor

rmax commented Jul 29, 2015

Is there any plan to replace mitmproxy requirement for tests?

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Jul 29, 2015

Member

@darkrho I don't know; I was thinking about porting it, not replacing. Are there alternatives?

Member

kmike commented Jul 29, 2015

@darkrho I don't know; I was thinking about porting it, not replacing. Are there alternatives?

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Jul 29, 2015

Member

@ianozsvald I know your pain; Scrapy is the only reason I'm using Python 2 now :) At EuroPython me and @dangra tried to unblock the further porting - the bottleneck was in Request and Response objects, and they are ported now in #1384. It is still a long road to full Python 3 support, but we're in a much better shape now - 480 507 tests are passing in Python 3, compared to 248 before the sprint. Working Request and Response objects open a gate for other's contributions, so I expect Python 3 Scrapy support to get more love soon.

Member

kmike commented Jul 29, 2015

@ianozsvald I know your pain; Scrapy is the only reason I'm using Python 2 now :) At EuroPython me and @dangra tried to unblock the further porting - the bottleneck was in Request and Response objects, and they are ported now in #1384. It is still a long road to full Python 3 support, but we're in a much better shape now - 480 507 tests are passing in Python 3, compared to 248 before the sprint. Working Request and Response objects open a gate for other's contributions, so I expect Python 3 Scrapy support to get more love soon.

@ianozsvald

This comment has been minimized.

Show comment
Hide comment
@ianozsvald

ianozsvald Jul 29, 2015

@kmike hey, that's lovely to hear (and Graham Markall [of Continuum] told me about the sprint), we'll certainly note this when we talk next week. Cheers!

@kmike hey, that's lovely to hear (and Graham Markall [of Continuum] told me about the sprint), we'll certainly note this when we talk next week. Cheers!

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Aug 5, 2015

Member

For anyone interested in contributing I've created a wiki page (https://github.com/scrapy/scrapy/wiki/Python-3-Porting) with some information & guidelines.

Member

kmike commented Aug 5, 2015

For anyone interested in contributing I've created a wiki page (https://github.com/scrapy/scrapy/wiki/Python-3-Porting) with some information & guidelines.

@kmike kmike removed this from the Scrapy 1.1 milestone Oct 30, 2015

@stonebig

This comment has been minimized.

Show comment
Hide comment

apparantly, twisted for python 3 is out ... https://twitter.com/hawkieowl/status/670885245328166912

@stonebig

This comment has been minimized.

Show comment
Hide comment
@stonebig

stonebig Nov 29, 2015

I don't understand, I suppose examples are not rewritten for Python3:

  • I can dowload and install Spider on Python 3.4
  • but this fails:
import scrapy 


class MySpider(scrapy.Spider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = [
        'http://www.example.com/1.html',
        'http://www.example.com/2.html',
        'http://www.example.com/3.html',
    ]

    def parse(self, response):
        self.logger.info('A response from %s just arrived!', response.url)

with this error

>>> ================================ RESTART ================================
>>> 
Traceback (most recent call last):
  File "D:/WinPython/basedir34/buildBarebone/winpython-3.4.3/notebooks/scrapy.py", line 1, in <module>
    import scrapy
  File "D:/WinPython/basedir34/buildBarebone/winpython-3.4.3/notebooks\scrapy.py", line 4, in <module>
    class MySpider(scrapy.Spider):
AttributeError: 'module' object has no attribute 'Spider'
>>> 

Any idea how it should be written in Python 3 ?

I don't understand, I suppose examples are not rewritten for Python3:

  • I can dowload and install Spider on Python 3.4
  • but this fails:
import scrapy 


class MySpider(scrapy.Spider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = [
        'http://www.example.com/1.html',
        'http://www.example.com/2.html',
        'http://www.example.com/3.html',
    ]

    def parse(self, response):
        self.logger.info('A response from %s just arrived!', response.url)

with this error

>>> ================================ RESTART ================================
>>> 
Traceback (most recent call last):
  File "D:/WinPython/basedir34/buildBarebone/winpython-3.4.3/notebooks/scrapy.py", line 1, in <module>
    import scrapy
  File "D:/WinPython/basedir34/buildBarebone/winpython-3.4.3/notebooks\scrapy.py", line 4, in <module>
    class MySpider(scrapy.Spider):
AttributeError: 'module' object has no attribute 'Spider'
>>> 

Any idea how it should be written in Python 3 ?

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Nov 29, 2015

Member

Hey @stonebig,

There are Scrapy parts which work in Python 3, but Scrapy as a framework is not usable for end users in Python 3 yet. Please wait or help us :)

Member

kmike commented Nov 29, 2015

Hey @stonebig,

There are Scrapy parts which work in Python 3, but Scrapy as a framework is not usable for end users in Python 3 yet. Please wait or help us :)

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Nov 29, 2015

Member

Apart from that, Spyder has nothing to do with Scrapy, and you're trying to import from your scrapy.py module, not from scrapy. There are other channels to get support - we're using http://stackoverflow.com (ask a question with Scrapy tag); there is also scrapy-users google group.

Member

kmike commented Nov 29, 2015

Apart from that, Spyder has nothing to do with Scrapy, and you're trying to import from your scrapy.py module, not from scrapy. There are other channels to get support - we're using http://stackoverflow.com (ask a question with Scrapy tag); there is also scrapy-users google group.

@stonebig

This comment has been minimized.

Show comment
Hide comment
@stonebig

stonebig Nov 29, 2015

ok. I'll go to the user group. Sorry for the noise.

ok. I'll go to the user group. Sorry for the noise.

@nyov

This comment has been minimized.

Show comment
Hide comment
@nyov

nyov Dec 1, 2015

Contributor

and updated in the wiki

Contributor

nyov commented Dec 1, 2015

and updated in the wiki

@redapple redapple added the Python 3 label Jan 27, 2016

@redapple redapple added this to the v1.1 milestone Jan 27, 2016

@redapple

This comment has been minimized.

Show comment
Hide comment
@redapple

redapple Jan 27, 2016

Contributor

Basic support is planned for v1.1
And we plan to make it more robust for v1.2

Contributor

redapple commented Jan 27, 2016

Basic support is planned for v1.1
And we plan to make it more robust for v1.2

@vmarkovtsev

This comment has been minimized.

Show comment
Hide comment

👍

@stonebig

This comment has been minimized.

Show comment
Hide comment
@stonebig

stonebig Jan 27, 2016

Great ! Is there a document that estimates the rough timeline of these two milestones ? spring 2016 and summer 2016 ?

Great ! Is there a document that estimates the rough timeline of these two milestones ? spring 2016 and summer 2016 ?

@redapple

This comment has been minimized.

Show comment
Hide comment
@redapple

redapple Jan 27, 2016

Contributor

@stonebig , we plan on releasing Scrapy 1.1 officially by the end of February 2016 (with a candidate release at least in the next few days)
Scrapy 1.2 would be a couple of months after that (we hope)

Contributor

redapple commented Jan 27, 2016

@stonebig , we plan on releasing Scrapy 1.1 officially by the end of February 2016 (with a candidate release at least in the next few days)
Scrapy 1.2 would be a couple of months after that (we hope)

@stonebig

This comment has been minimized.

Show comment
Hide comment
@stonebig

stonebig Jan 27, 2016

thanks a lot for this information, @redapple !

thanks a lot for this information, @redapple !

@KeremTubluk

This comment has been minimized.

Show comment
Hide comment
@KeremTubluk

KeremTubluk Feb 2, 2016

It has gone and past six days, @redapple!

:)

It has gone and past six days, @redapple!

:)

@redapple

This comment has been minimized.

Show comment
Hide comment
Contributor

redapple commented Feb 2, 2016

@manugarri

This comment has been minimized.

Show comment
Hide comment
@ABSmiLT

This comment has been minimized.

Show comment
Hide comment
@ABSmiLT

ABSmiLT Feb 29, 2016

it seems that the twisted already supports py3.3+

ABSmiLT commented Feb 29, 2016

it seems that the twisted already supports py3.3+

@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal Feb 29, 2016

it seems that the twisted already supports py3.3+

@ABSmiLT Yeah, AFAICT Twisted only recently supported 3 well enough for Scrapy. Hence all the discussion above and in the docs.

d0ugal commented Feb 29, 2016

it seems that the twisted already supports py3.3+

@ABSmiLT Yeah, AFAICT Twisted only recently supported 3 well enough for Scrapy. Hence all the discussion above and in the docs.

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Feb 29, 2016

Member

@ABSmiLT we've released scrapy 1.1rc1 with alpha-level Python 3 support about a month ago. 1.1rc2 will be released soon; it fixes several Python 3 compatibility issues we've found while testing 1.1rc1.

Member

kmike commented Feb 29, 2016

@ABSmiLT we've released scrapy 1.1rc1 with alpha-level Python 3 support about a month ago. 1.1rc2 will be released soon; it fixes several Python 3 compatibility issues we've found while testing 1.1rc1.

@ABSmiLT

This comment has been minimized.

Show comment
Hide comment
@ABSmiLT

ABSmiLT Mar 18, 2016

thanks for informing, @kmike @d0ugal
looking forward to the new stable version compatible with py3

ABSmiLT commented Mar 18, 2016

thanks for informing, @kmike @d0ugal
looking forward to the new stable version compatible with py3

@redapple

This comment has been minimized.

Show comment
Hide comment
@redapple

redapple May 11, 2016

Contributor

Scrapy 1.1.0 is on PyPI, with Python 3 support (finally).

What took you so long?

See release notes.

Have fun!

Contributor

redapple commented May 11, 2016

Scrapy 1.1.0 is on PyPI, with Python 3 support (finally).

What took you so long?

See release notes.

Have fun!

@redapple redapple closed this May 11, 2016

@nyov

This comment has been minimized.

Show comment
Hide comment
@nyov

nyov May 11, 2016

Contributor

Are we there yet? are we there yet are wethereyetarewethere — 💥 ...eh, WHAT?
😭 finally!

Congrats. Great job there keeping up the backports, @redapple. (Probably made a nice bang there, punching that "close"- button just then, right?)
And Thanks, everyone.

Contributor

nyov commented May 11, 2016

Are we there yet? are we there yet are wethereyetarewethere — 💥 ...eh, WHAT?
😭 finally!

Congrats. Great job there keeping up the backports, @redapple. (Probably made a nice bang there, punching that "close"- button just then, right?)
And Thanks, everyone.

@fish-ball

This comment has been minimized.

Show comment
Hide comment
@fish-ball

fish-ball May 12, 2016

Congratulations!!!!

Congratulations!!!!

@ABSmiLT

This comment has been minimized.

Show comment
Hide comment
@ABSmiLT

ABSmiLT May 16, 2016

@redapple FINALLY!
thanks @ all

btw, the "PY3 95%" label is now out of date?
image

ABSmiLT commented May 16, 2016

@redapple FINALLY!
thanks @ all

btw, the "PY3 95%" label is now out of date?
image

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike May 16, 2016

Member

@ABSmiLT no, it means 95% tests are passing in Python 3. There is a couple of features not ported yet (e.g. telnet console), and some tests are skipped. If we'd have PY2 badge it also won't be 100% according to this metric because some tests are also skipped in Python 2 :)

Member

kmike commented May 16, 2016

@ABSmiLT no, it means 95% tests are passing in Python 3. There is a couple of features not ported yet (e.g. telnet console), and some tests are skipped. If we'd have PY2 badge it also won't be 100% according to this metric because some tests are also skipped in Python 2 :)

@lothbrek

This comment has been minimized.

Show comment
Hide comment
@lothbrek

lothbrek May 17, 2016

So glad to see Python 3 support. Currently writing an automation/web crawling system in Ruby but may switch to Python with the help of Scrapy!

So glad to see Python 3 support. Currently writing an automation/web crawling system in Ruby but may switch to Python with the help of Scrapy!

@dangra

This comment has been minimized.

Show comment
Hide comment
@dangra

dangra May 18, 2016

Member

It has been a long road, great work!

Member

dangra commented May 18, 2016

It has been a long road, great work!

@ABSmiLT

This comment has been minimized.

Show comment
Hide comment
@ABSmiLT

ABSmiLT May 20, 2016

@kmike thanks a lot~ 💯

ABSmiLT commented May 20, 2016

@kmike thanks a lot~ 💯

@AverHLV

This comment has been minimized.

Show comment
Hide comment
@AverHLV

AverHLV May 21, 2016

But what about Twisted? I can`t run my Scrapy project on python 3.5 cause of Twisted errors.

AverHLV commented May 21, 2016

But what about Twisted? I can`t run my Scrapy project on python 3.5 cause of Twisted errors.

@rmax

This comment has been minimized.

Show comment
Hide comment
@rmax

rmax May 21, 2016

Contributor

@AverHLV Twisted can run on python 3 but not on windows because _win32stdio has not been ported yet. See:

Contributor

rmax commented May 21, 2016

@AverHLV Twisted can run on python 3 but not on windows because _win32stdio has not been ported yet. See:

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike May 21, 2016

Member

Yeah, there is an issue in Scrapy bug tracker about that: #1998

Member

kmike commented May 21, 2016

Yeah, there is an issue in Scrapy bug tracker about that: #1998

@ABSmiLT

This comment has been minimized.

Show comment
Hide comment
@ABSmiLT

ABSmiLT May 26, 2016

oops...am using python 3.5...so i have to continue waiting...loooooool

ABSmiLT commented May 26, 2016

oops...am using python 3.5...so i have to continue waiting...loooooool

@ABSmiLT

This comment has been minimized.

Show comment
Hide comment
@ABSmiLT

ABSmiLT May 26, 2016

i wish i can contribute to this project ASAP...still a newbie now...

ABSmiLT commented May 26, 2016

i wish i can contribute to this project ASAP...still a newbie now...

@dangra

This comment has been minimized.

Show comment
Hide comment
@dangra

dangra May 26, 2016

Member

hey @ABSmiLT, try the hack suggestion in #1998 (comment) and share your experience.

Member

dangra commented May 26, 2016

hey @ABSmiLT, try the hack suggestion in #1998 (comment) and share your experience.

@ABSmiLT

This comment has been minimized.

Show comment
Hide comment
@ABSmiLT

ABSmiLT May 28, 2016

@dangra okay, thanks a lot~

ABSmiLT commented May 28, 2016

@dangra okay, thanks a lot~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment