wish: to have pyuv ported to Pypy... #49

Closed
inercia opened this Issue Nov 30, 2012 · 50 comments

Comments

Projects
None yet
10 participants

inercia commented Nov 30, 2012

Hi there,

This is just a wish: it would be great to have pyuv working on Pypy. There is no problem compiling and installing it with Pypy, but some unit tests fail:

test_fsevent_dir (__main__.FSEventTest) ... ok
test_fsevent_nrefile (__main__.FSEventTest) ... ok
test_fsevent_basic (__main__.FSEventTestBasic) ... ok
test_fspoll1 (__main__.FSPollTest) ... RPython traceback:
  File "pypy_interpreter_gateway.c", line 842, in BuiltinCodePassThroughArguments1_funcrun_obj
  File "pypy_module_cpyext_api_4.c", line 40772, in generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI
  File "pypy_module_cpyext_pyobject.c", line 568, in make_ref
  File "pypy_module_cpyext_pyobject.c", line 1486, in create_ref
Fatal RPython error: AssertionError
Aborted (core dumped)

test_client1 (__main__.TCPErrorTest) ... ok
test_client2 (__main__.TCPErrorTest) ... ok
test_open (__main__.TCPErrorTest) ... ok
test_tcp_flags (__main__.TCPFlagsTest) ... ok
test_tcp_shutdown (__main__.TCPShutdownTest) ... ok
test_tcp1 (__main__.TCPTest) ... ok
test_tcp_write_cancel (__main__.TCPTest2) ... ok
test_tcp_list (__main__.TCPTestList) ... ok
test_tcp_list_null (__main__.TCPTestListNull) ... ok
test_tcp_list_unicode (__main__.TCPTestListUnicode) ... RPython traceback:
  File "pypy_interpreter_gateway.c", line 842, in BuiltinCodePassThroughArguments1_funcrun_obj
  File "pypy_module_cpyext_api_4.c", line 40772, in generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI
  File "pypy_module_cpyext_pyobject.c", line 568, in make_ref
  File "pypy_module_cpyext_pyobject.c", line 1486, in create_ref
Fatal RPython error: AssertionError
Aborted (core dumped)

test_udp_pingpong (__main__.UDPTest) ... RPython traceback:
  File "pypy_interpreter_gateway.c", line 842, in BuiltinCodePassThroughArguments1_funcrun_obj
  File "pypy_module_cpyext_api_4.c", line 40772, in generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI
  File "pypy_module_cpyext_pyobject.c", line 568, in make_ref
  File "pypy_module_cpyext_pyobject.c", line 1486, in create_ref
Fatal RPython error: AssertionError
Aborted (core dumped)

Thanks in advance

Alvaro

Owner

saghul commented Nov 30, 2012

Thanks for trying it out! What version of PyPy did you try? If it compiles ok (which didn't in the past, btw) I can try to track down the problems.

inercia commented Nov 30, 2012

Hi Saúl,

Thanks for your reply. I'm using Pypy 2.0beta1. The Pyuv installation was smooth, and many tests pass, but it seems there is a problem with callbacks... For example, when debugging Pypy with test_udp.py,

#0  0x00007ffff5c4d425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff5c50b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x0000000001a5ae33 in pypy_debug_catch_fatal_exception ()
#3  0x0000000001edfbaa in pypy_g_BuiltinCodePassThroughArguments1_funcrun_obj ()
#4  0x0000000002011414 in pypy_g_get_and_call_args ()
#5  0x000000000201157d in pypy_g_call_args ()
#6  0x0000000001ae7b46 in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
#7  0x000000000269ddbe in pypy_g_dispatch_bytecode__AccessDirect_None ()
#8  0x000000000269f4c3 in pypy_g_handle_bytecode__AccessDirect_None ()
#9  0x0000000002131942 in pypy_g_portal_3 ()
#10 0x0000000001d83f73 in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
#11 0x0000000001a9b807 in pypy_g_PyFrame_run ()
#12 0x0000000001bc29f6 in pypy_g_Method_call_args ()
#13 0x00000000020115db in pypy_g_call_args ()
#14 0x00000000021c0aed in PyObject_Call ()
#15 0x00000000028060e4 in PyObject_CallFunctionObjArgs ()
#16 0x00007ffff49e7bbc in on_udp_read () from /usr/local/pypy/site-packages/pyuv.pypy-20.so
#17 0x00007ffff4a09efb in uv__udp_recvmsg (loop=0x7ffff4c49320 <default_loop_struct>, w=0x45c3f80, revents=1) at src/unix/udp.c:238
#18 0x00007ffff4a09b8d in uv__udp_io (loop=0x7ffff4c49320 <default_loop_struct>, w=0x45c3f80, revents=1) at src/unix/udp.c:175
#19 0x00007ffff4a0e35f in uv__io_poll (loop=0x7ffff4c49320 <default_loop_struct>, timeout=-1) at src/unix/linux/linux-core.c:224
#20 0x00007ffff49f8dcd in uv__run (loop=0x7ffff4c49320 <default_loop_struct>) at src/unix/core.c:276
#21 0x00007ffff49f8e2c in uv_run (loop=0x7ffff4c49320 <default_loop_struct>) at src/unix/core.c:284
#22 0x00007ffff49de01e in Loop_func_run () from /usr/local/pypy/site-packages/pyuv.pypy-20.so

PyObject_CallFunctionObjArgs seems to be the root of the problem, as the same happens for other tests... Maybe it is a problem at Pypy, what do you think?

Thanks!

inercia commented Nov 30, 2012

And I forgot: I'm using pyuv HEAD from git...

Owner

saghul commented Nov 30, 2012

Doh, that function is called a lot inside the code :-S There may be some issues in PyPy due to the fact that we keep objects alive only in the C space. Unfortunately the backtrace doesn't show much. Testing this is definitely on my TODO list, so if you discover anything please do let me know, I shall do the same.

inercia commented Nov 30, 2012

It is weird: pycares works fine (as long as it uses select and not pyuv, of course), and it also relies on many calls to PyObject_CallFunctionObjArgs... what could be the difference? I will continue investigating the problem anyway...

Owner

saghul commented Nov 30, 2012

Indeed. It probably has to do with what arguments are passed to the function and how refcount affects them I guess...

Owner

saghul commented Feb 27, 2013

Hi @inercia,

I poked a bit on this but didn't get very far (I couldn't spend much time on it either unfortunately). IIRC you worked on some cffi bindings for libuv. If you are still working on those, would you be willing to merge the code with pyuv? If the implemented API is the same as pyuv's, which is basically an object oriented version of libuv, then we could detect the interpreter on setup.py and use the C extension or the CFFI version automatically. There would also be a switch for force either in case someone wanted to.

It doesn't have to be complete from day 1, it's acceptable that some functions just raise NotImplementedError when using CFFI, the test suite can be adapted to skip tests.

How do you feel about this?

inercia commented Mar 1, 2013

Hi @saghul,

I've been playing with my own cffi bindings for the last week, but I have found some instabilities with the latest libuv versions from github... I'll give it a try and see if I can use CFFI for this.

Owner

saghul commented Mar 1, 2013

@inercia fantastic! Let me know where I can look when you upload any code. Once the basics have been laid out I'll start a branch to make working with both backends possible and ease the integration of your work.

How'd you guys go with this?

Owner

saghul commented Oct 9, 2013

I haven't had the time to work on this yet. The idea is to create a package called pyuv and 2 submodules: _cpyuv and _pypyuv which would use CFFI (no need to build libuv for this one, assume it's installed) and in pyuv.init.py try to load _cpyuv and in case it fails load _pypyuv. On PyPy the extension module wouldn't be built and on CPython it would be used by default, unless some environment variable is set as an override I guess.

The structure would be similar to my python-fibers project.

I played a bit with cffi in pysophia, and also had a look at @inercia's work, unfortunately I don't know when I'll have time to start doing this. At least I got a plan! :-)

If you want to work on this please do! Send a pull request as soon as you have the loop and timers working so we can go through it before if gets too big :-)

tarruda commented Sep 14, 2014

For Neovim this would be great, it would enable usage of pypy as the python interpreter for python plugins through the python-client, which is backed by pyuv

tarruda referenced this issue in neovim/python-client Sep 15, 2014

Closed

[Python3 compat] Make it importable in python3 #25

smetj commented Sep 29, 2014

+1

veegee commented Nov 18, 2014

Is there any status update on this? If not, I can have a go at it with CFFI. I'm rewriting eventlet from the ground-up and basing it on pyuv, so the only way for it to work on pypy3 is through CFFI.

Owner

saghul commented Nov 18, 2014

No updates from me at least :-/ If you give it a go please do keep me
posted, I would love to integrate those changes into pyuv.

On Tue, Nov 18, 2014, 16:13 V G notifications@github.com wrote:

Is there any status update on this? If not, I can have a go at it with
CFFI. I'm rewriting eventlet from the ground-up and basing it on pyuv, so
the only way for it to work on pypy3 is through CFFI.


Reply to this email directly or view it on GitHub
#49 (comment).

tarruda commented Nov 18, 2014

Is there any status update on this? If not, I can have a go at it with CFFI. I'm rewriting eventlet from the ground-up and basing it on pyuv, so the only way for it to work on pypy3 is through CFFI.

I would love to see pyuv reimplemented on top of CFFI, but since you are writing an event loop project from the start, consider basing it on PEP 3156 which is the new python standard API for event loops.

The advantage of that API is that is can have many backends, including libuv which you can use on C python

Owner

saghul commented Nov 18, 2014

FWIW, have a look at my "rose" project, its the PEP 3156 API on top of pyuv
:-)
On Nov 18, 2014 4:33 PM, "Thiago de Arruda" notifications@github.com
wrote:

Is there any status update on this? If not, I can have a go at it with
CFFI. I'm rewriting eventlet from the ground-up and basing it on pyuv, so
the only way for it to work on pypy3 is through CFFI.

I would love to see pyuv reimplemented on top of CFFI, but since you are
writing an event loop project from the start, consider basing it on PEP
3156 https://www.python.org/dev/peps/pep-3156 which is the new python
standard API for event loops.

The advantage of that API is that is can have many backends, including
libuv https://github.com/saghul/rose which you can use on C python


Reply to this email directly or view it on GitHub
#49 (comment).

tarruda commented Nov 18, 2014

FWIW, have a look at my "rose" project, its the PEP 3156 API on top of pyuv
:-)

Thats what I linked to :)

veegee commented Nov 18, 2014

I like the asyncio module, but doesn't it depend on "yield from"? I picked eventlet because it uses greenlets and monkey-patching (which are compatible with more python versions including pypy3, as well as being easier to use without modifying your existing application code). I'm going to stick with greenlets, but I am interesting in asyncio if there's a way to use it as the event loop base without depending on "yield from" or generators as coroutines.

And I will definitely keep you guys posted when I begin implementing the pyuv interface via CFFI. I will most likely start in a few days or a week or so.

tarruda commented Nov 18, 2014

@veegee you can use asyncio for python 3.4+ and trollius for older versions.

As for yield from, you don't have to use it. I have been using only the event loop API with greenlets in another project(I find the greenlet API much simpler to deal with)

veegee commented Nov 22, 2014

Ok guys, I have officially begun work on a pyuv CFFI implementation. Please see my guv project for a pre-alpha proof-of-concept. It runs on python3 and pypy3 successfully. Only signals implemented so far, but CFFI makes this incredibly easy to work with. I'm guessing a complete implementation shouldn't take more than a few days.

Excuse the code quality - I've never used CFFI before and I did this over lunch. Will reorganize and restructure soon.

Edit: preliminary support for timers added. Next up: poll

Owner

saghul commented Nov 22, 2014

@veegee awesome! I had a quick look, some error handling is missing, but we can take care of that later. :-) I'll try to find the time to re-structure the code in order to merge the code in the future.

Thanks for working on this!

PS: One of the hard problems will probably be refcounting. Semantics will be different, due to the differences between CPython and PyPy, so some tests will need to be adjusted.

veegee commented Nov 22, 2014

@saghul pypy3 is garbage collected, not reference counted, as far as I know. And since I'm not using the python extension module API, we don't need to worry about reference counting anything. The only thing to note is that I have to manually allocate and free the libuv structs using custom C functions because ffi.new() can't allocate them directly since it doesn't know their size. The __del__() function seems to be very useful for freeing those structs, and as far as I can tell, nothing is leaking.

Owner

saghul commented Nov 22, 2014

Sorry, I should have been a bit more specific. When a timer is started, for
example, the Python object needs to be kept alive even if the user deletes
all references to it, until the callback fires. Not sure if ffi.callback
helps here.

Also, if a handle is garbage collected we need to uv_close it.

On Sun, Nov 23, 2014, 00:00 V G notifications@github.com wrote:

@saghul https://github.com/saghul pypy3 is garbage collected, not
reference counted, as far as I know. And since I'm not using the python
extension module API, we don't need to worry about reference counting
anything. The only thing to note is that I have to manually allocate and
free the libuv structs because ffi.new() can't allocate them directly since
it doesn't know their size. The del seems to be very useful for that,
and as far as I can tell, nothing is leaking.


Reply to this email directly or view it on GitHub
#49 (comment).

veegee commented Nov 22, 2014

@saghul Ah, I see. I'll add the call to uv_close().

The way ffi.callback() (as well as ffi.new()) works is they return a cdata object that has "ownership" on its memory. When the cdata object goes out of scope, the memory is automatically freed. That's why I have a self._ffi_cb object - its purpose is only to keep the cdata object alive so the callbacks can fire. It can be adjusted to make sure it's not deleted until it is safe to do so.

veegee commented Nov 23, 2014

Update: Got Timers, Poll, Signal mostly implemented correctly. No memory leaks on CPython as far as my memory profiler is telling me. Ignore what the README says as of 2014-11-23 4:00PM ET - I may have had another breakthrough just now: the eventlet "GreenPool" is very slow and the implementation is just terrible. Skipping their awful GreenPool implementation and spawning raw greenlets shows that the pyuv_cffi implementation is as fast as my raw epoll as well as your pyuv implementation. Very promising.

veegee commented Nov 23, 2014

Status update: all memory leaks fixed, everything seems to be going smooth. It's very fast on pypy3 (but then again, so is the plain epoll backend). More libuv handles implemented.

Owner

saghul commented Dec 9, 2014

@veegee I plan to look at starting to merge some code soon, but I noticed guv is LGPL, any chance you can re-license the pyuv_cffi part as MIT?

veegee commented Dec 9, 2014

Done. The whole thing (including pyuv_cffi) is now under a MIT license since it's probably better for network code anyway.

Note that pyuv_cffi is still incomplete; some methods are not implemented, and some handle types are missing. I've been focusing on guv, which has gained an impressive number of stars in the past days, so I didn't have time to finish pyuv_cffi (it only contains what is essential for guv). Finishing it is definitely in the issue tracker though

Owner

saghul commented Dec 10, 2014

That's great! At least it should give me a good start, I don't plan to have it complete by day one :-)

veegee commented Dec 10, 2014

For what it's worth, guv seems to be working exactly the same way for both of these lines of code:

import pyuv_cffi
and
import pyuv as pyuv_cffi

Owner

saghul commented Dec 10, 2014

@veegee awesome. My plan is to start small, ideally enough so that you can remove your pyuv_cffi package and use pyuv 1.1.0 (when ready) directly. Then we can continue to add the missing stuff :-)

veegee commented Dec 14, 2014

I just added support for the property Loop.handles, which returns a list of all specific handle objects tied to the loop.

digi604 commented May 20, 2015

+1

veegee commented May 20, 2015

This is actually very close to being done, but I'm having trouble finding the time to do anything lately. That'll change in ~1 month, but in the meantime, it would be great if I could get some help. The existing pyuv_cffi code is here: https://github.com/veegee/guv/tree/develop/pyuv_cffi and it contains everything needed for guv to work at the moment.

koehlma commented Dec 5, 2015

I have started working on a complete libuv CFFI based wrapper because I really need libuv to work together with pypy. It is not fully pyuv compatible, although it should be possible to write a thin pyuv compatibility layer: https://github.com/koehlma/uv

veegee commented Dec 5, 2015

@koehlma, you should be able to copy/paste my incomplete implementation. I didn't bother to finish implementing all features because they were not needed for my event loop, but the project layout certainly allows them to be very easily added in about an afternoon's worth of work.

koehlma commented Dec 5, 2015

@veegee, I have already implemented all stuff you had. But I also want to support all the DNS, FS and miscellaneous parts of libuv which requires a little bit more work.

Owner

saghul commented Dec 6, 2015

I'm sad you chose the LGPL license, that means I cannot look at that code
or integrate it.
On Dec 5, 2015 21:07, "Maximilian Köhl" notifications@github.com wrote:

I have started working on a complete libuv CFFI based wrapper because I
really need libuv to work together with pypy. It is not fully pyuv
compatible, although it should be possible to write a thin pyuv
compatibility layer: https://github.com/koehlma/uv


Reply to this email directly or view it on GitHub
#49 (comment).

koehlma commented Dec 6, 2015

Yes, but MIT is no option for me, sorry. Nevertheless it could still be integrated. In that case people who want to use PyPy together with libuv have to accept the conditions of LGPL or otherwise simply use CPython. Which is IMHO better than having no choice at all. Would that be acceptable? However I do not want to start a discussion about MIT vs. LGPL.

Owner

saghul commented Dec 7, 2015

I do not intend to start such a discussion either, sorry if my comment led you to believe otherwise!

My end goal here is to eventually have both CFFI and C backends (I already refactored the package structure to make room for it, but never found the time and patience to start). Since pyuv already has documentation and tests IMHO it's a good place to make it happen, since those don't have to be rewritten. A new decorator for skipping unimplemented parts of the CFFI backend could be used.

If anyone wants to commit to this efforts, I'm willing to at least helt review the code and make such person a comitter here so that they can maintain that part.

IANAL, but adding LGPL code to the project is not an option since the license that governs the project is MIT and I don't plan on dual-licensing it.

At any rate, thanks a lot for chiming in, and (as one of the libuv maintainers) I wish you good luck in your project!

Contributor

tilgovi commented Mar 24, 2017

I've got a branch that passes the tests on the brand new PyPy3 5.7 release. I'm not 100% sure how these changes feel yet, but should I open a PR for feedback?

Contributor

tilgovi commented Mar 25, 2017

Here is the comparison view:
v1.x...tilgovi:pypy

Owner

saghul commented Mar 25, 2017

@tilgovi Changes look good! Mind creating a PR? You probably also want to modify the Travis file so tests run on PyPy too :-)

Contributor

tilgovi commented Mar 25, 2017

Travis will have to wait until Travis updates their PyPy version (issue: travis-ci/travis-ci#7507).

I can add it and mark it as allowed to fail, though.

In the meantime, I would love to understand why the tp_clear member is NULL during dealloc for Loops and Handles in the basetype tests. If it were just Handles, I would suspect something with resurrection, but it happens on Loops, too. If there's a bug in cpyext, I would like to report it.

Contributor

tilgovi commented Mar 25, 2017

I can also have Travis for PyPy3 download a portable pypy3 build, but that'll add a lot of noise to the Travis file.

Owner

saghul commented Mar 25, 2017

Travis will have to wait until Travis updates their PyPy version (issue: travis-ci/travis-ci#7507).

I can add it and mark it as allowed to fail, though.

I can also have Travis for PyPy3 download a portable pypy3 build, but that'll add a lot of noise to the Travis file.

That's fine, I don't want to merge a patch which could break over time if we can't test it. IIRC there is ppa with latest PyPy, you could also use that.

In the meantime, I would love to understand why the tp_clear member is NULL during dealloc for Loops and Handles in the basetype tests. If it were just Handles, I would suspect something with resurrection, but it happens on Loops, too. If there's a bug in cpyext, I would like to report it.

TBH I have no idea what that could be :-S

Contributor

tilgovi commented Mar 28, 2017

I have opened #232 and after more investigation into the issues I feel pretty secure about it.

I think I have figured out why PyPy and CPython differ in the behavior of tp_clear with inheritance, but all the standard library modules in CPython invoke the tp_clear function by name, rather than by dereference from the type object. So, I feel good about the change here.

Let me know on the PR if there's anything else I can do here or if any of it makes anyone uneasy.

Owner

saghul commented Apr 21, 2017

#232 landed, so I'm closing this. It's not CFFI, but at this point I don't have the time (or inclination) to rewrite 9k lines of C code.

Huge props to @tilgovi for the patch!

saghul closed this Apr 21, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment