Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zmq support #116

Open
njsmith opened this issue Apr 11, 2017 · 9 comments
Open

zmq support #116

njsmith opened this issue Apr 11, 2017 · 9 comments

Comments

@njsmith
Copy link
Member

njsmith commented Apr 11, 2017

Everyone seemed really excited about this in curio, and it should be pretty simple, so hey, why not.

References:

This should go into a standalone package though – triozmq, not trio.zmq.

@kawing-chiu
Copy link

kawing-chiu commented Sep 14, 2018

Hi. What is the current status of this issue? It seems that ZMQSelector is not needed anymore in newer versions of pyzmq. And curio's zmq example is now quite simple. Can zmq be supported in trio through the same way?

@njsmith
Copy link
Member Author

njsmith commented Sep 14, 2018

Getting zmq to play nicely with regular event loops is pretty non-trivial. (That's why pyzmq used to try to dodge the issue by providing a custom loop.) We can do it though, it's just not trivial. See https://github.com/python-trio/trzmq and its readme for details of what needs to happen.

The change in pyzmq.asyncio is just that they finally did the non-trivial work described in that readme.

That curio example is just wrong AFAICT, because it completely misunderstands how the zmq waiting interface works. I guess Dave never tested it under load. If send ever blocks, it will deadlock, and recv will get stuck too under some circumstances.

@jiangrzh
Copy link

jiangrzh commented Nov 9, 2018

Hi, it's any plan to support nanomsg?

@njsmith
Copy link
Member Author

njsmith commented Nov 9, 2018

@jiangrzh I'm not very familiar with nanomsg, but will happily cheer you on if you want to get it working with Trio :-). Do you use the older nanomsg or the newer nng? Is there a Python wrapper already that you like?

@jiangrzh
Copy link

jiangrzh commented Nov 9, 2018

@njsmith Thank you for your response. I'm new to trio and nanomsg too. I think nng is better than nanomsg, but it seens python wrapper for nng is no ready for production?

@goodboy
Copy link
Member

goodboy commented Nov 9, 2018

@jiangrzh we've had brief discussion over at @codypiersall's new pynng.
Sounds like @gdamore is interested in easing the pains of this integration in nng itself as well 👍

@kawing-chiu
Copy link

kawing-chiu commented Nov 10, 2018

@njsmith I've read the notes on trzmq but can't say I understand it fully, since I have little knowledge on zeromq/trio internals.

As far as I understand it, the major problem is how to tame the "edge-triggered" behavior of the zmq FD. I've found a post on how to integrate zmq with the libev eventloop, not sure whether this approach is feasible for trio?

Another thing I found is that zeromq has a new poller api which is level-triggered. It is still a draft api but may become stable sometime later. And here is pyzmq's author asking about whether this new poller api can be used to integrate with other eventloops.

@njsmith
Copy link
Member Author

njsmith commented Nov 14, 2018

@kawing-chiu Thanks for the links! I think we do know how to integrate zmq with the trio event loop. It's just a bit complicated, so it will take a bit of work for someone to wrap their head around it, write the code, test and document it, etc. I haven't had time to do that, and even if I did then I probably wouldn't have time to focus on maintaining it well. Someone who actually uses zmq themselves would probably do a better job of that :-).

AFAICT from that poller API doc, currently it's just a more efficient version of zmq_poll, which doesn't help us – we need something fundamentally different than zmq_poll. Reading the discussion in the issue you linked (zeromq/libzmq#2941), it does sound like there's some tentative plan to change this in the future, but it doesn't sound like anyone is actually working on it currently, so... fingers crossed I guess!

@codypiersall
Copy link

codypiersall commented Jan 17, 2019

Just to follow up to @jiangrzh, we do have support for async operations on nng sockets in pynng. It couldn't have been done without help from @njsmith, @tgoodlet, and @gdamore! (Disclaimer: I'm the creator of pynng!)

Example usage:

import pynng
import trio

async def send_and_recv(sender, receiver, message):
    await sender.asend(message)
    return await receiver.arecv()

with pynng.Pair0(listen='tcp://127.0.0.1:54321') as s1, \
        pynng.Pair0(dial='tcp://127.0.0.1:54321') as s2:
    received = trio.run(send_and_recv, s1, s2, b'hello there old pal!')
    assert received == b'hello there old pal!'

The library is still a work in progress (what libraries aren't?). It can be installed via pip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants