zero-copy support in cffi backend

[mattip]

When trying out PyPy2 with spyder via the spyder-kernels package (which is extends the ipython protocol?), I discovered that CFFI-based Frame was not working correctly: it did not support the buffer protocol. PyPy extends the ability of pure python classes to support the buffer protocol. If you inherit from the __pypy__.bufferable.bufferable class and then instaniate an instance of your class b, callng memoryview(b) will call self.__buffer__(flags) and give you a chance to return a buffer.

I implemented this, but it seems there is some code in the cython backend to release the buffer via a gc callback? In any case, I unskipped some of the tests.

Does this look OK? The only tests that are failing are the ones that check tracker.done, which makes me think maybe this is leaking memory.

[minrk]

This is great! Supporting the buffer interface isn't enough for the cffi backend to support zero-copy sends, though, which is what the done tests cover. This is still an improvement, so if you put back the skips on the two tracker tests, I think this is great.

In the pypy2 tests, it looks like numpy.frombuffer doesn't work with buffer-interface-providers:

B = numpy.frombuffer(msg, A.dtype).reshape(A.shape)
TypeError: expected string or Unicode object, Frame found

maybe need a version check or direct check for the presence of this bufferable type?

[mattip]

In the pypy2 tests, it looks like numpy.frombuffer doesn't work with buffer-interface-providers

There was a bug in PyPy, the fix should appear in the pypy2 nightly build tomorrow.

[mattip]

Supporting the buffer interface isn't enough for the cffi backend to support zero-copy sends

Is there something I can do to get this going?

[minrk]

Is there something I can do to get this going?

You could give it a go. To implement zero-copy, Frame needs to match the Cython version more, which you can use as reference.

Mainly:

• send/Frame on a buffer with copy=False should not copy memory with zmq_msg_init_size, instead it should create a zmq_msg_t with zmq_msg_init_data referencing that memory and an associated GIL-less zmq_free_fn.
• The zmq_free_fn sends a message to the garbage collector (which is in Python and thus shared between backends).

[mattip]

in a322aa0 I

• created a separate build step for the cffi c-extension module
• changed the cffi interfaces to use that module
• copied the copy=False behaviour from message.pyx to message.py

Now test_multi_tracker in test_message.py hangs. How do I instrument the callbacks so I can see what is not being called? I don't see the fprintf output from the call to free_python_msg

[mattip]

any supported python should be able to use setuptools, and I think is to be preferred over distutils where possible

[minrk]

If we're switching to requiring setuptools, let's make sure to avoid the main reason we haven't required setuptools: that it breaks python setup.py install by implicitly doing egg installs and corrupting sys.path:

from setuptools.command.bdist_egg import bdist_egg

class bdist_egg_disabled(bdist_egg):
    """Disabled version of bdist_egg

    Prevents setup.py install from performing setuptools' default easy_install,
    which it should never ever do.
    """

    def run(self):
        sys.exit(
            "Aborting implicit building of eggs. Use `pip install .` to install from source."
        )

...
setup_args['cmdclass'] = {
    ...
    'bdist_egg': bdist_egg if 'bdist_egg' in sys.argv else bdist_egg_disabled,
}

[minrk]

Can this be in a .c file instead of inlined?

[minrk]

If we're switching to requiring setuptools, let's make sure to avoid the main reason we haven't required setuptools: that it breaks python setup.py install by implicitly doing egg installs and corrupting sys.path:

from setuptools.command.bdist_egg import bdist_egg

class bdist_egg_disabled(bdist_egg):
    """Disabled version of bdist_egg

    Prevents setup.py install from performing setuptools' default easy_install,
    which it should never ever do.
    """

    def run(self):
        sys.exit(
            "Aborting implicit building of eggs. Use `pip install .` to install from source."
        )

...
setup_args['cmdclass'] = {
    ...
    'bdist_egg': bdist_egg if 'bdist_egg' in sys.argv else bdist_egg_disabled,
}

[minrk]

zero-copy now works in PyPy. Thanks @mattip!

[mattip]

Thanks for finishing this up. Are there any benchmarks for pyzmq?

[mattip]

I found https://github.com/achimnol/asyncio-zmq-benchmark (that @minrk contributed to). Is that still relevant?