[Optimization request] Support buffers

[za3k]

Can you support buffer objects, in particular allow read access to memoryview on bytes in CPython only? My understanding is that the CFFI disallows use of memoryview on bytes because it wants to present a uniform interface for PyPy, but indexing into the underlying bytes object actually works fine as long as you know you're on CPython.

Discussion here links to some bugs that were open at the time: https://mail.python.org/pipermail/cryptography-dev/2015-December/000532.html

This would reduce the number of copies from 2-3 to 1 and allow streaming/chunked interaction with encryption and decryption.

This would be a CPython-only improvement.

[alex]

Since cffi doesn't support it, I don't really see how it'd be possible for us.

[za3k]

CFFI doesn't allow memoryview directly, but using pointer arithmetic on the underlying bytes object you can get around that, as far as I could tell?

[alex]

Can you show an example of what you're talking about? At first glace, this sounds incredibly subtle and non-portable.

[za3k]

It is incredibly non-portable and should be carefully wrapped in if CPython. But I think it would reduce the number of memory copies on cpython by something like 3X.

I'll submit an example patch for one of the functions.

[za3k]

Okay, it looks like it's actually impossible for two reasons:

1. CFFI disallows bytes as well, not just a memoryview backed by 'bytes'
2. It's impossible to get at the object a memoryview is wrapping in Python, although it's possible from Python's C API.

Too bad, it would have been really nice.

[stepheny]

Following the the mail link in the original post and referenced cffi issues, it seems that CFFI had finished some further development since then.
According to https://cffi.readthedocs.io/en/latest/ref.html#ffi-buffer-ffi-from-buffer, ffi.from_buffer now should be able to handle memoryview as expected.
But cffi would not implicitly convert python buffer to a char *, it has to be converted explicitly.
I'm not confident wrapping all possible usages would gain to performance. But zero copy will never harm performance. The result may be interesting.

[reaperhulk]

As of cryptography 2.5 we support the buffer protocol for many key inputs (and we have update_into in symmetric cryptography to write into a rw buffer).

[stepheny]

Great! It's a coincidence that I was looking into the dsa sign and verify part and totally missed update_into. Thank you for you reply!