Libsodium not bundled in Windows wheels

[chrisjbillington]

I was dissatisfied with the speed of zeromq curve encryption, until I noticed that it appears to vary by an order of magnitude across different ways of installing pyzmq. This script:

from __future__ import print_function

import sys
import os
import time
import zmq
import zmq.auth.thread

context = zmq.Context()
auth = zmq.auth.thread.ThreadAuthenticator(context)
auth.start()
auth.configure_curve(domain='*', location=zmq.auth.CURVE_ALLOW_ANY)
del auth # Ensure auth thread doesn't keep the interpreter open

server_publickey, server_secretkey = zmq.curve_keypair()
client_publickey, client_secretkey = zmq.curve_keypair()


server = context.socket(zmq.PULL)
server.curve_publickey = server_publickey
server.curve_secretkey = server_secretkey
server.curve_server = True
port = server.bind_to_random_port('tcp://127.0.0.1')

client = context.socket(zmq.PUSH)
client.curve_secretkey = client_secretkey
client.curve_publickey = client_publickey
client.curve_serverkey = server_publickey
client.connect('tcp://127.0.0.1:%d' % port)

print('Python Version: %s' % ' | '.join(sys.version.splitlines()))
print('Platform: %s' % sys.platform)
print('pyzmq version: %s'%zmq.__version__)
print('zmq version: %s'%zmq.zmq_version())

plaintext = os.urandom(100*1024**2)
start_time = time.time()
client.send(plaintext)
assert server.recv() == plaintext
print("time to encrypt and send 100MB:", time.time() - start_time, 's')

Returns the following for all the different Pythons I have installed, where I have anaconda2 and anaconda3, which came with pyzmq, system python 2 and 3, for which I installed pyzmq via apt, and two virtualenvs that I made using system Python 2 and 3, and installed pyzmq using pip install pyzmq --install-option="--zmq=bundled"

$ /usr/bin/python2.7 test.py 
Python Version: 2.7.14 (default, Sep 23 2017, 22:06:14)  | [GCC 7.2.0]
Platform: linux2
pyzmq version: 16.0.2
zmq version: 4.2.1
time to encrypt and send 100MB: 0.500539064407 s

$ /usr/bin/python3.6 test.py 
Python Version: 3.6.3 (default, Oct  3 2017, 21:45:48)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 16.0.2
zmq version: 4.2.1
time to encrypt and send 100MB: 0.49707913398742676 s

$ ~/anaconda2/bin/python test.py 
Python Version: 2.7.13 |Anaconda, Inc.| (default, Sep 30 2017, 18:12:43)  | [GCC 7.2.0]
Platform: linux2
pyzmq version: 17.0.0
zmq version: 4.1.6
~/anaconda3/bin/python test.py 
time to encrypt and send 100MB: 6.23277807236 s

$ ~/anaconda3/bin/python test.py 
Python Version: 3.6.3 |Anaconda custom (64-bit)| (default, Nov  3 2017, 19:19:16)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 16.0.3
zmq version: 4.2.2
time to encrypt and send 100MB: 2.81573748588562 s

$ ~/tmp/venvs/zmqtestpy2/bin/python test.py 
Python Version: 2.7.14 (default, Sep 23 2017, 22:06:14)  | [GCC 7.2.0]
Platform: linux2
pyzmq version: 17.0.0
zmq version: 4.1.6
time to encrypt and send 100MB: 6.02314901352 s

$ ~/tmp/venvs/zmqtestpy3/bin/python test.py 
Python Version: 3.6.3 (default, Oct  3 2017, 21:45:48)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 17.0.0
zmq version: 4.1.6
time to encrypt and send 100MB: 6.559395790100098 s

As you can see, the speed of encryption varies a lot, and in particular, is slowest with the fresh virtualenvs with the pip install and bundled zmq, and is fastest with the system versions.

At first it seems like zeromq 4.2 is what makes it fast - until you see that anaconda3 with zmq 4.2.2 is still way slower than the system ones. So it seems it's probably something about configuration rather than the specific versions of zmq in use.

So it appears maybe the bundled zeromq is a a bit slow with the encryption, and it would be better if the default install resulted in fast crypto. My users are mostly using anaconda, so if whatever ends up in anaconda could be as fast as the versions in the Ubuntu repos, that would be great!

[chrisjbillington]

It looks like the bundled zeromq probably isn't compiled with libsodium (unsurprising since it doesn't bundle libsodium), and the built-in tweetnacl is not as fast.

And since the conda package doesn't depend on libsodium, I'm guessing it uses tweetnacl as well.

[chrisjbillington]

Confirm that when I install from source using my system zeromq (which is compiled against libsodium), then the result is fast encryption.

[chrisjbillington]

So I made a mistake: as mentioned in the anaconda thread, pyzmq's conda package does depend on zeromq, and it links against it correctly. But it doesn't link against libsodium. So the problem with pyzmq not having fast crypto in conda is probably a problem with zeromq's packaging there, not pyzmq.

The remaining non-ideal situation for pyzmq is that its bundled zeromq does not include libsodium, so a pip-installed pyzmq has slow crypto. Even if you have libsodium and libzmq, pip install by default on linux installs a precompiled wheel which seemingly doesn't use libsodium.

[jjhelmus]

I believe this should be addressed with the 4.2.3 build 3 zeromq packages. @chrisjbillington can you run your benchmarks using these packages?

[chrisjbillington]

Thanks! I can confirm that with the _3 build of conda zeromq I get fast crypto:

Python Version: 3.6.3 |Anaconda custom (64-bit)| (default, Nov  3 2017, 19:19:16)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 17.0.0
zmq version: 4.2.3
time to encrypt and send 100MB: 0.5163590908050537 s

But with pip installed pyzmq (pyzmq-17.0.0-cp36-cp36m-manylinux1_x86_64.whl) in the same environment (after conda uninstall pyzmq) I still get slow crypto:

Python Version: 3.6.3 |Anaconda custom (64-bit)| (default, Nov  3 2017, 19:19:16)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 17.0.0
zmq version: 4.1.6
time to encrypt and send 100MB: 6.260281085968018 s

indicating there is a remaining problem with the wheel that pip installs but not with the conda pyzmq or zeromq packages.

[minrk]

The pyzmq linux wheels are meant to link libsodium, but I checked and you are right that they don't. I've fixed the builder so the next pyzmq wheels on Linux will bundle libsodium.

[chrisjbillington]

Thanks!

And just to clarify, the Windows wheels do not link to libsodium and are not meant to, is that correct?

[minrk]

Right.

[bluca]

Please don't use tweetnacl in production anywhere - it would be preferable to disable curve completely

[mingwandroid]

And just to clarify, the Windows wheels do not link to libsodium and are not meant to, is that correct?

Can anyone explain why this is?

[mingwandroid]

.. and if:

Windows wheels do not link to libsodium and are not meant to

.. and given @bluca said ..

Please don't use tweetnacl in production anywhere - it would be preferable to disable curve completely

.. how are people meant to use curve encryption with pyzmq on Windows?

[minrk]

I'm not opposed to using libsodium in the Windows wheels, I just don't know how to do it. Help would be welcome because I don't have time to work on it and won't for the foreseeable future. @SylvainCorlay recently fixed the zeromq builds in conda-forge to link libsodium and pyzmq to link zeromq, so if you use pyzmq from conda-forge you should get zeromq with libsodium. So my recommendation for Windows users would be to use conda to get pyzmq.

[mingwandroid]

I have this working now afaict, just need to release the packages. If the conda-forge are using zeromq < 4.2.4 then expect lots of bugs.

[SylvainCorlay]

Conda-forge uses "4.2.5".

[mingwandroid]

Ah good.

[minrk]

Windows wheels now have properly built libsodium

[chrisjbillington]

🥳 Fantastic.

Now I can recommend venvs for pyzmq packages on Windows with little downside!

@minrk FWIW the latest pyzmq windows conda package has no curve support at all due to zeromq not linking to libsodium (and I guess the tweetnacl fallback is either disabled or has been removed):

ContinuumIO/anaconda-issues#13339

Obviously this is anaconda's bug and not yours, but thought you might want to know.

[minrk]

Yup, seems like an anaconda issue. The conda-forge zeromq package on Windows links libsodium, I believe. If that's not the case, please report it

[minrk]

Nope, Windows conda-forge builds don't link libsodium, despite depending on it. I'll look at fixing that.

[minrk]

Fixed on conda-forge: conda-forge/zeromq-feedstock#71