TypeError exception (FFI related) when used in a multi threaded environment

[uri247]

I'm getting the following exception:

  File ".../hazmat/primitives/serialization.py", line 20, in load_pem_private_key
    return backend.load_pem_private_key(data, password)
  File ".../hazmat/backends/multibackend.py", line 276, in load_pem_private_key
    return b.load_pem_private_key(data, password)
  File ".../hazmat/backends/openssl/backend.py", line 691, in load_pem_private_key
    password,
  File ".../hazmat/backends/openssl/backend.py", line 831, in _load_key
    self._ffi.NULL
TypeError: initializer for ctype 'int(*)(char *, int, int, void *)' must be a pointer to same type, not cdata 'int(*)(char *, int, int, void *)'

The exception isn't deterministic, but it is guaranteed to happen after a short while. I'm running from a wsgi module within apache (mod_apache). I'm running cryptography 0.8.2

The code that I'm running is:

import cryptography.hazmat.backends
import cryptography.hazmat.primitives.serialization

with open(my_pem_file, 'rb') as f:
    pem = f.read()

pkey = cryptography.hazmat.primitives.serialization.load_pem_private_key(
    pem, password=None, backend=cryptography.hazmat.backends.default_backend())

From logs on the calling module, I'm learning that it when it happens it is always the first time I'm calling load_pem_private_key from a new thread.

[reaperhulk]

Thanks for the report. 0.8.2 contained a fix that was supposed to prevent this, but apparently it was incomplete. Are you able to replicate the problem if you replace cryptography.hazmat.backends.default_backend() with backend and add the import line from cryptography.hazmat.backends.openssl.backend import backend?

[uri247]

Thank you for replying so fast.

Yes, switching to openssl.backend also reproduce the same bug. It only short circuit the line 'multibackend' (obviously) and from serialization.py the trace goes directly to openssl/backend.py

File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/primitives/serialization.py", line 20, in load_pem_private_key
    return backend.load_pem_private_key(data, password)
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 691, in load_pem_private_key
    password,
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 831, in _load_key
    self._ffi.NULL
TypeError: initializer for ctype 'int(*)(char *, int, int, void *)' must be a pointer to same type, not cdata 'int(*)(char *, int, int, void *)'

[alex]

Hmm. I'm not sure I understand how this can happen, it seems like self._ffi should be the same from both _pem_password_cb and inside of _load_key.

Usually these are caused by loading the cffi library multiple times, but this doesn't look like it.

If you call other functions in cryptography, do you still see this issue?

[uri247]

I'm not doing many different types of crypto, so loading private key is typically the first thing I'm doing, and when this fail - there isn't a point to continue with other stuff. I'm not sure whether the exact trace, in the past I also did symmetric cryptography (using Fernet), and I had issues appearing only in Apache (as opposed to the reference wsgi server, which is single threaded).

[public]

Are you running mod_python with multiple threads per backend process or just multiple pre-forked worker processes?

[uri247]

Alex,

I’ve now switched to nginx -> uwsgi, and everything is working fine.

It is definitely a mod_wsgi (note: not mod_python, but mod_wsgi) only thing. Moreover, trying to repro the issue, I wrote a short sample program that would hammer the crypto simultaneously (attached) from multiple threads, but the package work just fine. I’ve also read the cryptography code, and couldn’t find anything. It appears mod_wsgi is doing something more evolving, maybe with FFI. At that point I decided to switch application server.

Apache working with multiple threads in the backend, and the logs shows that mod_wsgi creates only one process (different than httpd), with multiple threads. The problem happens on a first call from a different thread.

u.

From: Alex Stapleton [mailto:notifications@github.com]
Sent: Wednesday, May 6, 2015 4:10 PM
To: pyca/cryptography
Cc: Uri London
Subject: Re: [cryptography] TypeError exception (FFI related) when used in a multi threaded environment (#1868)

Are you running mod_python with multiple threads per backend process or just multiple pre-forked worker processes?

—
Reply to this email directly or view it on GitHub #1868 (comment) . https://github.com/notifications/beacon/AAx9r__cNtsCgmU-7VYrCLzyOYw4aKLPks5oGgoegaJpZM4EI-nh.gif

import time
import thread
from threading import Thread, Event
import itertools
import logging
from cryptography.hazmat.backends.openssl import backend
from cryptography.hazmat.primitives.serialization import load_pem_private_key, load_pem_public_key
from cryptography.hazmat.primitives.asymmetric import padding

logging.getLogger().setLevel(logging.INFO)
num_threads = 20
count_ready = itertools.count()
count_complete = itertools.count()
rendezvous = Event()
complete = Event()

def read_public():
return load_pem_public_key(public_pem, backend=backend)

def read_private():
return load_pem_private_key(private_pem, password=None, backend=backend)

def encrypt(msg):
pub_key = read_public()
cipher = pub_key.encrypt(msg, padding.PKCS1v15())
return cipher

def decrypt(cipher):
priv_key = read_private()
msg = priv_key.decrypt(cipher, padding.PKCS1v15())
return msg

def decrypt_wrapper(cipher, result, thread_index):
logging.info('decrypting thread no. %i, thread id = %d', thread_index, thread.get_ident())
c = count_ready.next()
if c == num_threads - 1:
print 'ready ',
for i in range(4):
print '.',
time.sleep(1)
print ' go'
rendezvous.set()
rendezvous.wait()
msg = decrypt(cipher)
print 'thread %02d: %s' % (thread_index, msg)
result[thread_index] = msg
c = count_complete.next()
if c == num_threads - 1:
complete.set()

def single_thread_way():
print 'the single thread way:'
orig_msg = 'The beauty is in the eyes of the beholder'
print 'original message:', orig_msg
cipher = encrypt(orig_msg)
roundtrip_msg = decrypt(cipher)
print 'roundtrip message:', roundtrip_msg

def multi_thread_way():
print 'the multi thread way:'
orig_msg = 'The beauty is in the eyes of the beholder'
print 'original message:', orig_msg
cipher = encrypt(orig_msg)
results = ['' for i in range(num_threads)]
threads = [Thread(target=decrypt_wrapper, args=(cipher, results, i)) for i in range(num_threads)]
for t in threads:
t.start()
complete.wait()

def main():
multi_thread_way()

private_pem = '''-----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQCqPHd71AEqiKOom4k/pglfqwySd9OFac6A/j64R8Xf4LdnokHc
oIWdZbX4Wn6A9rNzKJ7/7IJZPcyP/lGglCyFuA3aG8HqcGFytFFblnFYOYTmpFjH
pUOHFTzaOwRVAkwszwuyuJF3NpYeSnSyFumsDJFV/PkV+a6Az4/PpH9uTwIDAQAB
AoGACqu3oZ9sY0olWBBHziGOPNzv8M5qB9bWBufo7owxLxNa67gUAMxfE7qoaWTi
wRq1rOZC8S/WC9n/1JbQbwxarfzq0Bmz15y1XB8l36KtXeEbS/Q0Jt50EHgxkRHr
S0kcxMVqKAESlsMf6Ph0fmZXnR7aUWWaUO/dar+tpi4xNiECQQDUDiFYUlSngJqs
f0zdXUUBrqlncWeq/1GiJY8zQYg0K1S6/b/7HzpFHb/bpaGLtrjt2Ty3scJDg9Ry
MmYe8rRbAkEAzYPFca3SEtnQBbO48Jplkn9tYMrERVz0H/UQe+XfOc83Kc7LZ0le
PRTngikfUr0UHoE+aq/crIw5YhODNNMAHQJBAJ8PxlVg1F0cq2juyDJXzvZXFH0i
ZLewcoRlFghFhKxVQGZPBp2Qq/3CNfLFR+rr8cV/qPrFXMmV7lqGkXFvbncCQFhd
h0D43zHRy7sX6rYxfOj/t3T92mSEskcnZR4Q2emOaoakbxsLFeUnnt99dQVrXUI8
iizvTvA2HtpHz4ugChUCQQDGpGe89/w4KsM5BOEPUncjCGH/qCTPYotK/VSKqI7o
rYj1g6xQt4/5bsbN7BRBTOdnWqdo1AYews50wBNFj3zp
-----END RSA PRIVATE KEY-----
'''

public_pem = '''-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCqPHd71AEqiKOom4k/pglfqwyS
d9OFac6A/j64R8Xf4LdnokHcoIWdZbX4Wn6A9rNzKJ7/7IJZPcyP/lGglCyFuA3a
G8HqcGFytFFblnFYOYTmpFjHpUOHFTzaOwRVAkwszwuyuJF3NpYeSnSyFumsDJFV
/PkV+a6Az4/PpH9uTwIDAQAB
-----END PUBLIC KEY-----
'''

if name == 'main':
main()

[reaperhulk]

This problem will possibly go away with cffi 1.0 (when using the new precompile features). Let's leave this open for now and revisit once 1.0 is out and we've upgraded cryptography to use the new code paths.

[sigmavirus24]

This and #1776 definitely seem to be fixed by using master. 👍 Good work @reaperhulk and everyone else involved

[reaperhulk]

The 1.0 release (expect it in ~2 weeks) will contain this fix, so closing this.