Unable to cross compile due to cffi (0.14)

[Diaoul]

I maintain a Python package for Synology NAS at SynoCommunity/spksrc.
I had no problem cross compiling previous versions of pyOpenSSL, extensions were perfectly compiled, at compile time.
Since 0.14 it seems you switched to cffi which compiles the extensions at import time. The whole thing being done by cryptography under the hood. This is impossible to get to work on a cross compile environment because there is no compiler on the target. (SynoCommunity/spksrc#1225)

I wonder why the change, given that:

• having a compiler available (or the ability to compile) is a pretty strong requirement
• this is not a standard way to build extensions
• it worked before
• possible performance hit (seems to be the case "import OpenSSL" is 150x slower in 0.14 #137)

I'm obviously missing the pros of cffi here so please enlighten me.

Could you provide a way to avoid compilation at import time and do it at setup.py build time?

[exarkun]

Antoine Bertin notifications@github.com writes:

I maintain a Python package for Synology NAS at SynoCommunity/spksrc.
I had no problem cross compiling previous versions of pyOpenSSL, extensions were perfectly compiled, at compile time.
Since 0.14 it seems you switched to cffi which compiles the extensions at import time. The whole thing being done by cryptography under the hood. This is impossible to get to work on a cross compile environment because there is no compiler on the target.

Hi Antoine,

First off, sorry to have caused you trouble building pyOpenSSL. The
switch to cffi (by way of the cryptography module) was certainly an
invasive, pervasive one.

You can read about the reasoning behind the switch to using cffi here:

https://bugs.launchpad.net/pyopenssl/+bug/1152689

As you can see, cross compilation wasn't considered as part of that
decision. If I had a personal use-case for cross compilation I probably
would have thought to consider it. Also if anyone had raised this issue
while the cffi rewrite was underway or after any of the 0.14
pre-releases were issued then it might have been considered at that
time (hint hint: if you're interested in pyOpenSSL I encourage you to
follow releases and test pre-releases ;) Feedback about changes is
always welcome but it's always most welcome closer to development time).

I wonder why the change, given that:

• having a compiler available (or the ability to compile) is a pretty strong requirement

I'm not sure I fully understand this concern. It's true that building
pyOpenSSL required a C compiler before and that building pyOpenSSL
requires a C compiler now. It seems like not much has changed on that
front.

It is true that cffi includes a hook which will invoke the C compiler at
runtime if the internal extension module it wants to exist hasn't been
previously built. I think this is an interesting idea but probably not
a widely useful feature in the real world. I think some of the cffi
developers have come around to this thinking as well.

Are you aware of the ability, when using cffi, to build and distribute
the compiled version of the extension module and thus avoid the runtime
dependency on a C compiler?

Or have you found problems with using this functionality in practice
which result in the need to have a C compiler at runtime in order to
make cryptography usable? If this is the case, I wonder if it's that
cryptography is using cffi wrong, if cffi doesn't support
cross-compilation (I didn't think that distutils even supported
cross-compilation, but as I said above cross-compilation isn't something
that I've ever needed to dig into myself), or if there's just something
different about how cross-compilation is done for cffi.

Or perhaps there's something else I don't understand about this
functionality. :)

• this is not a standard way to build extensions

Given what I know about writing extension modules using the CPython/C
API directly and what I know about writing extension modules using cffi,
I strongly hope that cffi becomes the standard way. The ticket I
linked to above gives the objective rationale which led me to this
hope. Perhaps once we resolve the cross-compilation issues you'll also
become a believer in cffi. :)

• it worked before

Any change brings the possibility of breaking someone's usage of
pyOpenSSL. The best chance we have of never breaking the software for
anyone is to never change it. I think we can agree that's not an
option. :)

That said, I acknowledge that the switch to cffi was a big one and had
more than the usual level of risk of breaking the software for someone.
I tried to have a long lead up to the 0.14 release (the first based on
cffi). It was in development for over a year, there were several
pre-releases which I announced on this and other mailing lists, I
resurrected continuous integration testing practices for the project
before proceeding with the release, etc.

Clearly the efforts were insufficient - because it did break for you!
Again, I'm sorry about that. There is a limit to the amount of effort
available, though. I try to strike a good balance between keeping the
project moving and not rocking the boat for people. I can only try my
best, though.

• possible performance hit (seems to be the case "import OpenSSL" is 150x slower in 0.14 #137)

I considered the performance differences and decided to proceed anyway
for two main reasons.

First, for many parts of the pyOpenSSL API the actual hard work is being
done in OpenSSL and the performance of the bindings has minimal impact.
I verified this using Twisted's TLS benchmarks and found that for
performing actual TLS communication the difference was indeed minimal.
Of note here is that the benchmark I used benchmarks TLS
communication. It doesn't benchmark module import time. ;)

Second, the cffi-based implementation of pyOpenSSL is much faster than
the C-based implementation of pyOpenSSL on PyPy (it also has the
benefit of actually working and not relying on the (in my experience)
fragile "cpyext" layer in PyPy). I consider this to be an important
point because I consider PyPy to be the Python runtime that folks
interested in performance will be using. That is, a speedup on PyPy is
worth a slowdown on CPython (particularly if it's a big speedup vs a
small slowdown, which I judged (perhaps incorrectly) this change to be).

I know everyone can't switch to PyPy yet - but more and more people
should be able to, and making pyOpenSSL work on PyPy should also help
with that.

I'm obviously missing the pros of cffi here so please enlighten me.

I hope I've explained this sufficiently here.

Could you provide a way to avoid compilation at import time and do it at setup.py build time?

This sounds like it may bear further discussion. However, it probably
bears discussion on the cryptography mailing list - since pyOpenSSL
doesn't actually use cffi or include any extension modules.

My understanding is that setup.py build for cryptography will build
the extension module and prevent the need to have a C compiler at
runtime. If that's wrong, it sounds like something it would be good to
have fixed - in cryptography.

Thanks

[exarkun]

It was in development for over a year, there were several
pre-releases which I announced on this and other mailing lists

Hah. By "this" I meant the pyopenssl-users list hosted on python.org. Sorry about that, I got confused by my new mail client.

[Diaoul]

I didn't mean to be harsh, but after hours spend on pyOpenSSL (well, cryptography now) I was a bit pissed off :)
Thanks for the thorough explanation, I've been reading about cffi and it seems to be a good idea in the context of pyOpenSSL although the implementation (of cffi) seems really hacky to me.
You did enough job in communication, I just can't follow every software I package, that'd require a huge amount of time I don't have.

I did a bit of investigation but I still don't understand what exactly is going on. I'll try to explain briefly what I've done:

First I tried cross compiling cryptography like a standard module, it seems it installs cffi localy instead of relying on setuptools for the dependencies (weird, but not a problem). Then cffi is imported to do some additional steps I assume:

===>  Compiling for cryptography
cd /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4 && env PKG_CONFIG_LIBDIR=/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/lib/pkgconfig WORK_DIR=/home/spksrc/spksrc/spk/python/work-cedarview INSTALL_PREFIX=/usr/local/python TC=syno-cedarview  LD="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld" LDSHARED="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc -shared" CPP="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-cpp" NM="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-nm" CC="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc" AS="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-as" RANLIB="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-ranlib" CXX="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-g++" AR="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-ar" STRIP="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-strip" OBJDUMP="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-objdump" READELF="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-readelf" CFLAGS=" -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include " CPPFLAGS=" -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include " CXXFLAGS=" -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include " LDFLAGS=" -L/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/lib -Wl,--rpath-link,/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/lib -Wl,--rpath,/usr/local/python/lib " PYTHONPATH=/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/build/lib.linux-i686-2.7:/home/spksrc/spksrc/spk/python/work-cedarview/install/usr/local/python/lib/python2.7/site-packages/ /home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/hostpython setup.py build 
running build
Traceback (most recent call last):
  File "setup.py", line 174, in <module>
    "test": PyTest,
  File "/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Lib/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Lib/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Lib/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Lib/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "setup.py", line 88, in finalize_options
    self.distribution.ext_modules = get_ext_modules()
  File "setup.py", line 65, in get_ext_modules
    from cryptography.hazmat.primitives import constant_time, padding
  File "/home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/constant_time.py", line 49, in <module>
    _ffi = cffi.FFI()
  File "/home/spksrc/spksrc/spk/python/work-cedarview/install/usr/local/python/lib/python2.7/site-packages/cffi/api.py", line 56, in __init__
    import _cffi_backend as backend
ImportError: /home/spksrc/spksrc/spk/python/work-cedarview/install/usr/local/python/lib/python2.7/site-packages/_cffi_backend.so: wrong ELF class: ELFCLASS64

Obviously it fails, it tries to import a target library on the host machine.
So I do a little patching and disable the offending line with this patch. Everything works as expected, but when I try to import OpenSSL on the target, cffi tries to build the extensions which obviously fail as there is no compiler on the target.

So second thing I tried is to remove the patch and install a native cffi for the host python and tweak the PYTHONPATH so it get picked up when cross compiling.

===>  Compiling for cryptography
cd /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4 && env PKG_CONFIG_LIBDIR=/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/lib/pkgconfig WORK_DIR=/home/spksrc/spksrc/spk/python/work-cedarview INSTALL_PREFIX=/usr/local/python TC=syno-cedarview  LD="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-ld" LDSHARED="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc -shared" CPP="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-cpp" NM="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-nm" CC="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc" AS="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-as" RANLIB="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-ranlib" CXX="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-g++" AR="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-ar" STRIP="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-strip" OBJDUMP="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-objdump" READELF="/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-readelf" CFLAGS=" -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include " CPPFLAGS=" -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include " CXXFLAGS=" -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include " LDFLAGS=" -L/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/lib -Wl,--rpath-link,/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/lib -Wl,--rpath,/usr/local/python/lib " PYTHONPATH=/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/build/lib.linux-i686-2.7:/home/spksrc/spksrc/spk/python/work-cedarview/../../../native/python/work-native/install/usr/local/lib/python2.7/site-packages:/home/spksrc/spksrc/spk/python/work-cedarview/install/usr/local/python/lib/python2.7/site-packages/ /home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/hostpython setup.py build 
running build
self.ext_package: cryptography
running build_ext
building '_Cryptography_cffi_684bb40axf342507b' extension
creating /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/cryptography
creating /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat
creating /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives
creating /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__
/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include -fPIC -I/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Include -I/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8 -c cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.c -o /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.o
cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.c: In function '_cffi_f_Cryptography_constant_time_bytes_eq':
cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.c:224: warning: dereferencing type-punned pointer will break strict-aliasing rules
cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.c:239: warning: dereferencing type-punned pointer will break strict-aliasing rules
/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc -shared -L/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/lib -Wl,--rpath-link,/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/lib -Wl,--rpath,/usr/local/python/lib -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include -I/home/spksrc/spksrc/spk/python/work-cedarview/install//usr/local/python/include /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.o -L. -lpython2.7 -o /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.so
Traceback (most recent call last):
  File "setup.py", line 174, in <module>
    "test": PyTest,
  File "/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Lib/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Lib/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Lib/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/home/spksrc/spksrc/spk/python/work-cedarview/Python-2.7.8/Lib/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "setup.py", line 88, in finalize_options
    self.distribution.ext_modules = get_ext_modules()
  File "setup.py", line 65, in get_ext_modules
    from cryptography.hazmat.primitives import constant_time, padding
  File "/home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/constant_time.py", line 54, in <module>
    ext_package="cryptography",
  File "/home/spksrc/spksrc/native/python/work-native/install/usr/local/lib/python2.7/site-packages/cffi/api.py", line 340, in verify
    lib = self.verifier.load_library()
  File "/home/spksrc/spksrc/native/python/work-native/install/usr/local/lib/python2.7/site-packages/cffi/verifier.py", line 75, in load_library
    return self._load_library()
  File "/home/spksrc/spksrc/native/python/work-native/install/usr/local/lib/python2.7/site-packages/cffi/verifier.py", line 153, in _load_library
    return self._vengine.load_library()
  File "/home/spksrc/spksrc/native/python/work-native/install/usr/local/lib/python2.7/site-packages/cffi/vengine_cpy.py", line 149, in load_library
    raise ffiplatform.VerificationError(error)
cffi.ffiplatform.VerificationError: importing '/home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.so': /home/spksrc/spksrc/spk/python/work-cedarview/cryptography-0.5.4/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.so: wrong ELF class: ELFCLASS64

Here we can see the .so is compiled as expected but then it tries to import it (wtf?). This is where I'm lost, I don't get why it tries to import it and it seems to be an important step in the process done by cffi.
If I can get rid of that step I'll get a working (and compiled) cryptography which won't trigger a compilation on the target.

The tests are done by compiling on a Debian 32bits VM and building for a x86_64 target but I have plenty other targets (ARM, PPC, etc.).

Any help appreciated here 👍

[Diaoul]

From what I understand cryptography tries to import itself during setup (get_ext_modules in setup.py) to trigger compilation (in cryptography/hazmat/primitives/constant_time.py: _lib = _ffi.verify).
Calling verify results in the module being compiled (that's OK) but the imported (that's not).

This is the portion of the code used by cffi:

def load_library(self):
    """Get a C module from this Verifier instance.
    Returns an instance of a FFILibrary class that behaves like the
    objects returned by ffi.dlopen(), but that delegates all
    operations to the C module.  If necessary, the C code is written
    and compiled first.
    """
    with self.ffi._lock:
        if not self._has_module:
            self._locate_module()
            if not self._has_module:
                if not self._has_source:
                    self._write_source()
                self._compile_module()
        return self._load_library()

The problem here is self._load_library().

In setup.py, cryptography should not import itself, this is considered a bad practice in general, but cffi's documentation says otherwise. While this is OK in most cases (host == target) it is not in case if cross compilation.

I think the correct way to do that would be to call compile_module on the Verifier object instead of calling the verify on the FFI object during setup.

The issue is not in pyOpenSSL or cryptography but really in the internals of cffi.
I think now you have a better understanding of what happens inside cffi during setup and that you see how hacky it is compared to the standard python C-extensions.

[Diaoul]

I've tested the solution that consists of calling compile_module on the Verifier object: it works for constant_time and padding, I have the extensions built correctly.

def get_ext_modules():
    from cryptography.hazmat.bindings.commoncrypto.binding import (
    Binding as CommonCryptoBinding
    )
    from cryptography.hazmat.bindings.openssl.binding import (
    Binding as OpenSSLBinding
    )
    from cryptography.hazmat.primitives import constant_time, padding

    ext_modules = [
    OpenSSLBinding().ffi.verifier.get_extension(),
    constant_time._ffi.verifier.get_extension(),
    padding._ffi.verifier.get_extension()
    ]
    if CommonCryptoBinding.is_available():
    ext_modules.append(CommonCryptoBinding().ffi.verifier.get_extension())
    return ext_modules

It fails for OpenSSLBinding because the verify call is deeper in the code in cryptography/hazmat/bindings/utils.py and I don't know how to modify this.

At least it seems that we're on the right tracks to fix the issue, I'd be glad to test a patch for this.

[hynek]

Since we won’t move from CFFI anytime soon and seems to be fixed-ish in cryptography I’m gonna close this.