Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to cross compile due to cffi (from pyopenssl) #1325

Closed
Diaoul opened this issue Sep 7, 2014 · 24 comments
Closed

Unable to cross compile due to cffi (from pyopenssl) #1325

Diaoul opened this issue Sep 7, 2014 · 24 comments

Comments

@Diaoul
Copy link

Diaoul commented Sep 7, 2014

More information here: pyca/pyopenssl#157

Le me kow if we should continue the discussion here or in the pyopenssl issue.

@Diaoul
Copy link
Author

Diaoul commented Sep 21, 2014

The issue is with how cffi works (see the bold part in this comment).

The issue with cryptography importing itself to trigger compilation of ifs extensions through cffi. This is a major design flaw IMO.

@reaperhulk
Copy link
Member

We've had some discussions in the past about the way we currently use cffi, but I believe we've mostly been in a holding pattern waiting on cffi 1.0 (which has the removal of compile on import as a design goal). /cc @dstufft

@dstufft
Copy link
Member

dstufft commented Sep 25, 2014

It's not a requirement that importing triggers the compilation, but you have to be very careful not to trigger it. Cryptography could do what I've done in my own projects and disable the implicit compile all together and only allow compiling via setup.py. However I believe @alex was against this?

@alex
Copy link
Member

alex commented Sep 25, 2014

I'm not strictly against this, but I want to make sure we integrate it into
the test runner so just running py.test also triggers compile.

On Thu, Sep 25, 2014 at 1:19 PM, Donald Stufft notifications@github.com
wrote:

It's not a requirement that importing triggers the compilation, but you
have to be very careful not to trigger it. Cryptography could do what I've
done in my own projects and disable the implicit compile all together and
only allow compiling via setup.py. However I believe @alex
https://github.com/alex was against this?


Reply to this email directly or view it on GitHub
#1325 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@Diaoul
Copy link
Author

Diaoul commented Oct 12, 2014

Please tell me when I can test against master

@Ayrx
Copy link
Contributor

Ayrx commented Oct 25, 2014

@dstufft Can you point me to one of your projects that disable the implicit compile step? I'll see what I can do with this.

@dstufft
Copy link
Member

dstufft commented Oct 25, 2014

https://github.com/dstufft/http11/blob/master/src/http11/c/__init__.py#L102-L156

The important part here is that by constucting a Verifier manually we skip the implicit compile that the verify call does. The Library class just acts like a lazy proxy that will ensure that the library is loaded (and that using the Verifier's compile method raises an error) but will only load it once an attribute is accessed instead of on import. Then the important thing to do is make sure that importing never accessing one of those attributes.

@dstufft
Copy link
Member

dstufft commented Nov 8, 2014

This may be fixed in #1467 which avoids the implicit compile when running setup.py and only has distutils building it. This won't be reflected when running via tox (see #1468 for that problem).

@alex
Copy link
Member

alex commented Nov 15, 2014

I believe this is fixed now; going to close the issue, if there's any new issues let's file a new issue.

@alex alex closed this as completed Nov 15, 2014
@Diaoul
Copy link
Author

Diaoul commented Nov 15, 2014

Is this in a release already or do I need to use master?

@alex
Copy link
Member

alex commented Nov 15, 2014

No, it's not in a release yet; it'll be in the next one.

On Sat Nov 15 2014 at 10:06:24 AM Antoine Bertin notifications@github.com
wrote:

Is this in a release already or do I need to use master?


Reply to this email directly or view it on GitHub
#1325 (comment).

@yegorich
Copy link

yegorich commented Feb 6, 2015

I still have problems cross-compiling cryptography in Buildroot. I've tried both 0.7.2 release and master branch, but I always get the issue with _cffi_backend.so:

python-cryptography custom Building
(cd /home/user/Documents/versioned/buildroot/output/build/python-cryptography-custom//; PATH="/home/user/Documents/versioned/buildroot/output/host/bin:/home/user/Documents/versioned/buildroot/output/host/sbin:/home/user/Documents/versioned/buildroot/output/host/usr/bin:/home/user/Documents/versioned/buildroot/output/host/usr/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games" PYTHONPATH="/home/user/Documents/versioned/buildroot/output/target/usr/lib/python2.7/sysconfigdata/:/home/user/Documents/versioned/buildroot/output/target/usr/lib/python2.7/site-packages/" _python_sysroot=/home/user/Documents/versioned/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot _python_prefix=/usr _python_exec_prefix=/usr /home/user/Documents/versioned/buildroot/output/host/usr/bin/python setup.py build )
running build
Traceback (most recent call last):
File "setup.py", line 339, in
**keywords_with_side_effects(sys.argv)
File "/home/user/Documents/versioned/buildroot/output/host/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/home/user/Documents/versioned/buildroot/output/host/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/home/user/Documents/versioned/buildroot/output/host/usr/lib/python2.7/distutils/dist.py", line 971, in run_command
cmd_obj.ensure_finalized()
File "/home/user/Documents/versioned/buildroot/output/host/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
File "setup.py", line 105, in finalize_options
self.distribution.ext_modules = get_ext_modules()
File "setup.py", line 76, in get_ext_modules
from cryptography.hazmat.bindings.commoncrypto.binding import (
File "src/cryptography/hazmat/bindings/commoncrypto/binding.py", line 12, in
class Binding(object):
File "src/cryptography/hazmat/bindings/commoncrypto/binding.py", line 34, in Binding
"-framework", "Security", "-framework", "CoreFoundation"
File "src/cryptography/hazmat/bindings/utils.py", line 97, in build_ffi_for_binding
extra_link_args=extra_link_args,
File "src/cryptography/hazmat/bindings/utils.py", line 105, in build_ffi
ffi = FFI()
File "/home/user/Documents/versioned/buildroot/output/target/usr/lib/python2.7/site-packages/cffi/api.py", line 56, in init
import _cffi_backend as backend
ImportError: /home/user/Documents/versioned/buildroot/output/target/usr/lib/python2.7/site-packages/_cffi_backend.so: wrong ELF class: ELFCLASS32
package/pkg-generic.mk:182: recipe for target '/home/user/Documents/versioned/buildroot/output/build/python-cryptography-custom/.stamp_built' failed
make: *** [/home/user/Documents/versioned/buildroot/output/build/python-cryptography-custom/.stamp_built] Error 1

@ghost
Copy link

ghost commented Jun 26, 2015

Hello,

Has this been resolved yet? It still fails in Buildroot during cross-compilation. Tried version cryptography 0.9.1 and cffi 1.1.2.

Traceback (most recent call last):
File "./setup.py", line 307, in
**keywords_with_side_effects(sys.argv)
File "/home/ram/.../usr/lib/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
File "/home/ram/.../usr/lib/python2.7/setuptools/dist.py", line 269, in init
_Distribution.init(self,attrs)
File "/home/ram/.../usr/lib/python2.7/distutils/dist.py", line 287, in init
self.finalize_options()
File "/home/ram/.../usr/lib/python2.7/setuptools/dist.py", line 325, in finalize_options
ep.load()(self, ep.name, value)
File "/home/ram/.../usr/lib/python2.7/cffi/setuptools_ext.py", line 154, in cffi_modules
add_cffi_module(dist, cffi_module)
File "/home/ram/.../usr/lib/python2.7/cffi/setuptools_ext.py", line 48, in add_cffi_module
execfile(build_file_name, mod_vars)
File "/home/ram/.../usr/lib/python2.7/cffi/setuptools_ext.py", line 24, in execfile
exec(code, glob, glob)
File "src/_cffi_src/build_openssl.py", line 96, in
extra_link_args=extra_link_args(sys.platform),
File "./src/_cffi_src/utils.py", line 65, in build_ffi_for_binding
extra_link_args=extra_link_args,
File "./src/_cffi_src/utils.py", line 73, in build_ffi
ffi = FFI()
File "/home/ram/.../usr/lib/python2.7/cffi/api.py", line 56, in init
import _cffi_backend as backend
ImportError: /home/.../usr/lib/python2.7/_cffi_backend.so: wrong ELF class: ELFCLASS32

Any help would be appreciated!

@xuhcc
Copy link

xuhcc commented Jul 31, 2015

I've got the same error when cross-compiling using OpenEmbedded build system: _cffi_backend.so: wrong ELF class: ELFCLASS32.
Tried cryptography 0.9.3 and current master.

@alexykot
Copy link

alexykot commented Aug 6, 2015

This also affects my project while trying to cross-compile cryptography for ARM embedded system, with the same import error:

ImportError: /home/vagrant/poky/build/tmp/sysroots/qemuarm/usr/lib/python2.7/site-packages/_cffi_backend.so: wrong ELF class: ELFCLASS32

@reaperhulk
Copy link
Member

@alexykot what happens if you use the master branch rather than 0.9.3?

@alexykot
Copy link

alexykot commented Aug 6, 2015

Just tried that, same thing happens.

ImportError: /home/vagrant/poky/build/tmp/sysroots/qemuarm/usr/lib/python2.7/site-packages/_cffi_backend.so: wrong ELF class: ELFCLASS32

Do you need full traceback?

@reaperhulk
Copy link
Member

No, this looks like it might be something cffi has to handle. This is outside my experience so I'm not sure how to go about debugging it though.

@alexykot
Copy link

alexykot commented Aug 6, 2015

Bummer. OK, I'll try to get to cffi about this. Thanks for swift reply.
On 6 Aug 2015 10:12 pm, "Paul Kehrer" notifications@github.com wrote:

No, this looks like it might be something cffi has to handle. This is
outside my experience so I'm not sure how to go about debugging it though.


Reply to this email directly or view it on GitHub
#1325 (comment).

@ghost
Copy link

ghost commented Aug 6, 2015

@reaperhulk @alexykot I don't think it's a cffi issue. I tried to cross compile for a 32 bit system from a 64 bit system. cffi module _cffi_backend.so compiled OK for the target (that is 32 bit). The problem is that when cryptography tries to pre-cross compile (for 32 bit) cffi modules (Cryptography_cffi*.so), it uses (host) 64 bit compiler flags and tries to load the 32 bit cross compiled _cffi_backend.so. This is why you get "wrong ELF class: ELFCLASS32".

There needs to be some way to configure the C compiler flags from within the cryptography package while cross compiling the cffi modules. You can see how cffi module itself is allowing that in its setup.py file. Another python package I was able to configure the target c flags is python readline.

I got around this problem (dirty hack) by skipping the cffi modules pre-cross compilation process (Cryptography_cffi*.so) and copying (during the build time) those modules I compiled on the target system.

@reaperhulk
Copy link
Member

This sounds like the sort of thing that should be possible via setuptools magic. @dstufft?

@AndreMiras
Copy link

Well I could get it working myself by following midicase comment on vincentbernat/snimpy.

  1. build and install host cffi
  2. build and install target packages that depend in cffi
  3. build and install target cffi
    I've created a detailed install instructions https://github.com/AndreMiras/km/wiki/CFFI-Cross-Compile.

@alexykot
Copy link

alexykot commented Aug 7, 2015

Cool, I'll try this workaround asap and see if it would work.

@yegorich
Copy link

@vtmram FYI: python-cryptography and python-pyopenssl were both taken into Buildroot's next branch (https://git.busybox.net/buildroot/commit/?h=next&id=fbeeb4c8cb4e44c654ca27db9ae05a83202f4104)

As @AndreMiras pointed the trick was to install both host and target versions of CFFI. See python-cryptography for details.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

9 participants