I Keep getting OSError error: _raw_ecb.x86_64-linux-gnu.so: cannot open shared object file: #440

John-Oula · 2020-08-13T09:14:19Z

Python 3.7.3
Pycryptodome
Ubuntu 16.04

Tried running with python 3.5 and got the same error but when i do the self test it runs successfully under python 3.5

[Thu Aug 13 12:04:38.419037 2020] [wsgi:error] [pid 27768] [remote 157.55.39.158:13599] from Cryptodome.Cipher import AES
[Thu Aug 13 12:04:38.419041 2020] [wsgi:error] [pid 27768] [remote 157.55.39.158:13599] File "/usr/local/lib/python3.7/site-packages/Cryptodome/Cipher/init.py", line 27, in
[Thu Aug 13 12:04:38.419047 2020] [wsgi:error] [pid 27768] [remote 157.55.39.158:13599] from Cryptodome.Cipher._mode_ecb import _create_ecb_cipher
[Thu Aug 13 12:04:38.419051 2020] [wsgi:error] [pid 27768] [remote 157.55.39.158:13599] File "/usr/local/lib/python3.7/site-packages/Cryptodome/Cipher/_mode_ecb.py", line 47, in
[Thu Aug 13 12:04:38.419058 2020] [wsgi:error] [pid 27768] [remote 157.55.39.158:13599] """
[Thu Aug 13 12:04:38.419061 2020] [wsgi:error] [pid 27768] [remote 157.55.39.158:13599] File "/usr/local/lib/python3.7/site-packages/Cryptodome/Util/_raw_api.py", line 308, in load_pycryptodome_raw_lib
[Thu Aug 13 12:04:38.419068 2020] [wsgi:error] [pid 27768] [remote 157.55.39.158:13599] raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
[Thu Aug 13 12:04:38.419079 2020] [wsgi:error] [pid 27768] [remote 157.55.39.158:13599] OSError: Cannot load native module 'Cryptodome.Cipher._raw_ecb': Trying '_raw_ecb.x86_64-linux-gnu.so': /usr/local/lib/python3.7/site-packages/Cryptodome/Util/../Cipher/_raw_ecb.x86_64-linux-gnu.so: cannot open shared object file: No such file or directory, Trying '_raw_ecb.so': /usr/local/lib/python3.7/site-packages/Cryptodome/Util/../Cipher/_raw_ecb.so: cannot open shared object file:

texadactyl · 2020-08-13T17:18:43Z

What version of pycryptodome are you using?
Are your Python packages up to date?
Is your Ubuntu up-to-date?

Try this:

sudo apt update
sudo apt dist-upgrade
pip3 install -U pycryptodome # pycryptodomex or pycryptodome depending on what you did originally
python3 -m Cryptodome.SelfTest

If that doesn't sort out your issues, consider upgrading Ubuntu to 20.04.latest and repeating the above.

John-Oula · 2020-09-07T18:43:23Z

pycryptodomex==3.9.8
Yes
Im on ubuntu 16.04
the test is OK for python 3.7.3 but when I run my flask app I get the same error again on the server

fukawi2 · 2020-09-11T05:45:49Z

I too am getting this error on 16.04. I'm installing Salt and pycryptodome from http://repo.saltstack.com/py3/ubuntu/16.04/amd64/3001

vagrant@www1:/vagrant$ python3 -m Cryptodome.SelfTest
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3/dist-packages/Cryptodome/SelfTest/__main__.py", line 26, in <module>
    SelfTest.run(stream=sys.stdout, verbosity=1, config= {'slow_tests' : False })
  File "/usr/lib/python3/dist-packages/Cryptodome/SelfTest/__init__.py", line 60, in run
    tests = get_tests(config=config)
  File "/usr/lib/python3/dist-packages/Cryptodome/SelfTest/__init__.py", line 81, in get_tests
    from Cryptodome.SelfTest import Cipher; tests += Cipher.get_tests(config=config)
  File "/usr/lib/python3/dist-packages/Cryptodome/SelfTest/Cipher/__init__.py", line 31, in get_tests
    from Cryptodome.SelfTest.Cipher import test_AES;      tests += test_AES.get_tests(config=config)
  File "/usr/lib/python3/dist-packages/Cryptodome/SelfTest/Cipher/test_AES.py", line 1234, in get_tests
    from Cryptodome.Cipher import AES
  File "/usr/lib/python3/dist-packages/Cryptodome/Cipher/__init__.py", line 3, in <module>
    from Cryptodome.Cipher._mode_ecb import _create_ecb_cipher
  File "/usr/lib/python3/dist-packages/Cryptodome/Cipher/_mode_ecb.py", line 34, in <module>
    raw_ecb_lib = load_pycryptodome_raw_lib("Cryptodome.Cipher._raw_ecb", """
  File "/usr/lib/python3/dist-packages/Cryptodome/Util/_raw_api.py", line 189, in load_pycryptodome_raw_lib
    raise OSError("Cannot load native module '%s'" % name)
OSError: Cannot load native module 'Cryptodome.Cipher._raw_ecb'
vagrant@www1:/vagrant$

EDIT: I've tried installing pycryptodome using pip3 and I get the same result.

texadactyl · 2020-09-12T14:15:55Z

@fukawi2 @John-Oula

Never get packages from anywhere except pypi or the package owner. Okay, you say that had the same result from this repository.
Where is your source code sample? Please confine it to a small program (<20 lines if possible) as nobody wants to see 50,000 lines of user source code.
Self-test execution: python3 -m Crypto.SelfTest

My results running the self-tests are all as expected: Ran 38772 tests in 203.526s

My environment:

----- Ubuntu release -----
Description:	Ubuntu 20.04.1 LTS
Release:	20.04
Codename:	focal
----- OS kernel name/release/version -----
5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020
----- Machine hardware platform -----
x86_64
----- Video information -----
Graphics:  Device-1: Intel 3rd Gen Core processor Graphics driver: i915 v: kernel 
           Display: x11 server: X.Org 1.20.8 driver: modesetting unloaded: fbdev,vesa resolution: 1920x1080~60Hz 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 2500 (IVB GT1) v: 4.2 Mesa 20.0.8 
----- Audio information -----
Audio:     Device-1: Intel 7 Series/C216 Family High Definition Audio driver: snd_hda_intel 
           Sound Server: ALSA v: k5.4.0-42-generic

I highly recommend upgrading to *buntu 20.04 (LTS).

fukawi2 · 2020-09-14T03:37:56Z

Thanks for your reply @texadactyl

I understand your concern with this, but we don't have the resources to manually install every dependency for every tool we use from each languages implementation of a package manager. We have demonstrated that this problem does occur after testing installing manually from PyPi though.
I don't have specific code example since we're just trying to use Salt Stack, which pycryptodome is a dependency of.
As noted, the SelfTest tool fails for us (see above).

And the whole driver for this is trying to upgrade to 20.04, but we're following a dependency chain... Ubuntu 20.04 has deprecated Python 2, but our current version of Salt doesn't support Python 3, so we're trying to upgrade Salt to a version that works with Py3, then we can upgrade to 20.04. Performing major version bumps of Salt and Ubuntu at the same time is a much bigger risk than doing one at a time. Ubuntu 16.04 is still supported until early next year, and we plan to be upgraded before it's EOL'ed and out of support.

EDIT: Ok, I just spun up a clean 16.04 VM and the tests are passing on it: Ran 39559 tests in 92.821s. I will dig into it further.

fukawi2 · 2020-09-14T04:26:28Z

Update: This does actually look like a packaging problem with the package from the SaltStack repo (not sure why the SelfTest still failed after installing using pip3 - a red herring by the look of it).

The package installs to /usr/lib/python3/dist-packages/Cryptodome which doesn't appear to be correct - it should be /usr/lib/python3/dist-packages/Crypto?

~~EDIT: After symlinking /usr/lib/python3/dist-packages/Cryptodome to /usr/lib/python3/dist-packages/Crypto, the tests work now. I will file a bug against SaltStack. Not sure if this helps OP.~~

EDIT 2: No, this still isn't working. I think it's a conflict with the Python 3.8 packages we're installing. Still not specifically a pycrytodome issue, so I'll stop updating this but just wanted to note that the above is not a valid workaround by the look of it.

fukawi2 · 2020-09-14T07:49:38Z

OK, final update....

After much rabbit hole exploring this was caused by mixed Python versions in my environment.

The Salt packages and dependencies (specifically, python3-pycryptodome) for Ubuntu 16.04 are compiled for Python 3.5. We upgraded our 16.04 hosts to Python 3.8, and this broke loading of dynamic modules relying on the EXTENSION_SUFFIXES constant (which pycryptodome does)

Specifically, this breaks pycryptodome when it tries to load the _raw_ecb module which is compiled to _raw_ecb.cpython-35m-x86_64-linux-gnu.so - note the "35m" denoting it's targeting Python 3.5. However with Python 3.8, it expects to find it named _raw_ecb.cpython-38-x86_64-linux-gnu.so (note the "38" for Python 3.8).

Python 3.5 (correct EXTENSION_SUFFIXES)

>>> from importlib import machinery
>>> print(machinery.EXTENSION_SUFFIXES)
['.cpython-35m-x86_64-linux-gnu.so', '.abi3.so', '.so']
>>>

Python 3.8 (incorrect EXTENSION_SUFFIXES)

>>> from importlib import machinery
>>> print(machinery.EXTENSION_SUFFIXES)
['.cpython-38-x86_64-linux-gnu.so', '.abi3.so', '.so']
>>>

So... Check your Python versions!

Varbin · 2020-09-14T08:39:13Z

@fukawi2 @Legrandin Would it be possible to compile Pycryptodome against the stable ABI (PEP-3149 and PEP-384)? Extensions compiled that way would work for multiple Python versions.

texadactyl · 2020-09-14T13:02:18Z

@fukawi2
Glad you are able to uncover the true cause of discomfort.

Similar things have happened to me in the past. It is sometimes a painful waste of time but diagnosis is difficult with tools that are unable to flag errors at a sufficient level to zero in on the actual problem (a lot of guess-work by developers too - myself included). So, this was a good lesson for me too.

fukawi2 · 2020-09-14T23:47:29Z

Would it be possible to compile Pycryptodome against the stable ABI

I'm just a Salt user, and I'm not familiar enough with Python to fully understand that. I'm guessing it could be possible for the Salt team to do that?

...a lot of guess-work by developers too - myself included...

Sorry for all the noise posting updates; hopefully it helps others in the future though. I spent many hours trying to figure this out, and the current Google results weren't helpful in my case.

John-Oula · 2020-09-15T04:00:45Z

So you are saying a mix up of python versions might be the cause?
Because i have many versions on my ubuntu 16.04

John-Oula · 2020-09-15T04:04:51Z

OK, final update....

After much rabbit hole exploring this was caused by mixed Python versions in my environment.

The Salt packages and dependencies (specifically, python3-pycryptodome) for Ubuntu 16.04 are compiled for Python 3.5. We upgraded our 16.04 hosts to Python 3.8, and this broke loading of dynamic modules relying on the EXTENSION_SUFFIXES constant (which pycryptodome does)

Specifically, this breaks pycryptodome when it tries to load the _raw_ecb module which is compiled to _raw_ecb.cpython-35m-x86_64-linux-gnu.so - note the "35m" denoting it's targeting Python 3.5. However with Python 3.8, it expects to find it named _raw_ecb.cpython-38-x86_64-linux-gnu.so (note the "38" for Python 3.8).

Python 3.5 (correct EXTENSION_SUFFIXES)
>>> from importlib import machinery
>>> print(machinery.EXTENSION_SUFFIXES)
['.cpython-35m-x86_64-linux-gnu.so', '.abi3.so', '.so']
>>>
Python 3.8 (incorrect EXTENSION_SUFFIXES)
>>> from importlib import machinery
>>> print(machinery.EXTENSION_SUFFIXES)
['.cpython-38-x86_64-linux-gnu.so', '.abi3.so', '.so']
>>>
So... Check your Python versions!

Thats my output

from importlib import machinery
print(machinery.EXTENSION_SUFFIXES)
['.cpython-37m-x86_64-linux-gnu.so', '.abi3.so', '.so']

fukawi2 · 2020-09-15T04:20:08Z

Do you have _raw_ecb.cpython-37m-x86_64-linux-gnu.so in /usr/lib/python3? If not, then I'd be guessing you're having the same problem I was.

John-Oula · 2020-09-15T04:55:00Z

No i dont

John-Oula · 2020-09-15T04:59:16Z

Do you have _raw_ecb.cpython-37m-x86_64-linux-gnu.so in /usr/lib/python3? If not, then I'd be guessing you're having the same problem I was.

No i dont.How can i fix it?

fukawi2 · 2020-09-15T05:03:59Z

No i dont

You seem to be running under the wrong version then. We don't know enough about your environment to be able to tell you how to fix it. You said you have "many versions" of Python on your machine, so you're going to need to try and sort that out.

John-Oula · 2020-09-15T05:10:32Z

No i dont

You seem to be running under the wrong version then. We don't know enough about your environment to be able to tell you how to fix it. You said you have "many versions" of Python on your machine, so you're going to need to try and sort that out.

Well, im running under python3.7

Varbin · 2020-09-16T18:40:45Z

Do you have _raw_ecb.cpython-37m-x86_64-linux-gnu.so in /usr/lib/python3? If not, then I'd be guessing you're having the same problem I was.

No i dont.How can i fix it?

@John-Oula Install pycryptodome for the Python version you run your flask app with.

E.g. /path/to/python3.7 -m pip install --user pycryptodome (do NOT use sudo, --user might not be required)
You may need to install the Python headers and a compiler (Ubuntu packages build-essential and python3.7-dev).

John-Oula · 2020-09-20T08:34:36Z

Turns out it was my own mistake.I was importing from Cryptodome.Cipher import AES instead of Crypto.Cipher

John-Oula closed this as completed Sep 20, 2020

Varbin mentioned this issue Oct 20, 2020

Enhancement request: abi3 wheels #458

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I Keep getting OSError error: _raw_ecb.x86_64-linux-gnu.so: cannot open shared object file: #440

I Keep getting OSError error: _raw_ecb.x86_64-linux-gnu.so: cannot open shared object file: #440

John-Oula commented Aug 13, 2020

texadactyl commented Aug 13, 2020

John-Oula commented Sep 7, 2020

fukawi2 commented Sep 11, 2020 •

edited

texadactyl commented Sep 12, 2020 •

edited

fukawi2 commented Sep 14, 2020 •

edited

fukawi2 commented Sep 14, 2020 •

edited

fukawi2 commented Sep 14, 2020 •

edited

Varbin commented Sep 14, 2020

texadactyl commented Sep 14, 2020 •

edited

fukawi2 commented Sep 14, 2020

John-Oula commented Sep 15, 2020

John-Oula commented Sep 15, 2020

Python 3.5 (correct EXTENSION_SUFFIXES)

Python 3.8 (incorrect EXTENSION_SUFFIXES)

fukawi2 commented Sep 15, 2020

John-Oula commented Sep 15, 2020

John-Oula commented Sep 15, 2020

fukawi2 commented Sep 15, 2020

John-Oula commented Sep 15, 2020

Varbin commented Sep 16, 2020

John-Oula commented Sep 20, 2020

I Keep getting OSError error: _raw_ecb.x86_64-linux-gnu.so: cannot open shared object file: #440

I Keep getting OSError error: _raw_ecb.x86_64-linux-gnu.so: cannot open shared object file: #440

Comments

John-Oula commented Aug 13, 2020

texadactyl commented Aug 13, 2020

John-Oula commented Sep 7, 2020

fukawi2 commented Sep 11, 2020 • edited

texadactyl commented Sep 12, 2020 • edited

fukawi2 commented Sep 14, 2020 • edited

fukawi2 commented Sep 14, 2020 • edited

fukawi2 commented Sep 14, 2020 • edited

Python 3.5 (correct EXTENSION_SUFFIXES)

Python 3.8 (incorrect EXTENSION_SUFFIXES)

Varbin commented Sep 14, 2020

texadactyl commented Sep 14, 2020 • edited

fukawi2 commented Sep 14, 2020

John-Oula commented Sep 15, 2020

John-Oula commented Sep 15, 2020

Python 3.5 (correct EXTENSION_SUFFIXES)

Python 3.8 (incorrect EXTENSION_SUFFIXES)

fukawi2 commented Sep 15, 2020

John-Oula commented Sep 15, 2020

John-Oula commented Sep 15, 2020

fukawi2 commented Sep 15, 2020

John-Oula commented Sep 15, 2020

Varbin commented Sep 16, 2020

John-Oula commented Sep 20, 2020

fukawi2 commented Sep 11, 2020 •

edited

texadactyl commented Sep 12, 2020 •

edited

fukawi2 commented Sep 14, 2020 •

edited

fukawi2 commented Sep 14, 2020 •

edited

fukawi2 commented Sep 14, 2020 •

edited

texadactyl commented Sep 14, 2020 •

edited