pycryptodome doesn't work with cx_Freeze or py2exe

[sekrause]

I was looking for a PyCrypto replacement to use in a project which is deployed on Windows with cx_Freeze. At first pycryptodome looked like the perfect candidate, but the very dynamic loading of the binary modules through load_pycryptodome_raw_lib() and especially pycryptodome_filename() makes it pretty much impossible to use pycryptodome with either cx_Freeze or py2exe:

• Because of the dynamic loading through load_pycryptodome_raw_lib() cx_Freeze doesn't even find the modules to include in the binary distribution, so they're just missing.
• But even if the C-extensions were available at runtime, pycryptodome_filename() wouldn't be able to load them because __file__ doesn't work with cx_Freeze where all modules are compiled and in a ZIP file.

pycryptodome looks like a great library, but it would be much more useful if it just used normal import statements for the C-extensions.

[twbradio]

Thank you for the clues to let me fix this in my app. Please be advised, my code is probably a little clunky, but it is provided with the best of intentions. I changed _file_system to check to see if the code was frozen and then had it rewrite the pyd paths if it was:

    if dir_comps[0] != "Crypto":
        raise ValueError("Only available for modules under 'Crypto'")
    dir_comps = list(dir_comps[1:]) + [filename]

    check_frozen = None
    try:
        check_frozen = getattr(sys, 'frozen')
    except:
        pass

    if check_frozen != None:
        crypto_path = os.path.join(os.path.dirname(sys.executable), 'Crypto')
        return_value =  os.path.join(crypto_path, *dir_comps)
    else:
        util_lib, _ = os.path.split(os.path.abspath(__file__))
        root_lib = os.path.join(util_lib, "..")
        return_value = os.path.join(root_lib, *dir_comps)
    return return_value

I then added the following to setup.py to automatically copy the files into the distribution (using py2exe):

import Crypto
import sys, os

Mydata_files = list()
path = Crypto.__path__[0]
root_end = path.find('Crypto')
for folder,folder_name,files in os.walk(path):
    for file in files:
        if os.path.splitext(file)[1] == '.pyd':
            Mydata_files.append((folder[root_end:], [os.path.join(folder,file)]))

setup(
    data_files = Mydata_files,

[konradkonrad]

@twbradio huh! I ran into the same issue -- do you still use your solution? Could you maybe add a little more meat to it? Which pyd paths do you adjust?

[jdodds]

This is currently causing pain for me as well, moving an application that used PyCrypto + py2exe's zip+folder style loading on 3.4 up to 3.6. I'd love to see pycryptodome "playing nice" here; the alternative in my case is probably maintaining patches on a dead library and hoping it continues to work into the future.

Is pycryptodome doing things that can't be handled via normal import mechanisms? I'm willing to put in work solving this issue, but don't want to run headfirst into a dead-end or end up maintaining an awkward fork that solves my problems but doesn't work for anyone else :/

[TheOtherGuy]

Hi there, I have encountered issues using Pyinstaller with a module that uses Pycryptodome too.

This is the errror : OSError: Cannot load native module 'Crypto.Cipher._raw_ecb' . I note a similar error involving a different encryption method in #26

[TheOtherGuy]

Not sure if this helps anyone in troubleshooting, but for Pyinstaller, a hook was created to enable pycryptodomex to work successfully. Note that pycryptodome and pycryptodomex is different.

[Legrandin]

A PyInstaller hook for the pycryptodome package is also available (pyinstaller/pyinstaller#3424).

[Prev-I]

I've encountered the same problem using esky and py2exe. The generated exe is unable to load pycryptodome and throw this error:

File "zipextimporter.pyc", line 82, in load_module
File "Crypto\Cipher\_mode_ecb.pyc", line 46, in <module>
File "Crypto\Util\_raw_api.pyc", line 191, in load_pycryptodome_raw_lib
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Trying '_raw_ecb.pyd': [Error 126] 

I've tryed to replicate the solution proposed by @twbradio but I'm not sure where the modified code must be placed to work

[kdschlosser]

I have a solution for this issue when running under windows and using cx_Freeze I am fairly sure this should also be compatible with other opperating systems as well. you would have to give it a try to see if it works.

This requires no modification of any files in cx_Freeze or pycryptodome. I monkey patched the mechanism that loads the modules to be compatible with the changing of the locations and filenames (this is what cx_Freeze does I would imagine this code could be added easily to pycryptodome. having pycryptodome check for a frozen state and if so then to give this is a shot to see if it will work.

In the meantime you should be able to put this code anywhere in your code. you want to make sure that it gets run before any other import of Crypto takes place. this will ensure that there is no population of the Crypto.Util._raw_api into any of the modules that use the load_pycryptodome_raw_lib function

i personally recommend putting the code at the very top of the script that was passed to cx_Freeze.Executable this will ensure that it happens first.

Be sure to add Crypto to the list of packages passed to cx_Freeze.setup it has to be in the packages not in includes. the build will fail otherwise

import sys

if hasattr(sys, 'frozen'):
    import os
    from ctypes import CDLL
    from Crypto.Util import _raw_api


    def load_pycryptodome_raw_lib(name, _):
        for ext in _raw_api.extension_suffixes:
            mod_file_name = name + ext
            for path in sys.path:
                if path.endswith('.zip'):
                    continue
                mod_file_path = os.path.join(path, mod_file_name)
                if os.path.exists(mod_file_path):
                    return CDLL(mod_file_path)

        raise OSError("Cannot load native module '%s'" % name)


    _raw_api.load_pycryptodome_raw_lib = load_pycryptodome_raw_lib
    sys.modules['Crypto.Util._raw_api'] = _raw_api

    # Test
    from Crypto.Cipher import AES

** Edited - Checks for frozen state. also iterates over all sys.path entries this should allow for it's use in py2exe as well as cx_Freeze. I am not sure about any others. So long as the filename is the same as the 2 mentioned above it should work.

example filename would be

Crypto.Cipher._raw_aes.pyd

and the file location should be in the root of the built library folder. NOT in the zipfile. if it is in the zipfile then you will need to add pycryptodome to the zipexcludes

[Legrandin]

The recommendation is to use the more maintained PyInstaller project.

[kdschlosser]

well that fine. if you are writing a new packaging routine. if you have a project that already has a packaged and you do not want to have to write that portion of it over again but want to use a more up to day crypto library. then the above should work.

FYI: PyInstaller does not have pycryptodome on it's supported modules list.
https://github.com/pyinstaller/pyinstaller/wiki/Supported-Packages

and on their repo. pyinstaller/pyinstaller#3559. and it looks as tho the issue is still open and has not been fixed.

[Legrandin]

Support for both pycryptodome and pycryptodomex was added to PyInstaller v3.4, released in September 2018. That bug you refer to is older, so it should be probably closed by now.

[kdschlosser]

maybe make mention to them about closing that issue along with adding it to the list of supported modules. Dunno if this is of importance to you or not. I also saw that pyinstaller uses the pycrypto library. maybe then should switch to this repo because of pycrypto being unmaintained.

The code is there to allow this library to be packaged properly using py2exe and cxFreeze if the user already has a packaging module in place if they do not want to change what they have. I have tested it with cxFreeze i can't remember if i tested it with py2exe or not. I would imagine that since it did solve the issue for cxFreexe it would be a solution across the board for all packaging modules.