FileNotFoundError: [Errno 2] No such file or directory: '/tmp/_MEIRsuH58/jnpr/junos/facts'

[akbhat]

PyInstaller==3.2.1, Ubuntu 14 (Python 2.6, 2.7 and 3.5 installed)

I had a Python2.7 script that was frozen using Pyisntaller and working as expected. I wanted to move to Python3.5 so converted the script to Python3, removed and reinstalled Pyinstaller.

In the new environment, I can freeze the script (pyinstaller jcollect.spec --clean --onefile --log=DEBUG - debug.txt) but get the following error running the executable.
Traceback (most recent call last):
File "jcollect.py", line 5, in
from jutil import *
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 664, in _load_unlocked
File "", line 634, in _load_backward_compatible
File "/usr/local/lib/python3.5/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
exec(bytecode, module.dict)
File "jutil.py", line 8, in
from jnpr.junos.device import Device
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 664, in _load_unlocked
File "", line 634, in _load_backward_compatible
File "/usr/local/lib/python3.5/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
exec(bytecode, module.dict)
File "site-packages/jnpr/junos/init.py", line 1, in
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 664, in _load_unlocked
File "", line 634, in _load_backward_compatible
File "/usr/local/lib/python3.5/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
exec(bytecode, module.dict)
File "site-packages/jnpr/junos/device.py", line 28, in
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 664, in _load_unlocked
File "", line 634, in _load_backward_compatible
File "/usr/local/lib/python3.5/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
exec(bytecode, module.dict)
File "site-packages/jnpr/junos/factcache.py", line 4, in
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 664, in _load_unlocked
File "", line 634, in _load_backward_compatible
File "/usr/local/lib/python3.5/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
exec(bytecode, module.dict)
File "site-packages/jnpr/junos/facts/init.py", line 108, in
File "site-packages/jnpr/junos/facts/init.py", line 91, in _build_fact_callbacks_and_doc_strings
File "site-packages/jnpr/junos/facts/init.py", line 69, in _import_fact_modules
File "site-packages/jnpr/junos/facts/init.py", line 52, in _get_list_of_fact_module_names
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/_MEIOLklta/jnpr/junos/facts'
Failed to execute script jcollect

Using the following Analysis function in the spec file for compilation:

from PyInstaller.utils.hooks import collect_data_files
a = Analysis(['jcollect.py'],
pathex=['/home/abhat/Desktop/jcollect'],
binaries=[],
datas= collect_data_files('jnpr.junos'),
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)

[ghost]

What's happening is that the jnpr package is using the filesystem rather than the import machinery to attempt to import this package. You will need to monkeypatch this function with something like this (only an outline; may require further revision):

def _get_list_of_fact_module_names():
    """
    Get a list of fact module names.
    Gets a list of the module names that reside in the facts directory (the
    directory where this jnpr.junos.facts.__init__.py file lives). Any module
    names that begin with an underscore (_) are ommitted.
    :returns:
      A list of fact module names.
    """
    from importlib.util import find_spec
    import os
    loader = find_spec('jnpr.junos.facts').loader

    module_names = []
    facts_dir = 'jnpr/junos/facts'
    for path in loader.toc:
        if path.startswith(facts_dir):
            file = os.path.basename(path)
            if not file.startswith('_'):
                (module_name, _) = file.rsplit('.pyc', 1)
                module_names.append('%s.%s' % (__name__, module_name))
    return module_names

Not sure exactly what the toc entries look like, so I may have got that wrong. Comments, @htgoebel?

[akbhat]

@xoviat Thanks for the suggestion but the altered code broke the pure python version of the script so I presume needs some more tweaking. Over to @htgoebel

[ghost]

@akbhat This code is only for the frozen version of the script. So you would do something like:

if hasattr(sys, 'frozen'):
    jnpr.junos.facts._get_list_of_fact_module_names = _get_list_of_fact_module_names

If you want to debug this yourself, you can just print the loader.toc inside the frozen version and then fixup the patch.

[ghost]

Also, @htgoebel what do you think about patching os.listdir, os.isfile, and os.islink, and changing runtime behavior for paths within site packages?

[akbhat]

@xoviat @htgoebel I didn't quite get this to work. Can we get a proper fix checked in?

[htgoebel]

@xoviat I don't think its a good idea adding such magic to PyInstaller. Its against the Zen of Python, esp. "Explicit is better than implicit", "Special cases aren't special enough to break the rules" and "In the face of ambiguity, refuse the temptation to guess".

If we start patching os.listdir, os.isfile, and os.islink, why not os.stat, open() and file()? Wouldn't we need to patch the whole pythonlib to do it right? And then one comes with a C-extension which will fail. So where would this end if we want to make it correctly?

@akbhat

1. Here is a corrected version of xoviat's code (untested). But this into you main-script:

def _get_list_of_fact_module_names():
    from importlib.util import find_spec
    import os
    loader = find_spec('jnpr.junos.facts').loader

    module_names = []
    facts_dir = 'jnpr.junos.facts.'    # dots and trailing dot
    for path in loader.toc:              # HACK undocumented feature
        if path.startswith(facts_dir):
            module_name = path[len(facts_dir):]
            if not module_name.startswith('_'):
                module_names.append('jnpr.junos.facts.' + module_name)
    return module_names

Please keep in mind that this uses an undocumented feature which might go away in future versions of PyInstaller. If you still can' get this to work, please contact my via mail.

2. Please ask the vendor of that package to fix this issue in their code. Proper solutions would be
   1. name all sub-modules in jnpr/junos/facts/__init__.py (this is the easiest one)
   2. when running setup.py write existing sub-module's names into a text file and read it using pkg_resources (flexible, but most complicated to implement), or
   3. use the pkg_resources' plugin-in mechanism.

[akbhat]

@htgoebel - What email can I reach you at? FYI - I tried the above code but that did not seem to work. Let me know what you need from me to debug this further. BTW I also tried building after adding the following to the __init__.py in jnpr folder but that did not work either.

…

__all__ = ["junos/facts", "junos/cfgro", "junos/utils", "junos/ofacts", "junos/transport", "junos/op", "junos/factory", "junos/cfg", "junos/cfg/phyport", "junos/resources"]

[ghost]

His email should be on the contact page of crazy-compilers.com. I'm hesitant to write it here for fear that bots will pick it up.