How can I pyinstaller my script with pyface?

[B-C-WANG]

I want to package my script with mayavi, everything runs well on python, but after using pyinstaller to package my script, I got "No pyface.toolkits plugin found for toolkit wx". More realted information:
https://stackoverflow.com/questions/50337382/creating-standalone-exe-using-pyinstaller-with-mayavi-import
https://stackoverflow.com/questions/51236026/how-to-use-pyinstaller-or-cx-freeze-to-package-a-python-script-containing-traits

[Zulex]

Any fix for this?

[corranwebster]

No - we don't deploy apps at Enthought using PyInstaller or similar tools, so this hasn't been a priority. The problem that you are experiencing is around the use of entry points to set up the toolkit. You might be able to work around this by replacing the entire pyface/toolkit.py module in your checkout of pyface with something like:

from traits.etsconfig.api import ETSConfig

ETSConfig.toolkit = 'qt4'

from pyface.ui.qt4.init import toolkit_object

toolkit = toolkit_object

Adjust appropriately is you are using wxPython

This hard-codes the toolkit import, rather than delegating it to configuration.

You'll need to do something like this in TraitsUI as well, most likely.

[Huzaifg]

So I edited the toolkit.py file as mentioned above, I am now getting this error
Traceback (most recent call last): File "tkinter/__init__.py", line 1883, in __call__ File "arai.py", line 32, in plot from mayavi import mlab File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "/home/huzaifa/env/lib/python3.8/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module exec(bytecode, module.__dict__) File "mayavi/mlab.py", line 15, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "/home/huzaifa/env/lib/python3.8/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module exec(bytecode, module.__dict__) File "mayavi/core/common.py", line 21, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "/home/huzaifa/env/lib/python3.8/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module exec(bytecode, module.__dict__) File "pyface/api.py", line 16, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "/home/huzaifa/env/lib/python3.8/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module exec(bytecode, module.__dict__) File "pyface/clipboard.py", line 26, in <module> File "pyface/base_toolkit.py", line 184, in __init__ NotImplementedError: the qt4 pyface.ui.qt4 backend doesn't implement clipboard:Clipboard
I am trying to build an executable using pyinstaller which uses mlab

[corranwebster]

You have a tkinter import at the top of the traceback, which seems very wrong. Pyface only supports PyQt, PySide or WxPython as GUI libraries, and these are not generally compatible with Tk/Tkinter.

[corranwebster]

Additionally, I think you might need to tell PyInstaller to bundle everything in pyface.ui.qt4.* even if it is not directly imported (and possibly some other sub-packages too). It may be easier just to tell Pyinstaller to bundle all of pyface.

[Huzaifg]

@corranwebster Yes, the executable is a GUI made on tkinter that has a visualize button which opens up a mayavi window. It works fine when I just run the main script. This problem only pops up when I run the executable.
How do I ask Pyinstaller to bundle all of pyface? This is my .spec file

-- mode: python ; coding: utf-8 --
import os
import importlib

block_cipher = None

a = Analysis(['arai.py'],
pathex=['/home/huzaifa/Simulations/DATA'],
binaries=[],
datas=[(os.path.join(os.path.dirname(importlib.import_module('tensorflow').file),
"lite/experimental/microfrontend/python/ops/_audio_microfrontend_op.so"),
"tensorflow/lite/experimental/microfrontend/python/ops/")],
hiddenimports=['PIL._tkinter_finder', 'tensorflow.compiler.tf2tensorrt', 
'tensorflow.compiler.tf2tensorrt.ops', 'tensorflow.compiler.tf2tensorrt.ops.gen_trt_ops',
'tensorflow.python.keras.engine.base_layer_v1',
'tensorflow.python.ops.numpy_ops','mayavi'
,'traitsui.qt4','pyface.ui.qt4','traitsui.ui_traits','pkg_resources.py2_warn',
'pkg_resources.markers','tornado'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
[],
exclude_binaries=True,
name='arai',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='arai')

Do I add it to hidden imports?

[wlievens]

Has anyone been able to get this to work? I've been trying for days to no avail, and I'd really like to not have to get rid of traitsui...

[rahulporuri]

@wlievens what is the failure mode you see when you try to use pyinstaller?

[wlievens]

Thanks for the reply.

When I use pyface "out of the box" as a dependency, and I run the generated executable I get the following runtime exception:

...
  File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
  File "traitsui\qt4\toolkit.py", line 24, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
  File "pyface\toolkit.py", line 23, in <module>
  File "pyface\base_toolkit.py", line 285, in find_toolkit
KeyError: 'pyface.toolkits'
[20796] Failed to execute script __main__

I then applied the suggestion above at #350 (comment) and replaced the toolkit.py code with the more "build time" import. When I build the new executable and run it, I get the following error:

...
  File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
  File "traitsui\qt4\toolkit.py", line 29, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
  File "pyface\api.py", line 17, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
  File "pyface\clipboard.py", line 26, in <module>
  File "pyface\base_toolkit.py", line 190, in __init__
NotImplementedError: the qt4 pyface.ui.qt4 backend doesn't implement clipboard:Clipboard
[1168] Failed to execute script __main__

I have tried further hacking the various toolkit files and their imports but never got anywhere...

I really like traitsui so I'd love to keep using it, but the exe build is also a hard requirement.

[wlievens]

Does anyone have any ideas on how to proceed?

[rahulporuri]

@wlievens does the discussion in enthought/traitsui#458 help?

[wlievens]

That discussion is a big mess, mixing tips for pyinstaller and cx_freeze, but I tried a few suggestions from there, i.e. setting the env vars and forcing the pyface.api import early in my main file.

So I restarted my efforts and undid the changes suggested here:
#350 (comment)

And then in my own main file I started off with this:

enthought/traitsui#458 (comment)

I then got this error:

  File "pyface\base_toolkit.py", line 248, in import_toolkit
RuntimeError: No pyface.toolkits plugin could be loaded for qt4

So I went to look in that file and added some debug prints (I don't know how to use the built-in logger). In the import_toolkit function I printed the plugins list and the raised exception that causes it to not load a toolkit.

PLUGINS [EntryPoint(name='qt4', value='pyface.ui.qt4.init:toolkit_object', group='pyface.toolkits')]
ERROR No module named 'pyface.ui.qt4.init'

Now of course I do have pyface.ui.qt4 in my hiddenimports but I did not have pyface.ui.qt4.init so I added it to hiddenimports and that seems to bring me further to the next error.

Since the tip from #3050 contained turning on ETS_DEBUG, I now get this error:

  File "pyface\base_toolkit.py", line 170, in __call__
AttributeError: module 'sys' has no attribute 'exc_traceback'

here:
https://github.com/enthought/pyface/blob/master/pyface/base_toolkit.py#L170

Figured I'd report that here, but I can turn off ETS_DEBUG for now of course.

I ran again with debug off and now see this familiar error again:

  File "pyface\base_toolkit.py", line 190, in __init__
NotImplementedError: the qt4 pyface.ui.qt4 backend doesn't implement clipboard:Clipboard

So I'm stuck again I suppose.

[rahulporuri]

@wlievens im assuming you're working with pyface 7.3.0 (the latest release). If I understand you correctly, the exception is being raised because the return in this else statement doesn't happen -

pyface/pyface/base_toolkit.py

Lines 174 to 177 in ded5d29

	else:
	obj = getattr(module, oname, None)
	if obj is not None:
	return obj

i.e. the getattr(module, oname, None) returns None. I can't imagine why that would be happening.

Also, are you doing the following -

For PyInstaller, I had to add both the library and egg-info folders for both Pyface and Traitsui to datas

To give you a better understanding of the internals, pyface is an abstraction layer and multiple toolkit-specific implementations sit behind the abstraction i.e. the qt implementation and the wx implementation. For this reason, pyface dynamically chooses which toolkit to use depending on the packages installed in the environment or depending on the existence of specific environment variables. That determination happens by looking at the package metadata - which if im not wrong is stored in the .egg-info folders of pyface and traitsui.

Without those .egg-info folders, I dont think pyface and traitsui would work because they wouldn't know what toolkits (e.g. qt) are available and where they are available.

[wlievens]

Hi, thanks for the additional info. I did indeed see that and tried something but I'm not sure I did it right. I don't actually see the .egg-info folders but I do see .dist-info directories in my venv (e.g. @pyface-7.3.0.dist-info@) and I added those to pyinstaller's spec file.

    datas=[
        ('%s/pyface-7.3.0.dist-info' % packages_path, 'pyface-7.3.0.dist-info'),
        ('%s/traitsui-7.1.1.dist-info' % packages_path, 'traitsui-7.1.1.dist-info'),
    ],

packages_path points to the venv's site-packages dir.

[rahulporuri]

@wlievens .dist-info? Can you give us information on what platform, python version and packages you are working with? I'm not a 100% certain that the .dist-info is the same as .egg-info.

[wlievens]

The dist-info dirs contain the entry_points.txt file among others

> ls
direct_url.json          
entry_points.txt         
INSTALLER                
LICENSE-CC-BY-3.0.txt    
LICENSE.txt              
METADATA                 
RECORD                   
REQUESTED                
top_level.txt            
WHEEL

I'm on Windows 10. I use poetry to manage dependencies and virtual environment. Python is 3.8.6 (64 bit). The pyinstaller version is 4.2, pyface is 7.3.0 and traitsui is 7.1.1.

[rahulporuri]

Can you provide the script or spec file that you are using to setup/use pyinstaller? I personally don't have a lot of experience using pyinstaller but I can try to spend sometime this weekend to see what's going wrong. I can't promise a solution though.

[wlievens]

Thanks for looking into it. I'm continuing the investigation myself, too: I just added `pyface.ui.qt4.clipboard``` to hiddenimports and that seems to move the error to the next problem ...
Earlier I manually added all directories to the hiddenimports list, but maybe i need to crawl the pyface/ui/qt4 path and just automatically add every single file?

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None

packages_path = r'venv\gp-cobra-distribution-template-x9nC8UKb-py3.8\Lib\site-packages'

a = Analysis(
    ['gp_cobra_distribution_template\\__main__.py'],
    pathex=[packages_path],
    binaries=[
        ('%s/gp_wrapper_fx3/gp_native_fx3.dll' % packages_path, '.'),
        ('%s/numpy/.libs/libopenblas.QVLO2T66WEPI7JZ63PS3HMOHFEY472BC.gfortran-win_amd64.dll' % packages_path, '.')
    ],
    datas=[
        ('%s/pyface-7.3.0.dist-info' % packages_path, 'pyface-7.3.0.dist-info'),
        ('%s/traitsui-7.1.1.dist-info' % packages_path, 'traitsui-7.1.1.dist-info'),
    ],
    hiddenimports=[
        'importlib_metadata',
        'importlib_resources',
        'numpy',
        'pyface',
        'pyface.toolkit',
        'pyface.ui.qt4',
        'pyface.ui.qt4.action',
        'pyface.ui.qt4.clipboard',
        'pyface.ui.qt4.code_editor',
        'pyface.ui.qt4.console',
        'pyface.ui.qt4.data_view',
        'pyface.ui.qt4.fields',
        'pyface.ui.qt4.images',
        'pyface.ui.qt4.init',
        'pyface.ui.qt4.tasks',
        'pyface.ui.qt4.tests',
        'pyface.ui.qt4.timer',
        'pyface.ui.qt4.util',
        'pyface.ui.qt4.wizard',
        'pyface.ui.qt4.workbench',
        'pywin32_system32',
        'pywintypes',
        'scipy',
        'traitsui',
        'traitsui.qt4',
        'traitsui.qt4.extra',
        'traitsui.qt4.toolkit',
        'traitsui.toolkit',
        'traitsui.ui_traits',
    ],
    hookspath=['hooks'],
    runtime_hooks=[],
    excludes=['matplotlib', 'networkx'],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False
)
pyz = PYZ(
    a.pure,
    a.zipped_data,
    cipher=block_cipher
)
exe = EXE(
    pyz,
    a.scripts,
    [],
    exclude_binaries=True,
    name='main',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    console=True
)
coll = COLLECT(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    strip=False,
    upx=True,
    upx_exclude=[],
    name='main'
)

[wlievens]

I wrote this at the start of my .spec file

packages_path = r'venv\gp-cobra-distribution-template-x9nC8UKb-py3.8\Lib\site-packages'

auto_imports = []
crawl_path = os.path.abspath(os.path.join(packages_path, 'pyface', 'ui', 'qt4'))
for file in os.listdir(crawl_path):
    file_path = os.path.join(crawl_path, file)
    if file not in {'__pycache__', '__init__.py'} and (
            os.path.isdir(file_path) or (os.path.isfile(file_path) and file_path.endswith('.py'))):
        auto_imports.append('pyface.ui.qt4.%s' % file.replace('.py', ''))

And now I no longer get pyface-related errors! So I think it's resolved by doing that. I now get similar errors about dynamically-loaded qt5 backends from a different dependency (vispy) :-D But maybe I can solve it the same way...

[rahulporuri]

And now I no longer get pyface-related errors!

@corranwebster gave me a brief introduction to how packages like pyinstaller work i.e. they include modules in the installer only if they are imported explicitly. In the case of pyface AND traitsui, the packages dynamically import from the qt or wx backends depending on which toolkit is installed. So, for pyinstaller (or other similar tools) to work correctly, you will need to explicitly specify the pyface/traitsui submodules for the toolkit you use i.e. pyface.ui.qt4 or traitsui.qt4.

@wlievens thanks for getting back to us with what worked. We'll try to consolidate this information and document it for future users.

[wlievens]

I tried specifying pyface.ui.qt4 and traitsui.qt4 but that was not enough. I had to specify every single python file in pyface's qt4 directory, for it to work.

[rahulporuri]

I had to specify every single python file in pyface's qt4 directory, for it to work.

Yeah, that sounds about right. pyface.ui.qt4.* imports heavily from pyface.qt because pyface.qt is the abstraction layer between PyQt4, PyQt5 and PySide2 (and in the future PySide6).

Instead of adding modules from pyface that the pyinstaller needs, I would recommend including all of pyface and then slowly removing modules that you think are irrelevant to see what happens. I don't know if that would make the process easier or harder. Again, I don't have much experience actually using pyinstaller.

[wlievens]

I'm in the middle of a big fight with pyinstaller now over some unrelated issue (it cannot find some standard python modules when running as single executable ...), but if I get that resolved I will try to narrow down the acceptable configuration for pyface, and I'll report back here :-)

[TianSong1991]

I'm in the middle of a big fight with pyinstaller now over some unrelated issue (it cannot find some standard python modules when running as single executable ...), but if I get that resolved I will try to narrow down the acceptable configuration for pyface, and I'll report back here :-)

@wlievens Hi can you solve mayavi to use pyinstaller? If you solve ,please tell me the way to solve thanks.

[wlievens]

What I ended up doing I think to get traitsui & pyface working is this in my main.spec:

hiddenimports = []

def collect_imports(path, prefix):
    hiddenimports.append(prefix)
    for file in os.listdir(path):
        if file in {'__pycache__', '__init__.py', 'tests'}:
            continue
        child = f'{prefix}.{file}'
        file_path = os.path.join(path, file)
        if os.path.isdir(file_path):
            collect_imports(file_path, child)
        elif os.path.isfile(file_path) and file.endswith('.py'):
            hiddenimports.append(child[0:-len('.py')])

collect_imports(os.path.join(packages_path, 'pyface', 'ui', 'qt4'), 'pyface.ui.qt4')
collect_imports(os.path.join(packages_path, 'traitsui', 'qt4'), 'traitsui.qt4')

It essentially just lists all the packages and all the subpackages. Somehow this helped, I think.

[rahulporuri]

Somehow this helped, I think.

I mentioned the following in an earlier comment.

Instead of adding modules from pyface that the pyinstaller needs, I would recommend including all of pyface and then slowly removing modules that you think are irrelevant to see what happens.

Hi can you solve mayavi to use pyinstaller?

I haven't tested this but here again, I presume the solution will be to explicitly add all of tvtk and mayavi because I think those packages also contain similar abstraction layers which use qt or wx depending on the installed toolkit library.

[wlievens]

Instead of adding modules from pyface that the pyinstaller needs, I would recommend including all of pyface and then slowly removing modules that you think are irrelevant to see what happens.

Your comment is likely the reason why I came up with that solution, I just don't recall the specifics since it's ~7 months ago.

[TianSong1991]

Instead of adding modules from pyface that the pyinstaller needs, I would recommend including all of pyface and then slowly removing modules that you think are irrelevant to see what happens.

Your comment is likely the reason why I came up with that solution, I just don't recall the specifics since it's ~7 months ago.

@wlievens Thanks for you reply, I will try your advise.

[corranwebster]

I spent a few hours today and think I have a simple way that seems to work with PyInstaller 5 (may work with earlier versions, but that's what I have installed). The issues that people have are threefold:

• pyface does a lot of dynamic importing
• pyface has various non-python data files (eg. icons and images)
• pyface uses entry points

PyInstaller has utility functions for each of these cases: collect_submodules, collect_data_files and collect_entry_points. These functions let you easily populate the hiddenimports and datas variables in a .spec file or PyInstaller hook.

The following code can be used as a hook (eg. hook-pyface.api.py) and are sufficient to get pure Pyface code to work:

from PyInstaller.utils.hooks import collect_submodules, collect_entry_point, collect_data_files

# we need to know about the entry points
datas, hiddenimports = collect_entry_point("pyface.toolkits")

# make sure all .py files and data files are in the package
hiddenimports += collect_submodules('pyface')
datas += collect_data_files('pyface')

If using TraitsUI, you would need to write a very similar hook for TraitsUI. If using Enable or Chaco, you would need similar for Kiva and Enable (I haven't yet tried any of these).

These are not optimized - these will include wxPython backend files even if you are only using Qt, for example - but that shouldn't matter since the size of Pyface is small compared to Qt or Wx.

Having done this research, and since the code is not large, I'll likely add the PyInstaller hooks to Pyface in the next minor release, after which it will hopefully Just Work.

[hitbuyi]

Has anyone been able to get this to work? I've been trying for days to no avail, and I'd really like to not have to get rid of traitsui...

me too, any suggestion is appreciated

[corranwebster]

Please try the instructions and code here: https://github.com/jonathanrocher/ets_tutorial/tree/master/stage8.2_packaging_pyinstaller