New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The IPython hook and its dependencies don't find all required files #3535

Closed
bjones1 opened this Issue May 24, 2018 · 9 comments

Comments

Projects
None yet
8 participants
@bjones1
Member

bjones1 commented May 24, 2018

IPython's dependencies don't get correctly packaged. I think that Pyinstaller should, by default, package everything for any package that doesn't have a hook. (I remember having this discussion earlier, but don't remember where.)

For now, I'm going to disable this test, since fixing it will take some work.

Here's error messages from an Appveyor run. (To find this, download the log file; clicking on the Tests tab doesn't display the full traceback).

[00:37:56] Traceback (most recent call last):
[00:37:56]   File "C:\Users\appveyor\AppData\Local\Temp\1\pytest-of-appveyor\pytest-0\popen-gw1\test_ipython_onedir_0\test_ipython.py", line 3, in <module>
[00:37:56]     from IPython import embed
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\IPython\__init__.py", line 55, in <module>
[00:37:56]     from .terminal.embed import embed
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\IPython\terminal\embed.py", line 16, in <module>
[00:37:56]     from IPython.terminal.interactiveshell import TerminalInteractiveShell
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\IPython\terminal\interactiveshell.py", line 32, in <module>
[00:37:56]     from .debugger import TerminalPdb, Pdb
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\IPython\terminal\debugger.py", line 6, in <module>
[00:37:56]     from IPython.core.completer import IPCompleter
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\IPython\core\completer.py", line 146, in <module>
[00:37:56]     import jedi
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\__init__.py", line 41, in <module>
[00:37:56]     from jedi.api import Script, Interpreter, set_debug_function, \
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\api\__init__.py", line 23, in <module>
[00:37:56]     from jedi.api import classes
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\api\classes.py", line 11, in <module>
[00:37:56]     from jedi.evaluate.utils import ignored, unite
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\evaluate\__init__.py", line 73, in <module>
[00:37:56]     from jedi.evaluate import imports
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\evaluate\imports.py", line 24, in <module>
[00:37:56]     from jedi.evaluate import sys_path
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\evaluate\sys_path.py", line 6, in <module>
[00:37:56]     from jedi.evaluate.helpers import is_string
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\evaluate\helpers.py", line 12, in <module>
[00:37:56]     from jedi.evaluate.compiled import CompiledObject
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\evaluate\compiled\__init__.py", line 2, in <module>
[00:37:56]     from jedi.evaluate.compiled.context import CompiledObject, CompiledName, \
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\evaluate\compiled\context.py", line 16, in <module>
[00:37:56]     from . import fake
[00:37:56]   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
[00:37:56]   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
[00:37:56]   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
[00:37:56]   File "C:\projects\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 628, in exec_module
[00:37:56]     exec(bytecode, module.__dict__)
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\evaluate\compiled\fake.py", line 25, in <module>
[00:37:56]     _path_dict = _get_path_dict()
[00:37:56]   File "c:\python36-x64\lib\site-packages\jedi\evaluate\compiled\fake.py", line 19, in _get_path_dict
[00:37:56]     for file_name in os.listdir(base_path):
[00:37:56] FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\pytest-of-appveyor\\pytest-0\\popen-gw1\\test_ipython_onedir_0\\dist\\test_ipython\\jedi\\evaluate\\compiled\\fake'

Running the binary produced on Windows, then performing a completion (for example, import os then os.p ) produces:

In [2]: if (os.pException in thread Thread-90:
Traceback (most recent call last):
  File "lib\site-packages\parso\grammar.py", line 265, in load_grammar
KeyError: 'e:\\tmp\\test_ipython_onedir_0\\dist\\test_ipython\\parso\\python\\grammar36.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "lib\site-packages\parso\grammar.py", line 268, in load_grammar
FileNotFoundError: [Errno 2] No such file or directory: 'e:\\tmp\\test_ipython_onedir_0\\dist\\test_ipython\\parso\\python\\grammar36.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "lib\site-packages\prompt_toolkit\interface.py", line 865, in run
  File "lib\site-packages\IPython\terminal\ptutils.py", line 84, in get_completions
  File "lib\site-packages\IPython\terminal\ptutils.py", line 94, in _get_completions
  File "lib\site-packages\IPython\core\completer.py", line 438, in _deduplicate_completions
  File "lib\site-packages\IPython\core\completer.py", line 1791, in completions
  File "lib\site-packages\IPython\core\completer.py", line 1831, in _completions
  File "lib\site-packages\IPython\core\completer.py", line 1998, in _complete
  File "lib\site-packages\IPython\core\completer.py", line 1348, in _jedi_matches
  File "lib\site-packages\jedi\api\__init__.py", line 359, in __init__
  File "lib\site-packages\jedi\api\__init__.py", line 113, in __init__
  File "lib\site-packages\parso\grammar.py", line 283, in load_grammar
  File "lib\site-packages\parso\grammar.py", line 275, in load_grammar
NotImplementedError: Python version None is currently not supported.

@bjones1 bjones1 added the hooks label May 24, 2018

bjones1 added a commit to bjones1/pyinstaller that referenced this issue May 24, 2018

bjones1 added a commit to bjones1/pyinstaller that referenced this issue May 24, 2018

bjones1 added a commit to bjones1/pyinstaller that referenced this issue May 25, 2018

bjones1 added a commit to bjones1/pyinstaller that referenced this issue Jun 15, 2018

@lamourj

This comment has been minimized.

lamourj commented Jun 27, 2018

Hey there, is there a workaround for this issue in the meantime ?
I am happy to contribute to improving the hook as well, if pointed to the right places and documentation 😄

@dimitriepirghie

This comment has been minimized.

Contributor

dimitriepirghie commented Jun 28, 2018

I am also interested, @bjones can you provide more information ?

@bjones1

This comment has been minimized.

Member

bjones1 commented Jun 29, 2018

@lamourj and @dimitriepirghie, I'd welcome your help. I have no workaround, or time to track this down by myself. Based on the output above, it looks like the jedi and parso packages are missing some files -- see https://pyinstaller.readthedocs.io/en/v3.3.1/when-things-go-wrong.html.

@dimitriepirghie

This comment has been minimized.

Contributor

dimitriepirghie commented Jul 2, 2018

@lamourj and @bjones1 I have a fix/workaround, use the hook-jedi.py and add ipykernel.datapub as hidden import.

@bjones1 bjones1 closed this Jul 14, 2018

@lingchen1991

This comment has been minimized.

lingchen1991 commented Aug 4, 2018

@dimitriepirghie Hi, Mr. Epirghie. I am facing a similar problem. I would like to know more detail how you fix it. You mentioned that you use the hook-jedi.py and add ipykernel.datapub as hidden import. The hidden import is easy to do. I am wondering what and where the hook-jedi.py is and how do I use it. May I have your advice?

========================================================
I got your mean in "Add jedi hook"! It works! Thank you Mr. Epirghie!

@ecustmcilab

This comment has been minimized.

ecustmcilab commented Aug 23, 2018

@lingchen1991 Hi chen, i hat met this problem,and i also have no idea about how to get and use the hook-jedi.py. would you give more detail

@dimitriepirghie

This comment has been minimized.

Contributor

dimitriepirghie commented Aug 23, 2018

@ecustmcilab hook-jedi.py is merge on develop branch, sync you repository and it should works.

@yimelia

This comment has been minimized.

yimelia commented Nov 12, 2018

hi @dimitriepirghie , i add the hook-jedi.py , and use pyinstaller to pack my PyQt application with jedi, but failed:

18919 INFO: Loading module hook "hook-jedi.py"...

my code is:

import jedi
print(dir(jedi))
jedi.preload_module(["xx"])

and output:

 ['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

error:

    File "e:\project\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 6
31, in exec_module
    exec(bytecode, module.__dict__)
  File "app\widgets\text_editor\editor_core.py", line 14, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "e:\project\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 6
31, in exec_module
    exec(bytecode, module.__dict__)
  File "app\widgets\text_editor\editor_bridge.py", line 14, in <module>
AttributeError: module 'jedi' has no attribute 'preload_module'
@mccullerlp

This comment has been minimized.

mccullerlp commented Dec 3, 2018

For anyone hitting this again. IPython also will need the hook-parso.py added in #3901 for autocompletion to function.

cowo78 pushed a commit to cowo78/pyinstaller that referenced this issue Dec 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment