Help Taming RecursionError

[htgoebel]

EDIT: Fix en route. Original comment:

If you encounter a "RecursionError:maximum recursion depth exceeded", please help taming this beast. I was not yet able to reproduce this on my system, thus you need to provide data so I can try to fix the issue.

Please follow these instructions, which (hopefully) provide the required information). Thanks!

0. Prepare a clean environment

python -m venv pyivenv
cd pyivenv

source ./bin/activate    # Posix bash/zsh
.\Scripts\activate.bat   # Windows cmd.exe
.\Scripts\Activate.ps1   # Windows PowerShell

pip install -U pip  # upgrade pip

export PYTHONHASHSEED=0   # Posix
set PYTHONHASHSEED=0      # Windows cmd.exe
$env:PYTHONHASHSEED = "0" # Windows PowerShell
# verify hash randomization is disabled
python -c "import sys; assert sys.flags.hash_randomization == 0"

# install PyInstaller
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.tar.gz

1. go to you project and install its requirements

cd /path/to/you/application
pip install -r ....

pip list > pip-list.txt
pip freeze > pip-freeze.txt

Now comes the important part:

2. Generate a .spec-file (if you don't have one already):

pyi-makespec main.py

3. Verify pyinstaller runs into RecursionError in this setup:

pyinstaller main.spec

If this does not fail, ensure the .spec-file does not sys.setrecursionlimit()

4. Please try to generate a as minimal as possible version of you application, which will still make PyInstaller run into RecursionError. Ideally the application only consists of on import statement and still fails to freeze.

   Please, please take your time for doing so! The TRACE-output you will generate in the next steps will be HUGE and hard to analyze anyway. Please help keeping the data small. Otherwise I will have barely no chance to find the error.

5. Run PyInstaller in TRACE mode, redirecting the output into a file. The outout will be huge, several thousand lines, so you really want to redirect the output into a file::

pyinstaller main.spec --log=TRACE > failing.log 2>&1

6. Now iterative try to find a small new recursion limit which will make PyInstaller pass. Add these lines near the top of the .spec-file

import sys
sys.setrecursionlimit(sys.getrecursionlimit()+20)
print("Recursion limit:", sys.getrecursionlimit())

then run pyinstaller main.spec again. If it fails, increase the number, if it passes, decrease the number. Repeat until you have a lower bound (in steps 10) where PyInstaller passes.

7. With this lower bound run

pyinstaller main.spec --log=TRACE > passing.log 2>&1

8. Zip all the log-files and send them to my via email. You can find my E-Mail at http://www.pyinstaller.org/support.html

zip trace-logs.zip pip-list.txt pip-freeze.txt failing.log passing.log

I only need these log-file, nothing of the stuff built.

9. Keep this environment! And make sure you can reproduce it even in a few weeks time. It will take some time to analyze the data and to propose a fix - and we will need the environment to verify the fix.

[jnewbigin]

I have sent you an email with my logs.

For anyone else, I hit this problem with import mypy. Alas, with no workaround at this time

[Jaime02]

I'm having this annoying issue in my main computer. I am able to use pyinstaller in other computers with the same code so it is a computer specific problem. I have a virtual machine in my main computer where it works, It is very strange.

Edit: I'm able to generate the exe with Python 3.8.0 and:

altgraph       0.16.1
future         0.18.2
mpmath         1.1.0
pefile         2019.4.18
pip            19.2.3
PyInstaller    4.0.dev0+5d4bbbeea
PyOpenGL       3.1.0
PyQt5          5.13.2
PyQt5-sip      12.7.0
pywin32-ctypes 0.2.0
setuptools     41.2.0
sympy          1.6.dev0

Edit 2: Pyinstaller worked but fbs don't, it shows this error:

Log

``` fbs freeze c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\mpmath\ctx_mp_python.py:892: SyntaxWarning: "is" with a literal. Did you mean "=="? if other is 0: c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\mpmath\ctx_mp_python.py:986: SyntaxWarning: "is" with a literal. Did you mean "=="? if other is 0: Traceback (most recent call last): File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\runpy.py", line 192, in _run_module_as_main return _run_code(code, main_globals, None, File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\usuario01\AppData\Local\Programs\Python\Python38-32\Scripts\pyinstaller.exe\__main__.py", line 9, in File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\__main__.py", line 111, in run run_build(pyi_config, spec_file, **vars(args)) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\__main__.py", line 63, in run_build PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\build_main.py", line 838, in main build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build')) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\build_main.py", line 784, in build exec(text, spec_namespace) File "", line 18, in File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\api.py", line 98, in __init__ self.__postinit__() File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\datastruct.py", line 158, in __postinit__ self.assemble() File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\api.py", line 128, in assemble self.code_dict = { File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\api.py", line 129, in key: strip_paths_in_code(code) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\utils.py", line 654, in strip_paths_in_code consts = tuple( File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\utils.py", line 655, in strip_paths_in_code(const_co, new_filename) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\PyInstaller\building\utils.py", line 662, in strip_paths_in_code return code_func(co.co_argcount, co.co_kwonlyargcount, co.co_nlocals, co.co_stacksize, TypeError: an integer is required (got type bytes) Traceback (most recent call last): File "C:\Users\usuario01\AppData\Local\Programs\Python\Python38-32\Scripts\fbs-script.py", line 11, in load_entry_point('fbs==0.8.6', 'console_scripts', 'fbs')() File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\fbs\__main__.py", line 17, in _main fbs.cmdline.main() File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\fbs\cmdline.py", line 32, in main fn(*args) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\fbs\builtin_commands\__init__.py", line 120, in freeze freeze_windows(debug=debug) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\fbs\freeze\windows.py", line 18, in freeze_windows run_pyinstaller(args, debug) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\site-packages\fbs\freeze\__init__.py", line 47, in run_pyinstaller run(args, check=True) File "c:\users\usuario01\appdata\local\programs\python\python38-32\lib\subprocess.py", line 512, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['pyinstaller', '--name', 'Dibujo técnico', '--noupx', '--log-level', 'ERROR', '--noconfirm', '--windowed', '--icon', 'D:\\Pictures\\Camera Roll\\Windows\\No hay nada no entres\\He dicho que no\\EL QUE AVISA ES AVISADOR\\Pyton\\Prueba fbs\\src\\main\\icons\\Icon.ico', '--distpath', 'D:\\Pictures\\Camera Roll\\Windows\\No hay nada no entres\\He dicho que no\\EL QUE AVISA ES AVISADOR\\Pyton\\Prueba fbs\\target', '--specpath', 'D:\\Pictures\\Camera Roll\\Windows\\No hay nada no entres\\He dicho que no\\EL QUE AVISA ES AVISADOR\\Pyton\\Prueba fbs\\target\\PyInstaller', '--workpath', 'D:\\Pictures\\Camera Roll\\Windows\\No hay nada no entres\\He dicho que no\\EL QUE AVISA ES AVISADOR\\Pyton\\Prueba fbs\\target\\PyInstaller', '--additional-hooks-dir', 'c:\\users\\usuario01\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\fbs\\freeze\\hooks', '--runtime-hook', 'D:\\Pictures\\Camera Roll\\Windows\\No hay nada no entres\\He dicho que no\\EL QUE AVISA ES AVISADOR\\Pyton\\Prueba fbs\\target\\PyInstaller\\fbs_pyinstaller_hook.py', 'D:\\Pictures\\Camera Roll\\Windows\\No hay nada no entres\\He dicho que no\\EL QUE AVISA ES AVISADOR\\Pyton\\Prueba fbs\\src\\main\\python\\main.py']' returned non-zero exit status 1. ```

Edit 3: I am able to generate the .exe at my main PC, I had never been able to do it until now. I completely reinstalled Python 3.8.2 with the necessary modules and it worked!

Modules:

Package         Version
--------------- ------------------
altgraph        0.16.1
cycler          0.10.0
future          0.18.2
kiwisolver      1.1.0
mpmath          1.1.0
numpy           1.17.3
pefile          2019.4.18
pip             19.2.3
PyInstaller     4.0.dev0+g5d4bbbee
PyOpenGL        3.1.5
pyparsing       2.4.2
PyQt5           5.14.0
PyQt5-sip       12.7.0
PyQt5-stubs     5.13.1.3
python-dateutil 2.8.0
pywin32         225
pywin32-ctypes  0.2.0
setuptools      41.2.0
six             1.12.0
sympy           1.6.dev0

[Legorooj]

@Jaime02 crap - I must have acidentally deleted it. (Groan).

To replicate:

(In a fresh venv)

# Test.py
from sympy import *

pyinstaller test.py --log-level=TRACE

[Legorooj]

Ok I've traced the subsystem. It's modulegraph (suprise suprise) and the part known to be recursive. I'll let you know more when I have it.

[Legorooj]

Ok everyone. A fix is on the way.

• 👍 Upside: This fix will fix loads of bugs.

• 👎 Downside: Estimated landing period: several months. Probably Nov-Jan. (Maybe sooner?)

Fix: We (The pyinstaller developers) are replacing ModuleGraph... with ModuleGraph 2. A non-recursive implementation. Expect this to take some time.

[smasoudn]

@Legorooj
@htgoebel
I keep getting the following error when I use fastapi in my python application.

File "..../miniconda3/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", 
line 2912, in _load_package self._load_module(fqname, fp, buf, stuff)
File "..../miniconda3/lib/python3.7/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", 
line 2093, in _load_module m = self._load_package(fqname, pathname, packagepath)

RecursionError: maximum recursion depth exceeded while calling a Python object

I added import sys; sys.setrecursionlimit(50000) to the original python file as well as in the spec but still getting the same problem.

The interesting thing is that I don't get that error if I don't use fastapi (It works with Flask for example). Looks like PyInstaller has some issue with fastapi.

This is the code in my main python function (for reproducing the error):

from fastapi import FastAPI
import uvicorn
app = FastAPI()

if __name__ == "__main__":
    uvicorn.run("main:app", host="0.0.0.0", workers=1, port=5000)

[Legorooj]

@smasoudn this is the same error we've all been getting. I got it with sympy. I'd say that for some reason, modulegraph isn't being smart about recursion, because there's a bug. It's probably exploring the same branch of a fractal tree over and over again.

[Legorooj]

@AIprototype no, there isn't yet. I'll be getting to work on actually implementing the fix in a few weeks.

[Jaime02]

@AIprototype Try using Python 3.8.x in a fresh new environment

[xoviat]

@bwoodsend Is there another plan other than the one I had for slowly incorporating modulegraph2? Because I would be willing to take that forward if I had the support of the developers. As expected, the other plans are not working toward a solution.

[bwoodsend]

I don't see how slowly incorporating modulegraph2 can work. As there is only one API component in modulegraph (namely the modulegraph.ModuleGraph), either we use the new one or we use the old one - there's no nothing inbetween to increment through. So a fix IMO would have to be the full transition in one go. i.e. Remove all of modulegraph, add all of modulegraph2, mop up the resulting mess.

[xoviat]

@bwoodsend What about this: I can incrementally update the ModuleGraph API to match the modulegraph 2 API. There are a few different method names and some changes that need care. This way, modulegraph2 can be closer to a drop-in replacement. I can open a new tracking issue to track the methods that need to be renamed.

[bwoodsend]

If there's enough symmetry between modulegraph and modulegraph2 that that is possible then that sounds like a good plan.

[bwoodsend]

Ah no, that's something else entirely. That test is un-runable on Python >= 3.9.6 because of a recursion check added to Python itself so we've had to remove it (see e188739 which tells you the same thing in more words).

[rokm]

@mcepl what version of PyInstaller is that? That test was removed in 53949b0, which should be in 4.5 or 4.5.1.

The log.txt you attached seems to suggest you are using PyInstaller 3.6?

[mcepl]

@mcepl what version of PyInstaller is that? That test was removed in 53949b0, which should be in 4.5 or 4.5.1.

The log.txt you attached seems to suggest you are using PyInstaller 3.6?

Not by best moment, of course, when I upgraded to 4.5.1, everything works correctly. Thank you.