[BUG] Users report problems with installing modules when they have the latest 3.0.9 version of pyparsing installed

[ptmcg]

setuptools version

62.6.0 and "latest"

Python version

Unknown

OS

Unknown

Additional environment information

Users report problems with installing modules when they have the latest 3.0.9 version of pyparsing installed.

pyparsing/pyparsing#419

Description

Somehow, the vendored version of pyparsing (3.0.8) references a file (template.jinja2) which exists in the pkg_resources vendored version of pyparsing but has been removed in the latest version of pyparsing (3.0.9).

One user reports:

Warning occurs creating python executable file:

WARNING: Failed to collect submodules for 'pkg_resources._vendor.pyparsing.diagram' because importing 'pkg_resources._vendor.pyparsing.diagram' raised: FileNotFoundError: [Errno 2] No such file or directory: 'C:/python/Lib/site-packages/pkg_resources/_vendor/pyparsing/diagram/template.jinja2'

A second user reports:

I have the same issue, this should be fixed, because it's a very important problem. Now I can't load a submodule, and I have the latest versions of setuptools, pyparsing

Expected behavior

Expect installs with no warnings, since the vendored pyparsing does contain the template.jinja2 file.

How to Reproduce

Descriptions in pyparsing/pyparsing#419. I will instruct those reporting on that bug to add further details here.

Output

See Description above.

[jpp08022002]

I use windows 11, python 3.9.12 and setuptools version 63.2.0 and pyparsing 3.0.9 versions and got that error.

I make an executable with a main python file and a sub file. Before updating the above mentioned package, I got an other error, where suggested to update them. Besides the 2 python files, for my app I have a: png file, an ico file, and three modules: customtkinter, openpyxl and treepoem

[ptmcg]

The pyparsing 3.0.8 code that references the missing file is (see changes in pyparsing/pyparsing@f655b95)

from pkg_resources import resource_filename
...

with open(resource_filename(__name__, "template.jinja2"), encoding="utf-8") as fp:
    template = Template(fp.read())

The vendored version includes template.jinja2 where it is expected, but in pyparsing 3.0.9, this file no longer exists.

Why would the vendored pyparsing be looking for template.jinja2 in the installed 3.0.9 pyparsing?

[ptmcg]

I use windows 11, python 3.9.12 and the latest setuptools and pyparsing versions and got that error

@jpp08022002 - please be specific on the installed versions of setuptools and pyparsing, not just "latest" (you can just edit your previous comment); it would also help if you described what commands you ran that resulted in the error. Thanks

[ptmcg]

Will releasing a new pyparsing version containing the file resolve this issue? I don't mind doing it, but I'd at least like to know why doing it fixes this problem.

[abravalheri]

Hi @ptmcg thank you very much for reporting this.

I imagine that the best action here would be provide a fix on the setuptools end. Probably an update of the bundled version of pyparsing is the easiest thing to do.

Why would the vendored pyparsing be looking for template.jinja2 in the installed 3.0.9 pyparsing?

Setuptools uses a MetaPathFinder for its vendored dependencies, and MetaPathFinders are tricky 😢. There is a chance the bundled pyparsing thinks its __name__ is pyparsing instead of setuptools.extern.pyparsing or setuptools._vendor.pyparsing... I would have to investigate it better to understand what is really happening.

[ptmcg]

Given pyparsing's rollercoaster releases since 3.0.0, I wanted to double-check how much risk you would be looking at in upping to 3.0.9. The CHANGES log doesn't show any changes to internal parsing or whitespace-skipping (the two areas that give most problems in updating), mostly type annotations and doc string updates. And the one file deletion, of course. So I think an update in setuptools to pyparsing 3.0.9 is a safe option. (By comparison, 3.0.10 will have more extensive changes, so it will take a little more deliberation before jumping on it.)

[abravalheri]

In #3453, I am proposing to update the version of the vendored pyparsing, which should be a "dirty fix" for this problem.

However, to investigate this issue deeply (and see if there is anything else we can do) I would need a proper minimal reproduction, which seems to be missing in the original ticket...
@netmonitoring, @jpp08022002 could you please provide something like this?

(I saw the original problem involves pyinstaller... Does it mean that the error just happens with pyinstaller? I tried to have both packages installed in the reported versions inside the same directory and everything seems to be fine...).

[abravalheri]

Given pyparsing's rollercoaster releases since 3.0.0, I wanted to double-check how much risk you would be looking at in upping to 3.0.9. The CHANGES log doesn't show any changes to internal parsing or whitespace-skipping (the two areas that give most problems in updating), mostly type annotations and doc string updates. And the one file deletion, of course. So I think an update in setuptools to pyparsing 3.0.9 is a safe option. (By comparison, 3.0.10 will have more extensive changes, so it will take a little more deliberation before jumping on it.)

Thank you very much @ptmcg, this is very valuable information!

[jpp08022002]

reproduction:

Have these two python files, a png image and an ico file:

So I open auto-py-to-exe in anaconda,
Select the proces_excel_main_module.py as script location.
Select home.ico as icon file.
For additional files, select:

• proces_excel_sub_module.py
• app picture
• the home.ico file, because when I don't, it won't be in the output folder
• from site_packages in anaconda3:
  • treepoem
  • customtkinter
  • openpyxl
    Select also OneDir, and Windowed

After that running it, and after some time: I got that error.
When I run the executable, I can't execute the submodulefunction.

[abravalheri]

Thank you very much @jpp08022002.

I was looking for something that reduces the amount of moving parts... The original issue mentions only PyInstaller (no Anaconda, auto-py-to-exe, gui's etc)...

I tried the following:

# powershell
python -m virtualenv .venv

.venv\Scripts\python -m pip install 'setuptools==63.2.0' 'wheel==0.37.1' 'pip==22.1.0' 'pyparsing==3.0.9' 'pyinstaller==5.1'

mkdir testpkg
cd testpkg

echo "import pyparsing; print('hello world')" | out-file cli.py -encoding utf8
..\.venv\Scripts\pyinstaller cli.py --onefile
.\dist\cli.exe
# hello world

And everything seems to be fine...

[jpp08022002]

I still got the same error!
I use auto-py-to-exe, so there must be something done with that module to make it not searching for pyparsing/..../template.jinja !!!

I don't know very much about packages, but I want to make my python file executable with treepoem, customtkinter and openpyxl. This raised the above mentioned error and makes my executable not working!

[ptmcg]

@jpp08022002 It seems that the actual code in your files is relevant to reproducing this problem. Is this code in a public Github repository?

I know it is frustrating that @abravalheri is unable to reproduce a problem that is clearly reproducible to you. But they have tried their best with the information you have provided.

If there is any way that you can whittle down the problem on your end - such as pruning proces_excel_main_module.py down to just the smallest bit of code that still creates the problem - and then try reproducing just with pyinstaller without bringing in more tools, Anaconda, etc. Going through this process to create a Minimal Reproducible Example will help you isolate what particular feature is causing the problem, and documenting the environment setup steps and providing the minimal code will give @abravalheri a much better platform for reproducing it on their end.

[abravalheri]

Thank you guys (sorry if it took me some time, I was in holidays meanwhile).

I upgraded the vendored version of pyparsing to fix the issue (even if we don't fully understand what is happening). It should be available in a new release soon.

If at some point in the future someone would like to contribute with a minimal reproducer for the specific version of setuptools that was causing the problem, we can still try to investigate it deeper.