pkgutil.iter_modules is broken on MacOS

[gentlegiantJGC]

Description of the issue

I am trying to build an application on MacOS that uses pkguilt.iter_modules to dynamically find and load modules however it is not finding any modules.
The modules are there and can be imported with importlib.import_module.
Note that I am using PyInstaller with FBS but I don't believe the latter changes anything that would effect this.

The following is an example project structure

my_package
    my_module_1
    my_module_2
    my_module_3

Here is the code that does not work

import importlib
import pkgutil
m = importlib.import_module("my_package")
print("Found nested modules", list(pkgutil.iter_modules(m.__path__, m.__name__ + ".")))

The last line prints an empty list when built with PyInstaller.

I have done a bit of a dive into the code and I believe this is a rather complex mixup with paths and symbolic links.
https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py

It looks like #6539 had a go at supporting symbolic links but it only resolves symbolic links up to the _MEIPASS directory.

On MacOS _MEIPASS is equal to "/path/to/my_package.app/Contents/MacOS/"

my_package.__path__ is equal to ["/path/to/my_package.app/Contents/MacOS/my_package"]

/path/to/my_package.app/Contents/MacOS/my_package is a symbolic link to /path/to/my_package.app/Contents/Resources/my_package when there are data files present in that package.

The iter_modules hook expands symbolic links in the path variable which converts "/path/to/my_package.app/Contents/MacOS/my_package" to "/path/to/my_package.app/Contents/Resources/my_package"

The following code then skips over the directory because it does not start with SYS_PREFIX

                if not pkg_path.startswith(SYS_PREFIX):
                    # If the path does not start with sys._MEIPASS, it cannot be a bundled package.
                    continue

I don't know how this should be implemented but we need to resolve the expected path relative to the _MEIPASS directory and then expand symbolic links and only then skip if the package path does not start with the directory.

Context information (for bug reports)

• Output of pyinstaller --version: 5.13.0
• Version of Python: Python 3.11.4 (main, Aug 22 2023, 11:47:24) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
• Platform: OS X
• How you installed Python: pyenv
• Did you also try this on another platform? Does it work there? This only effects MacOS as far as I am aware

[gentlegiantJGC]

That bug report I just linked is another project that doesn't use FBS so I don't think it effects it.

[bwoodsend]

/path/to/my_package.app/Contents/MacOS/my_package is a symbolic link to /path/to/my_package.app/Contents/Resources/my_package however the code does not resolve that.

Are you adding your packages as data files then? Your package shouldn't be there at all -- it should be embedded inside the executable.

[gentlegiantJGC]

The actual code is embedded in the executable but the __path__ attribute still resolves to the Contents/Resources directory.

The import mechanism may still be resolving them via Contents/MacOS which redirects to Contents/Resources

[gentlegiantJGC]

I have just put together a minimal test example to debug this behaviour.
What I said earlier was wrong. my_package.__path__ is actually equal to ["/path/to/my_package.app/Contents/MacOS/my_package"]

Your iter_modules hook then resolves the symbolic links and expands it to "/path/to/my_package.app/Contents/Resources/my_package" like I said earlier.

The symbolic links in MacOS are only created when there are data files in that package so the issue only happens when data files are present in that package.

I will update the original post