Python PEX Binary can't resolve internal dependencies

[delucca]

Sure! Here’s a more streamlined and clear version of the question:

---

Hi everyone!

I'm new to Pants and really enjoying it. I've set up my application, which has two main directories: apps and libs. Here’s the structure:

.
├── apps
│   └── api
└── libs
    ├── common
    ├── config
    ├── logging
    └── storage

My pyproject.toml file looks like this:

[project]
name = "my_project"
version = "0.1.0"
dependencies = [
    "fastapi>=0.11.0",
    "uvicorn>=0.30.1",
    "loguru>=0.7.2",
    "google-cloud-storage>=2.17.0",
    "google-auth>=2.31.0",
    "msgpack>=1.0.8",
    "dependency-injector>=4.41.0",
    "typing-extensions>=4.12.2"
]

[tool.pants]
pants_version = "2.21.0"

[tool.pants.backend.python]
interpreter_constraints = ["==3.11.*"]

[project.optional-dependencies]
dev = [
    "pytest>=8.2.2",
    "pytest-docker>=3.1.1",
    "httpx>=0.27.0",
]

My root BUILD file (located at the root of the repo) is:

python_requirements(
    name="reqs",
    source="pyproject.toml",
)

python_test_utils(
    name="test_utils",
    sources=["conftest.py", "tests/utils.py"],
    dependencies=[":test_files"],
)

resources(
    name="test_files",
    sources=[
        "docker-compose.yaml",
    ],
)

The BUILD file for apps/api is:

python_source(
    name="dev",
    source="src/serve.py",
    restartable=True,
)

python_sources(
    name="api",
    sources=["**/*.py"],
)

python_tests(
    name="tests",
    sources=["tests/**/test_*.py"],
)

python_test_utils(
    name="test_utils",
    sources=["tests/conftest.py"],
)

pex_binary(
    name="prod",
    entry_point="api:src/serve.py",
    dependencies=[":api"],
)

The BUILD file for libs/logging is:

python_sources(
    name="logging",
    sources=["**/*.py"],
)

python_tests(
    name="tests",
    sources=["tests/**/test_*.py"],
)

python_test_utils(
    name="test_utils",
    sources=["tests/conftest.py"],
)

The Issue:

When I run pants run apps/api:dev, everything works fine, and my server starts without any issues. However, when I try to package it for distribution using pants package apps/api:prod, it builds a file under dist, but running python dist/apps.api/prod.pex results in the following error:

Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.10.13/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user/.pyenv/versions/3.10.13/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/user/.pex/unzipped_pexes/bcf78f6358b27c64889df89320647e5714c84b7c/__main__.py", line 105, in <module>
    from pex.pex_bootstrapper import bootstrap_pex
  File "/home/user/.pex/unzipped_pexes/bcf78f6358b27c64889df89320647e5714c84b7c/.bootstrap/pex/pex_bootstrapper.py", line 13, in <module>
    from pex.environment import ResolveError
  File "/home/user/.pex/unzipped_pexes/bcf78f6358b27c64889df89320647e5714c84b7c/.bootstrap/pex/environment.py", line 12, in <module>
    from pex import dist_metadata, pex_warnings, targets
  File "/home/user/.pex/unzipped_pexes/bcf78f6358b27c64889df89320647e5714c84b7c/.bootstrap/pex/dist_metadata.py", line 27, in <module>
    from pex.pep_440 import Version
  File "/home/user/.pex/unzipped_pexes/bcf78f6358b27c64889df89320647e5714c84b7c/.bootstrap/pex/pep_440.py", line 19, in <module>
    from pex.third_party.packaging import utils as packaging_utils
  File "/home/user/.pex/unzipped_pexes/bcf78f6358b27c64889df89320647e5714c84b7c/.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/utils.py", line 8, in <module>
    from .tags import Tag, parse_tag
  File "/home/user/.pex/unzipped_pexes/bcf78f6358b27c64889df89320647e5714c84b7c/.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/tags.py", line 5, in <module>
    import logging
  File "/home/user/src/project/libs/logging/__init__.py", line 1, in <module>
    from loguru._logger import Logger
ModuleNotFoundError: No module named 'loguru'

It seems like PEX can't resolve the loguru dependency, which is only used in libs/logging. What am I missing? How can I ensure all dependencies are correctly packaged?

Thanks in advance for your help!

[delucca]

Nevermind about this, I thought pex would package the dependencies as well, I just figured out I need to install the dependencies on the venv I'm running the pex code

[benjyw]

Hi @delucca . Your initial instinct was correct - the pex should contain all the third-party dependencies, including loguru, and you should not have to install anything in any venvs - pex is supposed to be completely standalone.

So to figure out what's going on, the next step is to look inside the .pex file. Does it contain loguru? A .pex is a zipfile, so you can poke in it with unzip -t and see what's inside.

[benjyw]

Looking more closely at your error, I think the issue is that your internal logging package is colliding with the standard library logging package.

The error in the stacktrace above is for (a vendored copy of) packaging. It does import logging, which is obviously not intended to load your code.

Regardless of pex, and even more so when considering pex, I highly recommend namespacing all your code under some unique top-level name. E.g., libs/myuniquename/logging, and import myuniquename.logging, to avoid such issues.