Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop requiring `__file__` in Python packages #69

Open
indygreg opened this issue Jul 2, 2019 · 3 comments

Comments

@indygreg
Copy link
Owner

commented Jul 2, 2019

The Problem

Many Python modules and scripts use __file__ to derive the filesystem path to the current file.

As documented at https://docs.python.org/3/reference/datamodel.html (search for __file__), __file__ is optional (the __file__ attribute may be missing for certain types of modules).

However, because Python has traditionally relied on filesystem-based imports and hasn't had a stable story around non-module resource handling, , __file__ is almost always defined and has been used to locate and load files next to Python source files for seemingly forever. This is arguably tolerable. But reliance on __file__ undermines tools - like PyOxidizer - which don't import Python modules from the filesystem. This in turn constrains the flexibility and utility of the larger Python ecosystem.

The Solution

Python code should be rewritten to not assume the existence of __file__. By doing so, Python code will be more compatible with more Python execution environments (such as PyOxidizer), and this benefits the overall Python ecosystem.

Instructions for writing portable Python code that doesn't rely on __file__ can be found at https://pyoxidizer.readthedocs.io/en/latest/packaging_pitfalls.html#reliance-on-file.

This Issue

This issue can serve as a focal point for tracking and coordinating Python packages and tools which currently rely on __file__ but shouldn't. If you file a GitHub issue against a project that relies on __file__, you can reference this issue by typing indygreg/PyOxidizer#69 and provide Python project maintainers with enough context to make informed decisions about the use of __file__ in their projects.

@QuantumChamploo

This comment has been minimized.

Copy link

commented Jul 7, 2019

I am having a similar issue, just when I am trying to import a package, for example numpy. So I am not sure how to change my code not to depend on file as the only line of code I have is "import numpy"

I have this issue when I use the repl mode, the eval mode, and when I try to run a script

indygreg added a commit that referenced this issue Jul 11, 2019

repackage: emit a warning when Python source contains __file__
The warning instructs them to look for more info at #69.

gnprice added a commit to gnprice/hello-pyoxidizer that referenced this issue Jul 16, 2019

Demonstrate pulling in a random library from PyPI.
I chose the library for this demo like so:
* Went to https://pypi.org/ and looked at "Trending projects".
* Tried each one, using example code from its README.
* Went with the first one that worked.

In the sample that happened to give me today, this was the 4th
on the list:

* `cpp-demangle` was first, and failed at `pip install` time
  with an error about `setuptools_rust`.  I think it just doesn't
  build from source in a stock environment; I get the same error
  with `pip install .`, in a fresh venv, after cloning the source.

* `pygrok` failed at import time, trying to use `__file__`.
  See indygreg/PyOxidizer#69.

* `python-whois` failed at import time: imports `past`, which
  imports `lib2to3`, which uses `__file__`.

* `area` works.
@Alex-Mann

This comment has been minimized.

Copy link

commented Aug 9, 2019

Is there a recommended strategy for patching these modules locally to try and get them to work with PyOxidizer? Currently I am unable to find a way to build when I use a module that contains the __file__ variable.

I have set up a venv, downloaded all the packages that I am using locally, and have tried to monkey patch the __file__ variables where I am hitting errors, but it seems like pyoxidize run is using a cached version of the modules somewhere (unless they get pulled directly from pip each time) since the same error is still showing even though I have removed the __file__ variables.

Is it possible to somehow search and replace the __file__ variable when it is found? I've only just started using this module but it seems like many python packages are probably using it as well. Expecting them to accept pull requests / updates to deprecate this in the near future doesn't really seem all that feasible.

@Alex-Mann

This comment has been minimized.

Copy link

commented Aug 9, 2019

Ok, after doing some more digging around the repo, I found the solution offered for the black example. I didn't see this earlier so I didn't know it existed. The package I'm using is GitPython, so what I did was add:

[[embedded_python_config]]
sys_paths = ["$ORIGIN/lib"]

[[packaging_rule]]
type = "pip-install-simple"
package = "gitpython"
install_location = "app-relative:lib"

Now, the python application that I am developing lives in a virtualenv, so originally I was just using that virtualenv to dictate what packages needed to be rolled up. Is there an equivalent command for install_location = "app-relative:lib" and sys_paths = ["$ORIGIN/lib"] to get this to work with a virtualenv?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.