Allow custom remote/local python management archive locations

[mmorys]

The new python management capability is awesome. Currently the distribution archives are pulled from https://github.com/indygreg/python-build-standalone or https://downloads.python.org/pypy, which is hard coded in

hatch/src/hatch/python/distributions.py

Line 15 in f89c577

DISTRIBUTIONS: dict[str, dict[tuple[str, ...], str]] = {

I would like to propose an update to support using custom distribution sources. This would support a few use cases:

1. Users may have a mirror of the archives on a local network mirror
2. Users may be offline
3. Users may have already installed a python distribution, which could be cached and installed from a locally stored archive

To solve this issue, I would propose the following changes:

1. Instead of storing distributions in a Python file, store them in a static config file like yaml
   1. See distributions.yaml which I made that replicates the contents of distributions.py, loadable using

      import yaml
      yaml.full_load(stream=open("distributions.yml", "r"))

   2. This makes it easier for users to provide custom distributions definitions that exist outside the hatch package. Loading in python files outside of the project is awkward and risky.
2. Provide an environment variable such as "HATCH_DISTRIBUTIONS" which, if defined, can point to a path on the local system with a custom distributions.yml. For example, a user could have

export HATCH_DISTRIBUTIONS="$HOME/.config/hatch/distributions.yml"

in their .bashrc, with their custom definitions in distributions.yml. If this environment variable doesn't exist, hatch will use the default distributions located at src/hatch/python/distributions.yml.

3. Provide an option flag in the hatch python command that allows overriding the distributions list. This could be something like

hatch python --dists /path/to/distributions.yml [...]

4. Implement reading of archives from remote addresses such as https://github.com/indygreg/python-build-standalone/releases/download/20231002/cpython-3.12.0%2B20231002-aarch64-unknown-linux-gnu-install_only.tar.gz as well as local files such as file:///path/to/local/cpython-3.12.0%2B20231002-aarch64-unknown-linux-gnu-install_only.tar.gz

One complication I haven't thought through is how to handle installed distributions when the user changes the distributions list...

I would be happy to begin implementing these changes if there are no major issues and they would be merged into a future release.

[johannesloibl]

I would also need this since our on-prem Gitlab does not allow access to GitHub, but we might cache the packages on Artifactory.

[a-reich]

Similar to @johannesloibl’s comment above, this is also needed for enterprise environments like mine where access to most of the internet is firewalled. Software is supposed to be installed from our internal mirrors of various indexes.

[cvvs]

Echoing @johannesloibl and @a-reich This is a blocker for me because I need to be able to pull from a private artifact repository or internal system for everything.

[ofek]

Going to release this tonight #1455

Please let me know if this is sufficient so that this issue may be closed.

[mmorys]

Going to release this tonight #1455

Please let me know if this is sufficient so that this issue may be closed.

This is a great minimal solution that will help lots of people, and addresses the core of the problem. This issue can be closed. I think if more features are needed in the future a new issue could be opened.

[ofek]

https://hatch.pypa.io/latest/how-to/python/custom/