Make host for private pypi deps configurable

[romanofski]

🚀 feature request

Relevant Rules

I think these apply to private repository rules.

Description

We are running Bazel in a network isolated environment. That is any network requests to upstream registries including pypi.org or files.pythonhosted.org will be blocked. The results in errors like these:

WARNING: Download from https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl failed: class java.net.SocketException Permission denied: getsockopt
INFO: Repository rules_python~~config~pypi__packaging instantiated at:
  <builtin>: in <toplevel>
Repository rule http_archive defined at:
  D:/a_work_temp/external/bazel_tools/tools/build_defs/repo/http.bzl:387:31: in <toplevel>
ERROR: D:/a_work_temp/external/bazel_tools/tools/build_defs/repo/http.bzl:136:45: An error occurred during the fetch of repository 'rules_python~~config~pypi__packaging':
   Traceback (most recent call last):
	File "D:/a_work_temp/external/bazel_tools/tools/build_defs/repo/http.bzl", line 136, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl] to D:/a_work_temp/external/rules_python~~config~pypi__packaging/temp10535568378801797934/packaging-24.0-py3-none-any.whl.zip: Permission denied: getsockopt
ERROR: error loading package 'tools/bazel': Encountered error while reading extension file 'requirements.bzl': no such package '@@rules_python~~pip~pip_dev//': no such package '@@rules_python~~config~pypi__packaging//': java.io.IOException: Error downloading [https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl] to D:/a_work_temp/external/rules_python~~config~pypi__packaging/temp10535568378801797934/packaging-24.0-py3-none-any.whl.zip: Permission denied: getsockopt
ERROR: D:/a/_work/1/s/tools/nginx/BUILD.bazel:11:24: error loading package 'tools/bazel': Encountered error while reading extension file 'requirements.bzl': no such package '@@rules_python~~pip~pip_dev//': no such package '@@rules_python~~config~pypi__packaging//': java.io.IOException: Error downloading [https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl] to D:/a_work_temp/external/rules_python~~config~pypi__packaging/temp10535568378801797934/packaging-24.0-py3-none-any.whl.zip: Permission denied: getsockopt and referenced by '//tools/nginx:clipchamp-dev-ssl-key'
ERROR: Analysis of target '//tools/nginx:server' failed; build aborted: Analysis failed

I think it's the dependencies configured here:

rules_python/python/private/pypi/deps.bzl

Line 54 in 5fe50fb

"https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl",

The problem is that I seem not to be able to configure the host that is used here. I thought I could cache the artifacts with a repository_cache, but that would still mean I need an unrestricted network once in a while which I don't have.

Describe the solution you'd like

Excuse my ignorance on that front, but perhaps an environment variable? PYPI_INDEX_URL might already work?
The private registries do support basic auth where the username is ignored and the password is a short lived token.

Describe alternatives you've considered

I've been looking into setting up a repository cache but unfortunately no network means no network.