Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support netrc-based authentication for python_repository rule #1417

Merged
merged 5 commits into from
Sep 21, 2023
Merged

feat: Support netrc-based authentication for python_repository rule #1417

merged 5 commits into from
Sep 21, 2023

Conversation

LINKIWI
Copy link
Contributor

@LINKIWI LINKIWI commented Sep 16, 2023

This change introduces support for netrc and auth_patterns attributes in python_repository (and by extension, python_register_toolchains). This allows consuming projects to fetch custom Python toolchain binaries from a private/authenticated HTTP host when specified directly by URL in python_register_toolchains.

The implementation proposed here mirrors that of http_archive: https://github.com/bazelbuild/bazel/blob/1cf392ff3918386858b8c038f82c013b1e04be98/tools/build_defs/repo/http.bzl#L116

Fixes #1215.

python/repositories.bzl Outdated Show resolved Hide resolved
python/repositories.bzl Show resolved Hide resolved
@LINKIWI
Copy link
Contributor Author

LINKIWI commented Sep 21, 2023

Thanks for the review. I colocated _get_auth in the same bzl file, but let me know if it should be placed somewhere else instead.

@LINKIWI LINKIWI requested a review from aignas September 21, 2023 21:15
@aignas
Copy link
Collaborator

aignas commented Sep 21, 2023

Thanks for the fixes, in general LGTM, but was wondering if the same behaviour could be achieved with using the --credential_helper feature. Have you tried it, does it work for your usecase?

@aignas
Copy link
Collaborator

aignas commented Sep 21, 2023

@LINKIWI, since the PR message will be used as a commit message, could you please move everything under Test Plan to a comment on the PR as opposed to the PR description?

@LINKIWI
Copy link
Contributor Author

LINKIWI commented Sep 21, 2023

Test Plan

Generated a patch based on this PR's changest and applied locally as follows:

...
http_archive(
    name = "rules_python",
    patch_args = ["-p1"],
    patches = ["//:rules_python.patch"],
    sha256 = "5868e73107a8e85d8f323806e60cad7283f34b32163ea6ff1020cf27abef6036",
    strip_prefix = "rules_python-0.25.0",
    urls = [...],
)
...

python_repository callsite:

...
load("@rules_python//python:repositories.bzl", "py_repositories", "python_register_toolchains")

py_repositories()

python_register_toolchains(
    name = "python_3_11_5",
    python_version = "3.11.5",
    tool_versions = {
        "3.11.5": {
            "url": "https://my.private.repository.com/python-3.11.5.tar.gz",
            "sha256": {
                "x86_64-unknown-linux-gnu": "fbed6f7694b2faae5d7c401a856219c945397f772eea5ca50c6eb825cbc9d1e1",
            },
            "strip_prefix": "python",
        },
    },
)
...

Verify successful fetch and build:

$ bazel build @python_3_11_5//...
INFO: Invocation ID: a9edaaf4-56b6-41e7-a956-624ec4641223
INFO: Analyzed 8 targets (1 packages loaded, 22 targets configured).
INFO: Found 8 targets...
INFO: Elapsed time: 0.672s, Critical Path: 0.00s
INFO: 3 processes: 3 internal.
INFO: Build completed successfully, 3 total actions

@LINKIWI
Copy link
Contributor Author

LINKIWI commented Sep 21, 2023

I briefly looked at the new credentials helper system and I do think it would work for my use case with some user-layer (i.e. external to rules_python and Bazel) logic. I haven't experimented with it yet though since it only landed into an LTS/stable cut recently.

More generally though I think there will be a long tail of migrations to the credentials helper pattern--having netrc support in rules_python by default would be nice in the interim, for behavior parity with other OSS rulesets.

@aignas aignas added this pull request to the merge queue Sep 21, 2023
Merged via the queue into bazelbuild:main with commit 0d0e183 Sep 21, 2023
2 checks passed
renovate bot added a commit to bazel-contrib/rules_bazel_integration_test that referenced this pull request Oct 6, 2023
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [rules_python](https://togithub.com/bazelbuild/rules_python) |
http_archive | minor | `0.25.0` -> `0.26.0` |

---

### Release Notes

<details>
<summary>bazelbuild/rules_python (rules_python)</summary>

###
[`v0.26.0`](https://togithub.com/bazelbuild/rules_python/releases/tag/0.26.0)

[Compare
Source](https://togithub.com/bazelbuild/rules_python/compare/0.25.0...0.26.0)

#### Using Bzlmod with Bazel 6

**NOTE: bzlmod support is still beta. APIs subject to change.**

Add to your `MODULE.bazel` file:

```starlark
bazel_dep(name = "rules_python", version = "0.26.0")

pip = use_extension("@&#8203;rules_python//python/extensions:pip.bzl", "pip")

pip.parse(
    name = "pip",
    requirements_lock = "//:requirements_lock.txt",
)

use_repo(pip, "pip")
```

#### Using WORKSPACE

Paste this snippet into your `WORKSPACE` file:

```starlark
load("@&#8203;bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    sha256 = "9d04041ac92a0985e344235f5d946f71ac543f1b1565f2cdbc9a2aaee8adf55b",
    strip_prefix = "rules_python-0.26.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.26.0/rules_python-0.26.0.tar.gz",
)

load("@&#8203;rules_python//python:repositories.bzl", "py_repositories")

py_repositories()
```

##### Gazelle plugin

Paste this snippet into your `WORKSPACE` file:

```starlark
load("@&#8203;bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "rules_python_gazelle_plugin",
    sha256 = "9d04041ac92a0985e344235f5d946f71ac543f1b1565f2cdbc9a2aaee8adf55b",
    strip_prefix = "rules_python-0.26.0/gazelle",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.26.0/rules_python-0.26.0.tar.gz",
)

### To compile the rules_python gazelle extension from source,
### we must fetch some third-party go dependencies that it uses.

load("@&#8203;rules_python_gazelle_plugin//:deps.bzl", _py_gazelle_deps = "gazelle_deps")

_py_gazelle_deps()
```

#### What's Changed

- doc: Note Python version changes in CHANGELOG by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1391
- fix: bcr releaser email by
[@&#8203;f0rmiga](https://togithub.com/f0rmiga) in
[bazelbuild/rules_python#1392
- Adding kwargs to gazelle_python_manifest by
[@&#8203;linzhp](https://togithub.com/linzhp) in
[bazelbuild/rules_python#1289
- docs: Use correct link to build badge image and build status page. by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1390
- feat(py_console_script_binary)!: entry points with custom dependencies
by [@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1363
- fix(whl_library): avoid unnecessary repository rule restarts by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1400
- refactor: add missing `//python/config_settings/private:distribution`
target by [@&#8203;philsc](https://togithub.com/philsc) in
[bazelbuild/rules_python#1402
- Import pycross_wheel_library by
[@&#8203;philsc](https://togithub.com/philsc) in
[bazelbuild/rules_python#1403
- refactor: upgrade certifi by
[@&#8203;cflewis](https://togithub.com/cflewis) in
[bazelbuild/rules_python#1397
- fix: don't set distribs in version transitioning rule by
[@&#8203;comius](https://togithub.com/comius) in
[bazelbuild/rules_python#1412
- fix(gazelle): upgrade rules_go: 0.39.1 -> 0.41.0 to work with upcoming
Bazel versions by [@&#8203;sgowroji](https://togithub.com/sgowroji) in
[bazelbuild/rules_python#1410
- fix: gazelle: Fix non-hermetic runfiles lookup by
[@&#8203;fmeum](https://togithub.com/fmeum) in
[bazelbuild/rules_python#1415
- feat: create toolchain type for py_proto_library by
[@&#8203;comius](https://togithub.com/comius) in
[bazelbuild/rules_python#1416
- internal: copy Starlark rule implementation from Bazel by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1418
- feat: add new Python toolchain versions by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1414
- internal(pystar): make starlark impl (mostly) loadable by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1422
- feat: generate py_library per file by
[@&#8203;raylu](https://togithub.com/raylu) in
[bazelbuild/rules_python#1398
- chore: bump default python versions by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1425
- feat: Support netrc-based authentication for python_repository rule by
[@&#8203;LINKIWI](https://togithub.com/LINKIWI) in
[bazelbuild/rules_python#1417
- refactor(pystar): load (but don't use) Starlark implementation. by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1428
- fix(gazelle): runfiles discovery by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1429
- feat, refactor(pystar): bzl_library for packaging.bzl; fix pystar doc
building and py_wheel by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1432
- refactor(toolchain): use a helper method to convert an X.Y version to
X.Y.Z by [@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1423
- pycross: Rename `pycross_wheel_library` and make it work by
[@&#8203;philsc](https://togithub.com/philsc) in
[bazelbuild/rules_python#1413
- fix: Skip printing unneccesary warning. by
[@&#8203;matts1](https://togithub.com/matts1) in
[bazelbuild/rules_python#1407
- refactor(bzlmod)!: simplify pip.parse repository layout by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1395
- feat(bzlmod): mark pip extension as os/arch dependent by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1433
- chore: bump internal_deps by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1322
- tests(pystar): CI configs that uses Starlark implementation of rules
by [@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1435
- internal(pystar): Copy @&#8203;bazel_tools//tools/python files to
rules_python by [@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1437
- internal(pystar): Make py_runtime_pair and autodetecting toolchain
mostly loadable. by [@&#8203;rickeylev](https://togithub.com/rickeylev)
in
[bazelbuild/rules_python#1439
- tests: Move base rule tests under tests instead of
//tools/build_defs/python by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1440
- tests(pystar): py_runtime_pair and py_runtime analysis tests by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1441
- fix(pystar): Use py_internal for runfiles_enabled,
declare_shareable_artifact, share_native_deps by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1443
- build(deps): bump urllib3 from 1.26.13 to 1.26.17 in
/examples/pip_repository_annotations by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[bazelbuild/rules_python#1447
- build(deps): bump urllib3 from 1.25.11 to 1.26.17 in
/examples/pip_install by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[bazelbuild/rules_python#1444
- fix: add missing `@bazel_tools` files to bzl_library dependencies. by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1457
- tests(pystar): add analysis tests to cover basic windows building by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1452
- docs: move dependency management into respective bzl packages by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1459
- feat(py_wheel): Normalize name and version by
[@&#8203;vonschultz](https://togithub.com/vonschultz) in
[bazelbuild/rules_python#1331
- chore: add new Python toolchains from indygreg by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1461

#### New Contributors

- [@&#8203;cflewis](https://togithub.com/cflewis) made their first
contribution in
[bazelbuild/rules_python#1397
- [@&#8203;sgowroji](https://togithub.com/sgowroji) made their first
contribution in
[bazelbuild/rules_python#1410
- [@&#8203;raylu](https://togithub.com/raylu) made their first
contribution in
[bazelbuild/rules_python#1398
- [@&#8203;LINKIWI](https://togithub.com/LINKIWI) made their first
contribution in
[bazelbuild/rules_python#1417

**Full Changelog**:
bazelbuild/rules_python@0.25.0...0.26.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/bazel-contrib/rules_bazel_integration_test).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4wLjMiLCJ1cGRhdGVkSW5WZXIiOiIzNy4wLjMiLCJ0YXJnZXRCcmFuY2giOiJtYWluIn0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

python_register_toolchains and python_repository do not accept netrc auth
2 participants