Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(bzlmod): mark pip extension as os/arch dependent #1433

Merged
merged 4 commits into from
Sep 28, 2023
Merged

feat(bzlmod): mark pip extension as os/arch dependent #1433

merged 4 commits into from
Sep 28, 2023

Conversation

aignas
Copy link
Collaborator

@aignas aignas commented Sep 27, 2023 •
edited
Loading

This ensures that under bzlmod with --lockfile-mode=update we
would generate an entry per os/arch, which is needed because the
hermetic toolchain interpreter path is os/arch dependent.

Summary:

  • add bazel_features dep
  • mark the pip extension as arch/os dependent

Related: bazelbuild/bazel#19154

aignas
aignas commented Sep 27, 2023
View reviewed changes
python/extensions/pip.bzl Outdated Show resolved Hide resolved
@aignas aignas force-pushed the feat/mark-extensions-os-dependent branch from 11189d1 to dc78494 Compare September 27, 2023 23:53
@aignas aignas marked this pull request as ready for review September 27, 2023 23:59
rickeylev
rickeylev approved these changes Sep 28, 2023
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
@aignas @rickeylev
Update CHANGELOG.md for consistency
565dd9a 
Co-authored-by: Richard Levasseur <richardlev@gmail.com>
@aignas aignas added this pull request to the merge queue Sep 28, 2023
Merged via the queue into bazelbuild:main with commit cdb5902 Sep 28, 2023
2 checks passed
@aignas aignas deleted the feat/mark-extensions-os-dependent branch September 28, 2023 12:33
@aignas aignas mentioned this pull request Sep 28, 2023
Closed
renovate bot added a commit to bazel-contrib/rules_bazel_integration_test that referenced this pull request Oct 6, 2023
@renovate
chore(deps): update dependency rules_python to v0.26.0 (#218)
45fc6b2 
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [rules_python](https://togithub.com/bazelbuild/rules_python) |
http_archive | minor | `0.25.0` -> `0.26.0` |

---

### Release Notes

<details>
<summary>bazelbuild/rules_python (rules_python)</summary>

###
[`v0.26.0`](https://togithub.com/bazelbuild/rules_python/releases/tag/0.26.0)

[Compare
Source](https://togithub.com/bazelbuild/rules_python/compare/0.25.0...0.26.0)

#### Using Bzlmod with Bazel 6

**NOTE: bzlmod support is still beta. APIs subject to change.**

Add to your `MODULE.bazel` file:

```starlark
bazel_dep(name = "rules_python", version = "0.26.0")

pip = use_extension("@&#8203;rules_python//python/extensions:pip.bzl", "pip")

pip.parse(
    name = "pip",
    requirements_lock = "//:requirements_lock.txt",
)

use_repo(pip, "pip")
```

#### Using WORKSPACE

Paste this snippet into your `WORKSPACE` file:

```starlark
load("@&#8203;bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    sha256 = "9d04041ac92a0985e344235f5d946f71ac543f1b1565f2cdbc9a2aaee8adf55b",
    strip_prefix = "rules_python-0.26.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.26.0/rules_python-0.26.0.tar.gz",
)

load("@&#8203;rules_python//python:repositories.bzl", "py_repositories")

py_repositories()
```

##### Gazelle plugin

Paste this snippet into your `WORKSPACE` file:

```starlark
load("@&#8203;bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "rules_python_gazelle_plugin",
    sha256 = "9d04041ac92a0985e344235f5d946f71ac543f1b1565f2cdbc9a2aaee8adf55b",
    strip_prefix = "rules_python-0.26.0/gazelle",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.26.0/rules_python-0.26.0.tar.gz",
)

### To compile the rules_python gazelle extension from source,
### we must fetch some third-party go dependencies that it uses.

load("@&#8203;rules_python_gazelle_plugin//:deps.bzl", _py_gazelle_deps = "gazelle_deps")

_py_gazelle_deps()
```

#### What's Changed

- doc: Note Python version changes in CHANGELOG by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1391
- fix: bcr releaser email by
[@&#8203;f0rmiga](https://togithub.com/f0rmiga) in
[bazelbuild/rules_python#1392
- Adding kwargs to gazelle_python_manifest by
[@&#8203;linzhp](https://togithub.com/linzhp) in
[bazelbuild/rules_python#1289
- docs: Use correct link to build badge image and build status page. by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1390
- feat(py_console_script_binary)!: entry points with custom dependencies
by [@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1363
- fix(whl_library): avoid unnecessary repository rule restarts by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1400
- refactor: add missing `//python/config_settings/private:distribution`
target by [@&#8203;philsc](https://togithub.com/philsc) in
[bazelbuild/rules_python#1402
- Import pycross_wheel_library by
[@&#8203;philsc](https://togithub.com/philsc) in
[bazelbuild/rules_python#1403
- refactor: upgrade certifi by
[@&#8203;cflewis](https://togithub.com/cflewis) in
[bazelbuild/rules_python#1397
- fix: don't set distribs in version transitioning rule by
[@&#8203;comius](https://togithub.com/comius) in
[bazelbuild/rules_python#1412
- fix(gazelle): upgrade rules_go: 0.39.1 -> 0.41.0 to work with upcoming
Bazel versions by [@&#8203;sgowroji](https://togithub.com/sgowroji) in
[bazelbuild/rules_python#1410
- fix: gazelle: Fix non-hermetic runfiles lookup by
[@&#8203;fmeum](https://togithub.com/fmeum) in
[bazelbuild/rules_python#1415
- feat: create toolchain type for py_proto_library by
[@&#8203;comius](https://togithub.com/comius) in
[bazelbuild/rules_python#1416
- internal: copy Starlark rule implementation from Bazel by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1418
- feat: add new Python toolchain versions by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1414
- internal(pystar): make starlark impl (mostly) loadable by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1422
- feat: generate py_library per file by
[@&#8203;raylu](https://togithub.com/raylu) in
[bazelbuild/rules_python#1398
- chore: bump default python versions by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1425
- feat: Support netrc-based authentication for python_repository rule by
[@&#8203;LINKIWI](https://togithub.com/LINKIWI) in
[bazelbuild/rules_python#1417
- refactor(pystar): load (but don't use) Starlark implementation. by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1428
- fix(gazelle): runfiles discovery by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1429
- feat, refactor(pystar): bzl_library for packaging.bzl; fix pystar doc
building and py_wheel by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1432
- refactor(toolchain): use a helper method to convert an X.Y version to
X.Y.Z by [@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1423
- pycross: Rename `pycross_wheel_library` and make it work by
[@&#8203;philsc](https://togithub.com/philsc) in
[bazelbuild/rules_python#1413
- fix: Skip printing unneccesary warning. by
[@&#8203;matts1](https://togithub.com/matts1) in
[bazelbuild/rules_python#1407
- refactor(bzlmod)!: simplify pip.parse repository layout by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1395
- feat(bzlmod): mark pip extension as os/arch dependent by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1433
- chore: bump internal_deps by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1322
- tests(pystar): CI configs that uses Starlark implementation of rules
by [@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1435
- internal(pystar): Copy @&#8203;bazel_tools//tools/python files to
rules_python by [@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1437
- internal(pystar): Make py_runtime_pair and autodetecting toolchain
mostly loadable. by [@&#8203;rickeylev](https://togithub.com/rickeylev)
in
[bazelbuild/rules_python#1439
- tests: Move base rule tests under tests instead of
//tools/build_defs/python by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1440
- tests(pystar): py_runtime_pair and py_runtime analysis tests by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1441
- fix(pystar): Use py_internal for runfiles_enabled,
declare_shareable_artifact, share_native_deps by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1443
- build(deps): bump urllib3 from 1.26.13 to 1.26.17 in
/examples/pip_repository_annotations by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[bazelbuild/rules_python#1447
- build(deps): bump urllib3 from 1.25.11 to 1.26.17 in
/examples/pip_install by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[bazelbuild/rules_python#1444
- fix: add missing `@bazel_tools` files to bzl_library dependencies. by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1457
- tests(pystar): add analysis tests to cover basic windows building by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1452
- docs: move dependency management into respective bzl packages by
[@&#8203;rickeylev](https://togithub.com/rickeylev) in
[bazelbuild/rules_python#1459
- feat(py_wheel): Normalize name and version by
[@&#8203;vonschultz](https://togithub.com/vonschultz) in
[bazelbuild/rules_python#1331
- chore: add new Python toolchains from indygreg by
[@&#8203;aignas](https://togithub.com/aignas) in
[bazelbuild/rules_python#1461

#### New Contributors

- [@&#8203;cflewis](https://togithub.com/cflewis) made their first
contribution in
[bazelbuild/rules_python#1397
- [@&#8203;sgowroji](https://togithub.com/sgowroji) made their first
contribution in
[bazelbuild/rules_python#1410
- [@&#8203;raylu](https://togithub.com/raylu) made their first
contribution in
[bazelbuild/rules_python#1398
- [@&#8203;LINKIWI](https://togithub.com/LINKIWI) made their first
contribution in
[bazelbuild/rules_python#1417

**Full Changelog**:
bazelbuild/rules_python@0.25.0...0.26.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/bazel-contrib/rules_bazel_integration_test).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4wLjMiLCJ1cGRhdGVkSW5WZXIiOiIzNy4wLjMiLCJ0YXJnZXRCcmFuY2giOiJtYWluIn0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@meteorcloudy
Copy link
Member

Thanks for addressing this issue in rules_python. But I'm just wondering is there any specific reason why rules_python couldn't generate repos for all supported platforms and select the correct repo during build time? This is what the java toolchains do and could avoid using use_os/use_arch.

One problem I found is that there is a bit of a usability issue of use_os/use_arch: if we enforce --lockfile_mode=error on CI build, a user who updates rules_python has to update the lockfile on each supported platforms to pass check. This is sometimes hard because some users may not have access to specific machines.

@aignas
Copy link
Collaborator Author

aignas commented Oct 20, 2023

Hi, I agree that the ergonomics of this feature is not ideal because of this reason. The way rules_python is implemented today it needs to resolve the Python interpreter for the host platform and pass that to the repository rule that then uses that to generate the BUILD.bazel content and/or to build a python wheel. I think it would be great to write an overview doc and outline possible ways to move forward as there are many moving parts and getting rid of this platform dependence is not trivial.

@meteorcloudy meteorcloudy mentioned this pull request Nov 8, 2023
Closed
copybara-service bot pushed a commit to bazelbuild/bazel that referenced this pull request Nov 8, 2023
@meteorcloudy @tkiela1
Enforce lockfile update
a7b1d36 
- Upgraded rules_python to 0.26.0 to include bazelbuild/rules_python#1433
- Updated the lockfile on all supported platforms.
- Enable `--lockfile_mode=update` for regular builds (mostly affecting local build)
- Enable `--lockfile_mode=error` for CI builds to ensure lockfile is up-to-date.

After this change, please make sure to update the lockfile by running `bazel mod deps` whenever you change the MODULE.bazel file or any .bzl file used in it.

Closes #20101.

PiperOrigin-RevId: 580553351
Change-Id: I54c3298e6c23c8392226e1e32ba203689f334226
meteorcloudy added a commit to meteorcloudy/bazel that referenced this pull request Nov 9, 2023
@meteorcloudy
[7.0.0] Enforce lockfile update
0e41839 
- Upgraded rules_python to 0.26.0 to include bazelbuild/rules_python#1433
- Updated the lockfile on all supported platforms.
- Enable `--lockfile_mode=update` for regular builds (mostly affecting local build)
- Enable `--lockfile_mode=error` for CI builds to ensure lockfile is up-to-date.

After this change, please make sure to update the lockfile by running `bazel mod deps` whenever you change the MODULE.bazel file or any .bzl file used in it.

Closes bazelbuild#20101.

PiperOrigin-RevId: 580553351
Change-Id: I54c3298e6c23c8392226e1e32ba203689f334226
@meteorcloudy meteorcloudy mentioned this pull request Nov 9, 2023
Merged
meteorcloudy added a commit to bazelbuild/bazel that referenced this pull request Nov 9, 2023
@meteorcloudy
[7.0.0] Enforce lockfile update (#20121)
69b861c 
- Upgraded rules_python to 0.26.0 to include
bazelbuild/rules_python#1433
- Updated the lockfile on all supported platforms.
- Enable `--lockfile_mode=update` for regular builds (mostly affecting
local build)
- Enable `--lockfile_mode=error` for CI builds to ensure lockfile is
up-to-date.

After this change, please make sure to update the lockfile by running
`bazel mod deps` whenever you change the MODULE.bazel file or any .bzl
file used in it.

Closes #20101.

PiperOrigin-RevId: 580553351
Change-Id: I54c3298e6c23c8392226e1e32ba203689f334226
@dtolnay dtolnay mentioned this pull request Dec 28, 2023
Merged
@willjschmitt willjschmitt mentioned this pull request Aug 7, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fmeum fmeum fmeum left review comments

@rickeylev rickeylev rickeylev approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@aignas @meteorcloudy @fmeum @rickeylev