Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use bazel's downloader for downloading pip packages #1357

Open
cameron-martin opened this issue Aug 2, 2023 · 1 comment
Open

Use bazel's downloader for downloading pip packages #1357

cameron-martin opened this issue Aug 2, 2023 · 1 comment
Assignees
@aignas
Labels
type: feature request type: pip

Comments

@cameron-martin
Copy link

🚀 feature request

Relevant Rules

The relevant rule is whl_library.

Description

Currently whl_library downloads pip packages by executing a python script. However this bypasses bazel's downloader, causing us to lose a few nice features such as:

  • The ability to rewrite these urls to a mirror by using --experimental_downloader_config.
  • Sharing the downloader cache between machines to speed up downloads.

Describe the solution you'd like

It would be nice to use bazel's downloader for these, by using repository_ctx.download.

Describe alternatives you've considered

Not doing this and keeping it like it is, I suppose.
@aignas aignas mentioned this issue Oct 12, 2023
Merged
github-merge-queue bot pushed a commit that referenced this issue Oct 17, 2023
@aignas
refactor(whl_library): split wheel downloading and extraction into se…
915d7a0 
…parate executions (#1487)

Before the PR the downloading/building of the wheel and the extraction
would be done as a single step, which meant that for patching of the
wheel to happen, we would need to do it within the python script. In
order to have more flexibility in the approach, this PR splits the
process to two separate invocations of the wheel_installer, which
incidentally also helps in a case where the downloading of the wheel
file can happen separately via http_file.

Related issues #1076, #1357
@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

@github-actions github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Jan 29, 2024
@aignas aignas added type: feature request type: pip and removed Can Close? Will close in 30 days if there is no new activity labels Jan 29, 2024
michaelboulton added a commit to michaelboulton/rules_python that referenced this issue Mar 3, 2024
@michaelboulton
feat: Download platform agnostic wheels using rctx.download
9c097d5 
If we can determine that there's a platform agnostic wheel
(dep-ver-py3-none-any.whl) then download it using rctx.download,
allowing it to be cached.

This partially solves bazelbuild#1357 but it still doesn't handle dependencies
that have platform specific wheels like https://pypi.org/project/psycopg2/.
michaelboulton added a commit to michaelboulton/rules_python that referenced this issue Mar 3, 2024
@michaelboulton
feat: Download platform agnostic wheels using rctx.download
ff07ec4 
If we can determine that there's a platform agnostic wheel
(dep-ver-py3-none-any.whl) then download it using rctx.download,
allowing it to be cached.

This partially solves bazelbuild#1357 but it still doesn't handle dependencies
that have platform specific wheels like https://pypi.org/project/psycopg2/.
michaelboulton added a commit to michaelboulton/rules_python that referenced this issue Mar 3, 2024
@michaelboulton
feat: Download platform agnostic wheels using rctx.download
620a941 
If we can determine that there's a platform agnostic wheel
(dep-ver-py3-none-any.whl) then download it using rctx.download,
allowing it to be cached.

This partially solves bazelbuild#1357 but it still doesn't handle dependencies
that have platform specific binary wheels like https://pypi.org/project/psycopg2/
michaelboulton added a commit to michaelboulton/rules_python that referenced this issue Mar 3, 2024
@michaelboulton
feat: Download platform agnostic wheels using rctx.download
2babfab 
If we can determine that there's a platform agnostic wheel
(dep-ver-py3-none-any.whl) then download it using rctx.download,
allowing it to be cached.

This partially solves bazelbuild#1357 but
it still doesn't handle dependencies that have platform specific binary wheels
like https://pypi.org/project/psycopg2/
michaelboulton added a commit to michaelboulton/rules_python that referenced this issue Mar 3, 2024
@michaelboulton
feat: Download platform agnostic wheels using rctx.download
636fc40 
If we can determine that there's a platform agnostic wheel
(dep-ver-py3-none-any.whl) then download it using rctx.download,
allowing it to be cached.

This partially solves bazelbuild#1357 but
it still doesn't handle dependencies that have platform specific binary wheels
like https://pypi.org/project/psycopg2-binary/
@michaelboulton michaelboulton mentioned this issue Mar 3, 2024
Closed
@aignas aignas mentioned this issue Mar 10, 2024
Closed
2 tasks
@aignas aignas self-assigned this Mar 26, 2024
@aignas aignas mentioned this issue Mar 29, 2024
Closed
github-merge-queue bot pushed a commit that referenced this issue Apr 5, 2024
@aignas
feat(bzlmod): support bazel downloader when downloading wheels (#1827)
4a615be 
This introduces 3 attributes and the minimal code to be able to download
wheels
using the bazel downloader for the host platform. This is not yet adding
support for targeting a different platform but just allows us to get the
wheels
for the host platform instead of using `pip`.

All of this is achieved by calling the PyPI's SimpleAPI (Artifactory
should work
as well) and getting the all URLs for packages from there. Then we use
the `sha256`
information within the requirements files to match the entries found on
SimpleAPI
and then pass the `url`, `sha256` and the `filename` to `whl_library`,
which uses
`repository_ctx.download`.

If we cannot find any suitable artifact to use, we fallback to legacy
`pip` behaviour.

Testing notes:
* Most of the code has unit tests, but the `pypi_index.bzl` extension
could have more.
* You can see the lock file for what the output of all of this code
would be on your
  platform.
* Thanks to @dougthor42 for testing this using the credentials helper
against a private
  registry that needs authentication to be accessed.

Work towards #1357
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aignas aignas

Labels
type: feature request type: pip
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aignas @cameron-martin