Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Cross compilation" of py_binary/py_image/py_library targets #260

Open
nimish opened this issue Nov 15, 2019 · 22 comments
Open

"Cross compilation" of py_binary/py_image/py_library targets #260

nimish opened this issue Nov 15, 2019 · 22 comments
Assignees
@aignas

Comments

@nimish
Copy link

nimish commented Nov 15, 2019

Hi,

I have a py_binary that depends on a python pip library (grpcio) that has a native extension bundled in. This means that to create a linux container i'd need to have the pip_import rule download the manylinux wheel, not the host one (macos in my case).

Is there a way to force this? Otherwise py_image will happily just bundle up wheels with darwin native libs. py_binary will also only make host-runnable things.
@surlyengineer surlyengineer mentioned this issue Nov 30, 2019
Closed
@ali5h
Copy link

ali5h commented Dec 16, 2019

use https://github.com/ali5h/rules_pip/ and use pip_install(["--platform=linux_x86_64"])

@nimish
Copy link
Author

nimish commented Dec 16, 2019

@ali5h this works (thanks!) but is there integration into the bazel platform selection functions? I don't want separate targets for Linux and Mac.

E.g. Something that works with https://docs.bazel.build/versions/master/platforms.html

@ali5h
Copy link

ali5h commented Dec 16, 2019

you can define multiple piplib repos for different platforms and use select to pickup correct one

@nimish
Copy link
Author

nimish commented Dec 16, 2019 •
edited
Loading

Is that documented and supported anywhere? That would be the ideal case, for bazel to automatically pick up the right pip repo for the right target platform.

E: should it not just be built in to the py_binary/py_library rule, to select the right target platform libs automatically?

@RemiTorracinta
Copy link

Would really, really like to see this as well.

@cwaeland cwaeland mentioned this issue Aug 19, 2020
Open
@github-actions
Copy link

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

@github-actions github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Apr 14, 2021
@thundergolfer thundergolfer mentioned this issue May 9, 2021
Closed
@thundergolfer thundergolfer removed the Can Close? Will close in 30 days if there is no new activity label May 9, 2021
@thundergolfer thundergolfer self-assigned this May 9, 2021
@thundergolfer thundergolfer mentioned this issue May 9, 2021
Closed
@brduarte
Copy link

brduarte commented Aug 6, 2021

Help

@meastham meastham mentioned this issue Aug 14, 2021
Closed
13 tasks
@UebelAndre
Copy link
Contributor

Seems #510 and #531 and addressing the same issue? I'd love to have that functionality so excited to so folks interested enough to open PRs 😄

Hopefully @alexeagle and @meastham can combine forces to get the awesome functionality over the line! 🚀

@UebelAndre
Copy link
Contributor

Friendly ping to @alexeagle and @meastham, are either of you able to take another look at your pull requests? I'd love to have that functionality 😃

@UebelAndre UebelAndre mentioned this issue Jan 6, 2022
Closed
12 tasks
@UebelAndre
Copy link
Contributor

I wonder if pipenv could be used to facilitate generating cross platform dependency graphs. Does anyone have experience with the tool?

@meastham
Copy link

meastham commented Jan 9, 2022 •
edited
Loading

Friendly ping to @alexeagle and @meastham, are either of you able to take another look at your pull requests? I'd love to have that functionality 😃

Sorry for the long delay on this! I did some investigation on some of the open questions, I'll see if we can find a path to getting this merged.

I don't think the PRs are exactly addressing the same issue. #531 allows having different requirements files for different platforms, but doesn't not allow downloading wheels for a different platform than the host platform. #510 allows downloading wheels for arbitrary platforms ("cross compiling"), but requires every platform to share one requirements file which has potential problems. Conceptually they could be compatible but the implementations would need to be modified to harmonize the way platform selection is done.

@kchodnicki kchodnicki mentioned this issue Jan 13, 2022
Closed
@jvolkman
Copy link
Contributor

I'm all for the wheel-only approach taken in #510. I don't want native compilation happening at all in the loading phase, as it can lead to hard-to-diagnose cache misses between machines (in my experience).

Thinking out loud here. If adding a wheel-only option, can we remove pip from the build-time process entirely? Instead of using pip download, why not just use bazel's built-in http tools to download the wheel? Instead of using pip-compile to generate a locked requirements.txt, write something that generates a .bzl file with compiled dependencies in a more bazel-centric fashion. Maybe resolvers from poetry or pipenv are used for this, which apparently support multi-platform resolve. Dependencies between libraries - including platform specific - could be explicit in the generated file and defined using bazel's own select or whatever.

I'm sure there are gotchas here (and significant work), but maybe detaching from pip could open up new avenues.

@alexeagle
Copy link
Collaborator

I think it should also be possible to compile python/C++ sources into wheels, however those need to happen in actions so they are debuggable and so that the target platform can be used.

@gopher-maker
Copy link

Ping on this @alexeagle and @meastham. #510 is exactly what I need today and having #531 would be a strong nice to have. Any progress here? Thanks!

@meastham
Copy link

Ping on this @alexeagle and @meastham. #510 is exactly what I need today and having #531 would be a strong nice to have. Any progress here? Thanks!

Hey @gopher-maker,

I'm just blocked on getting some guidance from a repository owner on how to resolve the outstanding issues with #510 (not a complaint btw, I'm sure everybody is quite busy!).

FWIW we've been using it for a fairly large Python codebase without significant problems for about 9 months now, so if you're feeling adventurous you can use it already. It looks like it now needs some non-trivial rebasing work; I'll see if I can get to that this week.

@laurynaslubys laurynaslubys mentioned this issue Apr 29, 2022
Closed
stephane-caron added a commit to upkie/vulp that referenced this issue May 27, 2022
@stephane-caron
Disable PyPI dependencies for the Raspberry Pi
7d82d40 
Related issue: bazelbuild/rules_python#260
@adeeb10abbas
Copy link

Any leads on this yet? I am running into a similar issue trying to use Mujoco in a bazel workspace.

@alexeagle
Copy link
Collaborator

@f0rmiga has been working on something related to this at a client, I don't have any update sorry.

@f0rmiga
Copy link
Collaborator

f0rmiga commented Mar 16, 2023 •
edited
Loading

There's a current effort that @philsc is writing in a doc, and has the collaboration from @jvolkman. I wrote a resolver to download Python packages using http_file. It's similar to how Gazelle does it for Go third-party deps. I'll start a draft PR in the coming days to have it maintained in rules_python. It will change quite a bit the current workflow to work with wheels in Bazel, and I don't have all the answers yet, so I don't foresee it being in a release soon.

@adeeb10abbas
Copy link

Ah, thanks for the answers folks! I will try to find some other workaround.

@pvcnt
Copy link

pvcnt commented Mar 18, 2023

@f0rmiga Could you please post the link to the PR when it's open in this thread? I'm very much interested in this also!

@aignas aignas mentioned this issue Nov 16, 2023
Closed
@aignas aignas mentioned this issue Dec 4, 2023
Closed
@aignas aignas mentioned this issue Dec 13, 2023
Open
@aignas
Copy link
Collaborator

aignas commented Dec 21, 2023

I am looking to fix this for bzlmod in #1643 at least for the whl downloading part, I'll leave this ticket to track the sdist cross-compilation.

@aignas aignas assigned aignas and unassigned thundergolfer May 13, 2024
@aignas aignas mentioned this issue Jun 1, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Jun 6, 2024
@aignas
feat(bzlmod): support cross-platform whl setups within the main hub r…
7de43d1 
…epo (#1837)

With this change we add support for platform-specific wheel registration
and
doing the selection of which wheel is used at build time.

This supports:
- Different package versions for different platforms.
- Use string_flags to configure what to fetch/select:
    - only whls, only sdist or auto mode.
    - libc version and `musl` vs `glibc` selection.
    - universal2 vs arch wheels for mac.
    - target osx version selection.

Summary of changes:
- The `uv pip install` would only warn the user of yanked packages but
  would not refuse to install them. Update our implementation to better
  match the same behaviour.
- A previous PR has added the support for passing it in the
`requirements_by_platform` and this just add the necessary code to make
  sure that we can also do the dependency management when parsing the
  `whl` `METADATA` files.
- Only configure `dev_pip` deps for `linux` and `osx` platforms to not
raise
  issues later.
- Add a function for generating a `whl_library` name from a `filename`.
- Add a macro for generating all config settings for a particular set of
parameters.
- Update `render_pkg_aliases` to also use those config settings.
- Update the toolchain selection `target_settings` to use the
`py_linux_libc`
config setting. With this the user can register a `musl` linux toolchain
if
  needed. We can also use similar `flag_values` to resolve #1876.
- Integrate everything in the `pip` extension and setup cross-platform
  select statements.

Work towards #1357, #260

Stacked on #1937
This was referenced Jun 12, 2024
Merged
Merged
github-merge-queue bot pushed a commit that referenced this issue Jun 17, 2024
@aignas
refactor(bzlmod): generate fewer targets for pip_config_settings (#1974)
fce7354 
Remove the `python_version` from `flag_values` and reduces the number of
targets we create by a lot because we no longer need to create all of
the micro
version combinations to also match when we want to match e.g. 3.10 on a
particular platform by using a trick I've learned in #1968:
```starlark
select({
    "is_python_3.x": "my_config_setting"
    "//conditions:default": "is_python_3.x"
})
```

Then further optimization was done on how we create the aliases and
config
settings and optimizing the usage of `pip_whl` flag finally reduced the
internal number targets. The number config settings targets in
`//tests/private/pip_config_settings/...` where we have multiple python
versions has changed:

* `:is_*` - from **995** to **996** (the extra addition being
is_python_default).
* `:_.*is_*` - from **28272** to **2480** (just python version) and then
to **496** (final).

Work towards #260
aignas added a commit to aignas/rules_python that referenced this issue Jun 22, 2024
@aignas
chore: remove dead code from whl_target_platforms
2dc4500 
Work towards bazelbuild#260
@aignas aignas mentioned this issue Jun 22, 2024
Merged
aignas added a commit to aignas/rules_python that referenced this issue Jun 22, 2024
@aignas
fix(bzlmod): pass only non-default values to whl_library
0826786 
Before this change `isolated`, `quiet` and `timeout` would get dropped
if they were specified to some non-default value. This fixes it.

Work towards bazelbuild#260.
@aignas aignas mentioned this issue Jun 22, 2024
Merged
@aignas
Copy link
Collaborator

aignas commented Jun 22, 2024

Remaining things to stabilize this feature for bzlmod:

  • Fix experimental_index_url fails to import TensorFlow package correctly in the build context #1996.
  • Instead of sha, use the sanitized full wheel name. Potentially lift some code from rules_pycross.
  • Use hub_name as the prefix for the whl_library repos. This means that we reuse the same whl_library for multiple python versions if that is configured.
  • Graduate the following flags from experimental:
    • experimental_index_url -> index_url and default it to https://pypi.org/simple. Users could still set it to empty string to fallback to the old behaviour.
    • experimental_index_url_overrides. This potentially should go to the pip.override tag class?
    • experimental_extra_index_urls
    • experimental_target_platforms. This should potentially be dropped from the public API and only the whl_library should have it.
  • Move around code doing all of the plumbing for PyPI to //python/private/pypi so that CODEOWNERS could be easily set.

Out of scope:

github-merge-queue bot pushed a commit that referenced this issue Jun 22, 2024
@aignas
chore: remove dead code from whl_target_platforms (#2001)
d003204 
Remove dead code only used in tests.

Work towards #260
github-merge-queue bot pushed a commit that referenced this issue Jun 22, 2024
@aignas
fix(bzlmod): pass only non-default values to whl_library (#2002)
0fd7b64 
Before this change `isolated`, `quiet` and `timeout` would get dropped
if they were specified to some non-default value. This fixes it.

Work towards #260.
github-merge-queue bot pushed a commit that referenced this issue Jun 28, 2024
@aignas
fix(bzlmod): only expose common packages via the requirements constan…
8d40b19 
…ts (#1955)

With this change the default `gazelle` instructions still work and users
do not need to worry about which package is present on which platform.

Work towards #260
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aignas aignas

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests