Package Ray

[h-vetinari]

Checklist

• Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml"
• License file is packaged (see here for an example)
• Source is from official source
• Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged)
• If static libraries are linked in, the license of the static library is packaged.
• Build number is 0
• A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details)
• GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there

This tries to pick up where #10637 of @dHannasch left off. I've added him to the recipe maintainers due to his PR, but can remove him if he wants to.

I've tried to work independently up until this point, but I've hit a pretty big road block where I please need guidance @conda-forge/core:

The upstream build script is a ~1000 line bazel build. It seemed too daunting a task to rewrite this in a build.sh, and so I wanted to use the script itself.

Not knowing much of bazel until today, I found out the hard way that it pulls in a lot of different things, which we would therefore also have to package (and link) correctly. I'm not sure how to go about this - the most elegant way I could think of would be to be able to specify conda-packages in the bazel script, but that doesn't seem to be possible currently: bazelbuild/rules_python#73.

Some notes:

• I want to start by building 0.8.0 (instead of the more current 0.8.4) because Add recipe for Modin #8471 Add recipe modin #11161 needs this exact pin.
• While upstream bazel supports windows, conda doesn't yet: Windows support bazel-feedstock#36, try building for windows bazel-feedstock#37
• I think the multi-output build is an elegant way to solve:
  • the naming problem from Add ray. #10637 (ray exists in bioconda already)
  • mapping the different install variants to conda packages
• I started packaging some of the missing dependencies (e.g. package googlesearch #11155, which becomes a hard dep by 0.8.4), although much remains to be done. Also: add openai/gym #11162 package blist #11163
• The patches are not meant to be definitive. I'm just trying to get things to work.

Closes #10637 ray-project/ray#585

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/ray) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipes/ray:

• It looks like the 'ray-base' output doesn't have any tests.
• It looks like the 'ray-debug' output doesn't have any tests.
• Recipe with the same name exists in bioconda: please discuss with @conda-forge/bioconda-recipes.

[vnlitvinov]

Are the licenses for redis included?

I did the grep-ing over Bazel configuration files and included licenses for anything that was potentially included in the resulting bits (omitting several repos with Bazel rules only). Should be fine now.

[vnlitvinov]

Okay, licenses added (and I've reworked the patch that removes intallation requirements - those are handled by meta.yaml - to make sure we do not miss changes there, or at least get warnings).
Should be ready for review again.

cc @beckermr @conda-forge/core

[h-vetinari]

Overall: great work! 🙃

[h-vetinari]

PS. With all the work you've put into this, I consider you the "owner" of this PR - I'll refrain from directly committing unless you ask for support (in which case I'll try, but cannot promise anything in terms of turn-around times).

[vnlitvinov]

@beckermr @conda-forge/core @h-vetinari it should be again ready for review, this time with working ray-rllib package (and with more sturdy patching for requirements - if upstream changes them we'll get a patch conflict instead of runtime error).

Or I rather hope it would be ready, as it's still about 4-4.5 hours until Windows part of CI is done (if all goes well). CI is green again, yay!

[h-vetinari]

Very cool! 👍

(cannot approve my "own" PR in GH, but: LGTM)

[beckermr]

@kkraus14 Did your contact have any comments here or should we go ahead?

[kkraus14]

@kkraus14 Did your contact have any comments here or should we go ahead?

I never got a response, so full steam ahead.

[beckermr]

Thanks @kkraus14!

[h-vetinari]

Amazing! Been quite a journey since @dHannasch started this effort almost a year ago...

Thanks everyone for the help, and especially to @vnlitvinov for the herculean effort to get this over the finish line! 🥳

[bollwyvl]

Congratulations 🎉🎊👏!

[dHannasch]

This is fantastic! Sorry I disappeared. This is amazing work.