Support boa recipe format

[wolfv]

Your question:

I would like to start a discussion to see if we can support the new recipe format of boa and use it directly for builds. We've been using boa in the RoboStack project quite successfully for a while now and I think it would be nice to (optionally!) support the new recipe format in conda-forge as well.

I am listing a couple of points that we'll need to fix & improve in the boa & conda-forge projects to turn this into reality:

• boa needs to create correct and compatible hashes. To this point I would actually like to do things slightly different vs. conda-build. conda-build deeply inspects all build scripts to find used variables using regex's. I don't think that's a good idea and we already have very good hash input from the build & host environment + CONDA_BUILD_SYSROOT + channel_targets + target_platform (the latter three are also hard-coded in conda-build, but scattered in multiple places).
• conda-smithy needs support for parsing recipe.yaml instead of meta.yaml (this should be quite straightforward as it's always "pure yaml")
• the autotick bot would also need to be aware of the new recipe format

I'd be happy to hear what others think! Obviously, we'll also have to work more on documentation of the entire boa project.

[hmaarrfk]

Can you remind us of the key advantage of boa recipe formats?

[jaimergp]

I think this is a good starting point.

[beckermr]

A few thoughts:

1. contemplating the needed bot changes makes my head hurt a bit.
2. I'd rather handle 1 once we have a spec recognized by the community as a whole.

Maybe the next step here is to get agreement on the new spec on the conda community calls (maybe a CEP) and then move on from there?

[jezdez]

This sounds exciting generally speaking, the docs linked by Jaime above are really worth reading and make a ton of sense to me. Especially since you've already gained experience with the format in robostack, it seems as if cross-community adoption would be achievable via the CEP format.

+1 on submitting this as a CEP.

[wolfv]

Well the way this started out was with an attempt to make a spec with lots of community input: https://hackmd.io/axI1tQdwQB2pTJKt5XdY5w

Unfortunately besides myself not many people have invested more time into this :) I would hope that changes once we make some additional inroads in conda-forge (and once you try it I don't think you want to go back, except for complicated recipes).

We have a spec implementation using pydantic in the repository: https://github.com/mamba-org/boa/blob/master/boa/schemas/model.py -- might need a bit more love but the general stuff is there.

I can write a more formal CEP but I'd like to also encourage people to try out what's already there.

[jakirkham]

It might be worth picking a few recipes in conda-forge and opening draft PRs showing what kind of changes would occur. Maybe with recipes of increasing complexity or based on key recipe traits? This would hopefully help people wrap their heads around what is involved

Does boa have something similar to conda render (at least in terms of API)? This would be needed for conda-smithy to re-render

[wolfv]

@jakirkham excellent idea! I just started with my all time favorite recipe of xtensor which is pretty simple: https://github.com/conda-forge/xtensor-feedstock/pull/152/files

As you can see the changes are fairly minimal. After that, the file is instantly parseable with a YAML parser which makes a lot of things simpler -- for example, if we ever switch to a new recipe spec, we can just translate all recipes by machine.

I think it will be a cool exercise to get conda-smithy to recognize this and produce the proper pipeline, so I might tackle that next in order to get us to see the builds working.

[wolfv]

@beckermr

1. contemplating the needed bot changes makes my head hurt a bit.

I've started looking a bit into the required interfaces in conda-smithy and I think it might not be too bad -- we're anyways "duck-typing" the same conda_build.Metadata object to pass it into conda-build at certain stages so that it can take care of the postprocessing, and we use conda_build to load the conda_build_config.yaml file.

[wolfv]

Actually, it turned out to be not that many changes necessary to get an initial version of conda-smithy to work & recognize boa recipes since we already duck-type the Metadata class: conda-forge/conda-smithy#1583

The xtensor feedstock just started building with boa (obv. a bit hacky) and one interesting thing we need to fix in boa is to consider even variables used in skip expressions as part of the variant. So that win and vc<14 works -- will have to see how to deal with that!

[beckermr]

I suspect the bot will need more extensive changes due to how it handles the recipes in fairly adhoc ways.

[wolfv]

On the bright side all that should be much simpler since we don't need to wrestle so much jinja :) I'll take a stab at it at some point soon.

[marcelotrevisani]

@wolfv I am very interested in this, especially to add it to souschef. I was thinking to the user to be able to handle the recipe without knowing which "version" they will export, just after all the modifications and the developer can chose if it is going to export to the "normal" conda recipe or to this new version

I think that would be also useful in the case that someone wants to convert from the current version to this new one.

I remember that you sent an url where the new recipes was being defined, can you share that again please?

[marcelotrevisani]

the only thing I would recommend is to have like a key value pair to explicit mention that is a "new version"
maybe something like
recipe_version: 2 or mamba_recipe: 1
that would help when we need to pragmatically read and manipulate the recipe, and also to track new features and things that might be added to the new recipe, something similar we can find in docker files and such

[wolfv]

Yes, I agree about the versioning. Could be very useful to have. Right now, the obvious differentiator is the file name which makes it easy to spot but in order to future proof it would be good to version (although we could again do it via filename, like recipe.v2.yaml etc.). Not sure what people will find less annoying. Maybe just a regular yaml key will be fine.

Support in souschef would be awesome, for sure. I wrote some mediocre parser & converter from meta.yaml to recipe.yaml but using souschef would likely improve the result. The new format is really pleasant to work with. Let me know if you want to give it a stab!

[jakirkham]

Yeah a top-level key with the version seems useful from a parsing standpoint.

[beckermr]

Right. When this gets submitted as a CEP I suspect we'll version it there. Having a standard will make it easier to code interoperable tools like souschef, the bot, etc.

[wolfv]

I think we should have at least a toplevel key with the recipe version for the rendered recipe in the final package (for reproducibility), maybe even throw the boa & conda-build version in there :)

I am starting by writing additional documentation for the current recipe spec. I'll try to figure out wrt to CEP / CFEP on how to proceed. Not sure if CEP is the right place (since this is not exactly concerning "conda" per se (final conda package metadata is completely unchanged except for the recipe folder), more the conda ecosystem -- although I would be more than happy if conda-build adopts the same format) but I would still be happy to use either as a place for discussion.

Again, the really nice thing about the new format is that it's completely machine readable. That means that we can change around the spec super easily later on. For example, if we decide that instead of

test:
  requires:
    - ...

We want to add another key under requirements for test then we can do that with a simple yaml.load / yaml.dump.

requirements:
  host:
   - ...
  test:
   - ...

[wolfv]

PS: if people have / find issues with the current implementation or spec, you can always file issues on the boa repo. I'd love it if people give it a try!

[jaimergp]

Is there a "reverse" translator? recipe.yaml to meta.yaml? So conda-build can ingest this new format even without modifications; sort of a "legacy" mode.

[wolfv]

That would be simple to write! Just need to shuffle a couple of keys around and translate

context:
  version: 1.2.3
  name: myname
  full_name: {{ myname | lower }}

to {% set version = "1.2.3" %} etc

[wolfv]

Ah, and translate dictionaries with sel(win): ... to ... # [win] etc.

[dhirschfeld]

As a user who's just starting out with boa I'm excited to see this as I've found it to be very quick for building my packages! If it can be opt-in alongside the current meta.yaml that would make for a smooth transition.

I find the way multiple outputs are handled to be more intuitive and being machine-parsable is a big plus for some of our workflows (I'm only learning about souschef now! 🤦)

[BastianZim]

Just my two cents as a maintainer in the ecosystem as well:

I would love to have the new recipe as it would make a couple of things much simpler when creating packages:

1. I like the way it handles Preprocessing selectors as that makes generating them easier and updating them too. Especially interesting with Add CEP for optional dependencies in repodata.json conda/ceps#9
2. The Build time features would be awesome to have. I have some packages that can be compiled with different options and right now I need to create an additional output for them and copy/paste the entire build instructions. If I could just define the flags and then boa translates that into multiple outputs that would make life much easier.
   One example here is or-tools which can be compiled with different solvers. Since they all have different licences it is not feasible to include them all in one build. If I could just add the CMAKE flags and then each solver would be a different package, it would cut down development time and also allow us to offer more versions of a package. (Maybe also interesting with Providing user's license choice #1608)
3. The new Outputs would make it much simpler to create packages that have multiple names but are otherwise the same. Additionally, since only the extra name needs to be specified, it might be possible to automatically solve the - vs _ discussion by just adding that automatically in the backend, like the license_family key.
4. Using maybe a slightly adjusted Outputs it would also be finally possible to more easily include things like requests[socks] without having a huge recipe since only the new name and dependencies need to be included. This could maybe also get auto translated using slightly changed preprocessing selectors which would then get translated into multiple outputs during build.

[wolfv]

Thanks for all the input! I think we might want to take a two-tiered approach here :)

1. Get support for a conservative new-style recipe in conda-forge. This would basically be only parts that directly translate to and from meta.yaml and no "additional" features. Once we introduce optional dependencies we might have to adjust the bot in more difficult ways etc. The same goes for build-time features.
2. Allow some time to pass and the first couple of converted recipes to be successfully used
3. Specify how we really want to handle build-time features and optional dependencies and other necessary innovations!

For this, we'd add a --strict mode or something to boa that will disallow any of the cool (but more R&D) features in recipes and only allow the classic stuff. This will then hopefully ease the initial transition for the bots etc. and then we can see further, hopefully also with a clearer vision and better spec!

[beckermr]

I think a CEP is the right place. The conda community org covers both conda and conda build. Even if conda build doesn't implement the standard right away, having an accepted standard ensures that the community as a whole is on solid footing. It also allows others to build their own tools against the standard should they want to.