REP-002: Recipe Repositories (2019-2020 Roadmap)

[nerdvegas]

This REP outlines the work necessary to enhance Rez so that it is able to install a package and its dependencies from public rez package repositories.

Motivation

A major goal of rez is the ability to install packages from a public repository. Currently, studios that use rez often need to spend significant resources "rezifying" existing third party packages, so they can be integrated into their pipeline.

Currently, due to the programmatic nature of rez package definitions, it's already possible to author a package that installs third party software, and this could be shared from a public repo (on GitHub for eg). For a example, see this python package. However, there are significant limitations to this approach currently:

• There is no dependency tracking. Ie, a built package's requirements are not installed also. If its requirements don't already exist in a rez package repository local to the studio, then the build will fail.
• There is no build configuration tracking. Ie, it is not possible to build multiple variants of a package that have been built in different ways (eg, debug vs release).
• There is no way to consume packages in a predictable way wrt build configuration. For example, even if we used stub packages to represent the build mode of a package, there still is not a reliable way to guarantee that the correct variant is chosen in a resolved environment.

Goals

The goal of this REP is to overcome these shortcomings and to provide a tool and API for installing a rez package from a public repository (a so-called "recipe repository").

A practical aspect of this goal is to provide the entirety of the VFX Reference Platform as a rez repository, so that any given package in the platform can be built easily, and its build configuration managed easily.

It is not the goal of this REP to provide the ability to automatically download public repositories (ala pip etc) nor to provide support for artifact repositories, where packages have already been built and their binaries are downloaded instead. While it may make sense to extend to this functionality, they should be introduced in separate REP(s). An installation workflow that requires downloading a public package repository first, then building packages from source, already achieves the goals described in this REP. Furthermore, this already represents a significant amount of work.

Nomenclature

• Let "recipe repository" refer to repositories of rez packages that have not yet been built. This is in contrast to existing rez "package repositories", which are repos of built, reusable packages that have already been released in a given studio.
• Let "recipe" refer to a package definition from a recipe repository.

Example

Here is a (simplified) example of what the process may look like. In it, we:

• Download the public rez recipes repository;
• Make it visible to rez;
• Perform an installation of openvdb, specifying that we want a debug build, and its maya plugin to be present.

~/recipes]$ git clone git@github.com:nerdvegas/rez-recipes.git
~/recipes]$ export REZ_RECIPES_PATH=$(pwd):$REZ_RECIPES_PATH
~/recipes]$ rez-install openvdb --features debug==1 maya==1
...
Installed 2 packages; 12 requirements were already installed
Installed openexr-2.2.1: ~/packages/openexr/2.2.1/_v/a
Installed openvdb-6.1.2: ~/packages/openvdb/6.1.2/_v/d
Already installed: boost-1.61.0: ~/packages/boost/1.61.0/_v/e
...

How It Works

A simplified description of how this works:

• Packages in recipe repositories dynamically construct a variant to install, based on the options passed to rez-install.
• "Features" are a new feature - a way for packages to describe things that they have. Features can be listed in variants, and users or packages requesting other packages, can also request their features. In the example above, the user is requesting a debug build of openvdb, that contains the maya plugin. The openvdb recipe responds to this - it detects these feature requests, and dynamically constructs a variant definition to build, which includes these features. If the recipe detects an invalid or missing mandatory feature, it can halt the installation.
• A recipe does not have to expose all its build settings as features - this is up to the recipe author.
• The recipe is able to pass the selected features onto its build system. For example, it might set an env-var that its CMakeLists.txt will read, or enable a cmake option via the command line.

Example Recipe

Here is a simplified (and deliberately incomplete, in the interests of succinctness) example of what an openvdb recipe might look like.

name = "openvdb"

version = "6.1.2"

# here for reuse. In rez, functions in the package.py that aren't formal rez attribs 
# are stripped on install
def _get_debug():
    if ".{this.name}.feature.debug" in request:
        if intersects(request[".{this.name}.feature.debug"], "1"):
            return 1
        elif intersects(request[".{this.name}.feature.debug"], "0"):
            return 0
        else:
            stop("Invalid 'debug' feature, choose 0 or 1")
    else:
        return 0  # default

@early()
def variants():
    # common requirements, using new requirements expansion syntax
    result = [
        "boost-1.61+//harden",
        "ilmbase-2.2+//harden",
        "openexr-2.2+//harden",
        "!openexr-2.2.5"  # avoid something funky
    ]

    # build mode option
    debug = _get_debug()
    result.append(".{this.name}.feature.debug==%d" % debug)  # adds feature to variant

    # openvdb requires debug boost if it's building in debug mode (let's just say)
    if debug:
        # like all requests, packages can use them just as users can
        result.append(".boost.feature.debug==1")

    return [result]

def build_target_commands():
    if _get_debug():
        add_build_args("-DOPENVDB_DEBUG=1")

To explain:

• variants is an early bound attribute. This is so the variant's requirements can change depending on the request (which includes package features).
• The package will be installed if its dynamically generated variant has not already been installed at the studio (ie the same package with a variant of the same requirements is not already somewhere on REZ_PACKAGES_PATH).
• The package requires boost version 1.61 or greater, however that requirement needs to be baked out to the full boost version when the package is installed (this is the role of the harden directive).
• Once in the build environment, the build_target_commands function is used to add some args to the underlying build system. In this case, it's setting some cmake options that match the package features that were requested by the user.
• Note: the .pkg.feature.debug syntax will make more sense further down.

What Do Installed Packages Look Like?

A package installed via rez-install (as opposed to one installed via rez-build or rez-release) is identical to any other rez package. However, it always installs using the new hashed variants feature. This is because a rez-installed package might have any number of complex requirements present in its variants.

A hashed-variants- based package installation looks like so on disk:

<root>/mypkg/1.0.1/package.py
                  /83e0c415db1b602f9d59cee028da6ac785e9bacc  # variant 0...
                  /f071dd2646834b8ea5f41a9d2e2116ca0c415dbea  # variant 1...
                  /...
                  /7c34cff017bc4f97b1ebffcabd8eef3859cee028da6  # variant N

And like so in its package definition (here we show what openvdb might look like had we installed it in debug- and non-debug- modes)

name = "openvdb"

version = "6.1.2"

variants = [
    ["boost-1.61.1", "ilmbase-2.2.3", "openexr-2.2.3", ".openvdb.feature.debug==0"],
    ["boost-1.61.1", "ilmbase-2.2.3", "openexr-2.2.3", ".openvdb.feature.debug==1"]
]

Each variant subdirectory is a hash of that variant's requirements.

Related Tasks

Following is a breakdown of the work that needs to be done in order to support the features shown above. They are listed in approximate order - smaller tasks with less/no blocking tasks are listed first.

REP-002.001: Deprecate existing requirements expansion syntax [trivial]

The current requirements expansion feature has two problems:

• Expansion happens too early. It happens when the package definition is read, and before the build environment is resolved. For rez-install to work, expansion needs to happen after the build env is resolved, so that a resolve can be done on a recipe repository before any actual package builds occur.
• The syntax is not descriptive enough. For example, it is not possible to describe the case where you want a requirement on foo-1.2+, but want to harden the requirement to foo-*.*.

Deprecation warnings need to be added, and the documentation removed. This is to be replaced with the new "requirement harden directive" feature described below.

REP-002.002: Add intersects function [easy]

Package features are in fact just a special kind of package request (more on this later). As such, their value is a version range, although typically they're just set to exact values (eg ==0 or ==1). In order for a package recipe to alter its definition based on feature requests, it needs to be able to test them for their value. For example, intersects(request[".{this.name}.feature.debug"], "1") would equate to true if the "debug" feature for the current package is present in the request, and intersects with the request "1" (eg, as rez-install foo --features debug==1 would do).

An interesting note is whether you should use intersects to test objects in the request, or in the resolve. Which you should use depends on where you are in a package definition. For example, in any early-bound package attribute, only the request is available (by definition, early-bound attributes only exist in unbuilt packages). In a commands function, conversely, both the request and resolve objects are available. However, the resolve would typically be used in this case. Otherwise, the package would effectively ignore options requested by other packages in the resolve. For example, rez-env foo could pull in packages foo-1.0.0 and bah-2.3.0, but foo-1.0.0 may have listed .bah.cli==0 as a requirement. However, if bah's commands tests the request and not the resolve, it won't see its cli option being set to zero.

Note that it would be very common to test both that a package is in the request/resolve, and that it intersects with a given version range. In fact the code snippet given just earlier isn't quite correct - it would result in a KeyError if ".{this.name}.feature.debug" wasn't present in request at all. To deal with this, convenience functions such as request_intersects and resolve_intersects would almost certainly be provided. The raw intersects function would still be useful in some cases however. Consider:

def commands():
    if "foo" in this.requires and intersects(this.requires["foo"], "1"):
        # do something when curent package requires foo-1
        ...

REP-002.003: Add --env option to rez-build / rez-release / rez-install [easy]

The --env option would be used to add extra requests to the build environment of a package. For example, perhaps the package you're building has a requirement on boost, but it's also using the requirement hardening directive (eg boost-1.61//harden). You could then use --env boost-1.61.4 to restrain the build to a specific version of boost. Also, since package features are really just a special kind of package request (more on this later), --env can also be used to specify features (although there will almost certainly be a --features arg provided, to make this more intuitive - but the end result would be the same).

REP-002.004: Add build_target_commands package attribute [easy]

Currently there is no good way for a package being built to convey information about the build to its underlying build system (eg cmake). The package's commands function is irrelevant, because this is only used after an already-installed package has been resolved into an env, in order to configure the env (set env vars etc). This is often a point of confusion for new rez users.

This task aims to add a new build_target_commands function. This is a rex-based function (the same as the other commands functions), that is run post-build-env-resolve, and only on the package being built.

For example, here a "foo" package is checking for a debug option, and is setting an env-var which its cmake script might then read and respond to:

def build_target_commands():
    if intersects(request[".{this.name}.option.debug"], "==1"):
        env.FOO_DEBUG = 1

REP-002.005: Add new add_build_args command for use in build_target_commands [easy-medium]

This is a new command that would add args to the underlying build system's command line invocation. Following from the example above, the package may wish to set a cmake option directly instead:

def build_target_commands():
    if intersects(request[".{this.name}.option.debug"], "==1"):
        add_build_args(["-DFOO_DEBUG=1"])

REP-002.006: Add new requirements harden directive (replaces req exp) [medium]

This is a replacement for the current requirements expansion feature.

The new syntax uses the // delimiter, which will be set aside to use as a general "directive". Whenever a package request is followed by //, the following is always some function(*nargs, **kwargs) formed string, which has some custom effect on the request. For example, consider:

requires = [
        "boost-1.61+//harden(3)"
    ]

This directive will cause the requirement to behave like boost-1.61+ during the build environment resolve, but after the resolve, the request will be modified to (eg) boost-1.61.0. Internally, the directive might achieve this simply by attaching metadata to the request object, which would be used post-resolve, to alter the request (in this case, to "harden" it). With no args, the directive hardens to the exact package version (eg boost==1.61.0).

There are currently no plans for any further directives, but by establishing the meaning of the syntax now, we future proof things and allow for this as a possibility.

REP-002.007: Ephemeral packages [medium]

An "ephemeral package" is a package that does not exist. It can be used to represent abstract things - for example, the host platform, or a build setting that a package was built with. Ephemeral package names always start with .. Since they don't exist, they never resolve to a specific version (like packages do - eventually an actual package version from a repository is chosen). But besides this, they act somewhat like real packages do.

For example, if pkgA-1.0.0 requires .unreal-1, and pkgB-2.0.0 requires .unreal-2, then rez-env pkgA-1.0.0 pkgB-2.0.0 will fail to resolve, and will cite a version conflict on .unreal as the reason why.

Ephemeral packages can also be thought of simply as a way to pass information to packages. Consider the following package commands:

name = "foo"
version = "1.0.1"

def commands():
    env.PYTHONPATH.append("{root}/python")
    if not intersects(resolve[".{this.name}.cli"], "0"):
        env.PATH.append("{root}/bin")

This package uses an ephemeral package as a switch that can be used to disable its command line tools. To do so one would:

]$ rez-env foo-1.0.1 .foo.cli==0

A later task introduces a different, more intuitive syntax, that equates to the same thing:

]$ rez-env foo-1.0.1/cli==0

When an ephemeral package is used this way, it is generally referred to as a package option.

REP-002.008: Package features [hard] [requires: 002.007]

A package feature is a special case of ephemeral packages. It is used to represent some property of a package, such as its build mode, or a specific compiler flag that was used in its build.

Unlike standard ephemeral packages, if a request is made for a package feature, then only packages with that feature can be selected in the resolve. To illustrate, consider the following two resolves:

]$ rez-env foo .foo.cli==1  # a package option request
]$ rez-env foo .foo.feature.debug==1  # a package feature request

The first resolve is just passing information to any packages that wish to read it (although we can assume that only the foo package will be interested in .foo.* ephemerals). What packages choose to do with this information is up to them - there are no guarantees (though if a package suggests that it accepts an option such as cli and then does something different with it, that might be considered a dick move).

The second resolve is restricting the resolve to only those foo packages matching feature debug==1. The solver is aware of package features, and enforces this behaviour. The way that a package tells rez that it provides this feature, is to include its ephemeral as a requirement. This is why the example openvdb package at the start of this document adds .openvdb.feature.debug==1 to its dynamically created variant.

Similar to standard ephemerals, a convenience syntax will also be introduced:

]$ rez-env foo>1:debug==1

REP-002.009: Rez-pip 'extras' [easy] [requires: 002.008]

The rez-pip tool does not currently support "extras". It should be straightforward to use package features to implement this.

For example, here's how a package would add a requirement when its extra is enabled:

early()
def variants():
    result = [...]
    if ".{this.name}.feature.extra.test" in request:
        results.extend([
            "pytest",
            ".{this.name}.feature.extra.test"
        ])
    return [result]

And here's how a package would require another package with extras enabled:

requires = [
    "pyfoo-1.2+<2:extra.dev"
]

One downside to using features to implement extras, is that a different variant of the package needs to be built and installed for every combination of extras possible. Furthermore, separate copies of the package's code aren't really necessary in python, because all extras do is add extra requirements. One enhancement we might consider is to somehow allow separate variants to share the same installed payloads. However, duplicated variants isn't a terribly big deal in the meantime, so this potential task is left outside of the scope of this REP.

REP-002.010: Add recipes_path config setting [trivial]

When using rez-install, a distinction will need to be made between recipe repositories and standard package repositories. Rez needs to know this in order to determine what packages must be built to satisfy a rez-install invocation - as opposed to which packages are already present in the studio's package repositories.

REP-002.011: Implement rez-install tool and API [medium] [requires: 002.010, 002.008, 002.003]

Implement the tool and API necessary for installing a package and any missing dependencies from one or more recipe repositories.

The process would be along the lines of:

• Perform a resolve on the package being installed
• The result of the resolve may contain one or more packages from recipe repositories (rather than standard package repositories)
• Iterate over all packages from recipe repositories, in reverse dependency order
• Perform a standard build and install on each of them, being sure to include the same set of extra requests (ie, features and options) present in the original rez-install call
• Summarise the results.

REP-002.012: Add convenience ephemeral request syntax [easy-medium]

This task would standardize on and implement the various convenience syntaxes mentioned earlier. Specifically:

• Package option: foo-1/bah-0 becomes foo-1, .foo.bah-0;
• Package feature: foo-1:bah-0 becomes foo-1, .foo.feature.bah-0

REP-002.013: Add "provides" feature [medium-hard]

In VFX there are often cases where a DCC ships with its own copies of other libraries (python is a classic example). Having rez resolve an environment that includes such a DCC, but also includes a standard installation of such a library, can be problematic.

To help manage this situation, it would be useful if a package could describe that it also provides another package as part of its own installation. This is another case where ephemerals come to the rescue - along with some changes to the solver.

To indicate that a package provides its own copy of another package, you would include it as a package feature:

requires = [
    ".{this.name}.feature.provides.python-2.7.3"
]

When the solver encounters a "provides" ephemeral, it would use it in lieu of its possibly already-resolved, real package counterpart. If a provided package ends up in the resolve, it plays no part in configuring the environment (the responsibility for that is left to the package - typically a DCC - doing the providing).

Note that the distinction between (eg) python-2.7.3 and python==2.7.3 as a provides statement is important. Whereas python-2.7.3 would take precedence over any python package within that range (including for eg python-2.7.3.1), python==2.7.3 would provide for that exact version only. If python-2.7.3.1 ended up in the resolve in this case, it would cause a conflict.

We will also need to consider what it means if >1 package provide the same package. Perhaps a configurable setting would cause a warning or error; if a warning, perhaps the first package encountered wins.

Interestingly, implementing "provides" as a package feature would allow you to select a package based on what it provides. I don't know if this would be useful or not, but you could request a package that provides a specific version of another package, like so:

]$ rez-env maya:provides.python-2.3.6

REP-002.014: Per-variant timestamps [hard]

A long-standing design flaw in rez is that packages can only be associated with one timestamp, despite the fact that variants can be added to it at different times. This means that it isn't actually possible to 100% accurately resolve to the correct set of variants that existed at a given time, for any package with more than one variant.

In practice this has not tended to be a big issue, because variants currently tend to all be built and installed around the same time - a package install usually involves installing all its variants. However, with rez-install making it much easier to build and install differing variants of a package when needed, one would expect to see more variation in the times at which a given package's variants are created. It follows then, that for timestamping to be sufficiently useful, we probably need timestamps to be correctly associated with variants, not packages.

This ticket has been marked hard, but that may or may not be pessimistic. I suspect that this change will touch a lot of code, and so it's hard to predict what all the side effects might be.

REP-002.015: Implement the VFX Reference Platform as recipes [hard]

To take full advantage of the combined new functionality of this REP, we should make all packages in the VFX reference platform available in a public recipe repository. Clearly this will be a lot of work and I think we'll need to split it amongst developers to get it done within a reasonable time frame.

Undoubtedly, going through this process will find cases that we may not have considered. Even though this task is listed last, it can probably be started in parallel with some of the tasks mentioned, to help find and fix shortcomings during development.

REP-002.016: Add "requires" feature [medium-hard]

A long-standing challenge in rez has been how to ensure that a specific variant of a package gets chosen in a resolve. Consider:

variants = [
    ["python-3"],
    ["python-3", "houdini-17"]
]

It is not necessarily true that a request including houdini-17 will result in the second variant being selected - the first variant doesn't conflict with that request, for example. There are rules to control the order in which a package's variants are preferred, but in large resolves there are many cases where variants are selected in a way that is unintuitive.

Consider then, if a feature was automatically created for every requirement in a package. For example, a python-3 requirement would also create the feature .{this.name}.feature.requires.python-3. Because features can be requested directly, we would then have a way to directly control variant selection.

For example, here's how we might select a variant of the foo package, which requires houdini, but not python-3:

]$ rez-env foo-1+:requires.houdini,!requires.python-3

We probably won't actually create these features per package requirement (due to overhead). However, "requirement features" would be a special case of feature that act as if this is the case.

REP-002.017: Support developer packages in repositories

Currently, developer packages (ie those where early bound attributes are expanded, among other things) are only supported for the case where a package is being directly built (typically via rez-build). In order for rez-install to work, developer packages must also work when present within a recipe repository.

It isn't yet know whether existing package repositories need to be updated to enable this casem or whether a new repository type should be created instead.

Challenges

There are some potential issues that require further thought, and whose solutions may not become obvious until more work is done.

• How might we express sensible defaults for package features? For example, if openvdb is available in debug and non-debug modes, and a request does not specify either, how do we make sure that the non-debug variant is chosen, unless another package specifically required its debug variant? Do we need to introduce the concept of "global" feature defaults, that might be specified like (eg) rez-env default:debug==0 ...?
• How do we ensure that packages follow some form of standardisation, to avoid confusion? For example, it might be confusing (and hard to set defaults) if one package used a debug feature (set to 0 or 1), but another had a build_mode feature (set to "debug" or "release").
• We can expect significantly more variants and requirements in packages in general, as a result of these new features. This, along with updates to solver behaviour, may result in slower resolve performance. It is possible that we will need to put higher priority on optimisations for this reason (whether these be based on things like a language port, or algorithmic changes, or execution model changes, or a combination of all of the above).
• It's possible that the "provides" mechanism is too simplistic. For example, what if a package is requested with certain features, and a "provided" package overrides it? Provided packages aren't able to express features, so this would always fail. Does that mean that we can't support provided packages with features? Do we need to? Would it be sufficient that the DCC package doing the providing, also own the feature for the provided package? Eg, perhaps maya-2019.1.0 actually has the feature python-2.7.3:debug==1, if it was a build of maya shipping with a debug build of python (just for argument's sake).