Skip to content

Basic Concepts

ajohns edited this page Nov 29, 2017 · 74 revisions

Overview

Rez manages packages. You request a list of packages from rez, and it resolves this request, if possible. If the resolution is not possible, the system supplies you with the relevant information to determine why this is so. You typically want to resolve a list of packages because you want to create an environment in which you can use them in combination, without conflicts occurring. A conflict occurs when there is a request for two or more different versions of the same package - a version clash.

Rez lets you describe the environment you want in a natural way. For example, you can say: “I want an environment with...”

  • “...the latest version of houdini”
  • “...maya-2009.1”
  • “...the latest rv and the latest maya and houdini-11.something”
  • “...rv-3.something or greater”
  • “...the latest houdini which works with boost-1.37.0”
  • “...PyQt-2.2 or greater, but less than PyQt-4.5.3”

In many examples in this documentation we will use the rez-env command line tool. This tool takes a list of package requests and creates the resulting configured environment. It places you in a subshell - simply exit the shell to return to a non-configured environment.

Versions

Rez version numbers are alphanumeric - they support any combination of numbers, letters and underscores. A version number is a set of tokens, separated by either dot or dash. For example, here is a list of valid package version numbers:

  • 1
  • 1.0.0
  • 3.2.build_13
  • 4.rc1
  • 10a-5

Version number tokens follow a strict ordering schema and are case sensitive. Underscore is the smallest character, followed by letters (a-z and A-Z), followed by numbers. The ordering rules are like so:

  • Underscore before everything else;
  • Letters alphabetical, and before numbers;
  • Lowercase letters before uppercase;
  • Zero-padded numbers before equivalent non-padded (or less padded) number ('01' is < '1');
  • If a token contains a combination of numbers and letters, it is internally split into groups containing only numbers or only letters, and the resulting list is compared using the same rules as above.

The following table shows some example version token comparisons:

smaller token larger token
0 1
a b
a A
a 3
_5 2
ham hamster
alpha beta
alpha bob
02 2
002 02
13 043
3 3a
beta3 3beta

Versions are compared by their token lists. The token delimiter (usually dot, but can also be dash) is ignored for comparison purposes - thus the versions '1.0.0' and '1-0.0' are equivalent. If two versions share the same token list prefix, the longer version is greater - thus '1.0.0' is a higher version than '1.0'.

Note that no special importance is given to specific characters or letters in Rez version numbers - the terms 'alpha' and 'beta' for example have no special meaning. Similarly, the number of tokens in a version number doesn't matter, you can have as many as you like. While you are encouraged to use semantic versioning (see here), it is not enforced.

Packages

A package is a versioned piece of software, that may have dependencies on other packages. Packages are self-contained - they have a single package definition file (typically package.py), which describes everything we need to know about the package in order to use it. Rez manages any kind of package, whether it be a python package, compiled package, or simply build code or configuration data.

Here is an example package definition file (see here for further details of each attribute):

name = "foo"

version = "1.0.0"

description = "Something that does foo-like things."

requires = [
  "python-2.6",
  "utils-1.1+<2"
]

tools = [
  "fooify"
]

def commands():
  env.PYTHONPATH.append("{root}/python")
  env.PATH.append("{root}/bin")

The requires section defines the requirements of the package. The commands section describes what happens when this package is added to an environment. Here, the bin directory in the package installation is appended to PATH, and similarly the python subdirectory is appended to PYTHONPATH.

Package Repositories

Packages are installed into package repositories. A package repository is a directory on disk, with packages and their versions laid out in a known structure underneath. Going on with our (foo, bah, eek) example, here is how the package repository might look:

/packages/inhouse/foo/1.1
                     /1.2
                     /1.3
/packages/inhouse/bah/2
                     /3
                     /4
/packages/inhouse/eek/2.5
                     /2.6
                     /2.7

# more detailed example of foo-1.1
/packages/inhouse/foo/1.1/package.py
                         /python/<PYTHON FILES>
                         /bin/<EXECUTABLES>

Here we have a package repository under the directory /packages/inhouse. The actual package content (files, executables etc) is installed into each leaf-node version directory, as shown for foo-1.1. The package definition file, in this case package.py, is always stored at the root of the package - right under the version directory for that package.

Rez only requires that the package's package.py file is at the root of the package installation. The layout of the rest of the package - for example, the python and bin directories - is completely up to the package's own build to determine. You should expect to see a package's commands section match up with its installation though. For example, notice how the path for foo's python files and binaries match what its package commands specified from earlier - "{root}/python" and "{root}/bin" will expand to these paths respectively.

Package Search Path

Rez finds packages using a search path in much the same way that python finds python modules using PYTHONPATH. You can find out what the search path is, using the rez command line tool rez-config (which you can also use to find any other rez setting):

]$ rez-config packages_path
- /home/ajohns/packages
- /packages/inhouse
- /packages/vendor

If the same package appears in two or more repositories on the search path, the earlier package is used in preference. This happens at the version level - an earlier package "foo-1.0.0" will hide a later package "foo-1.0.0", but not "foo-1.2.0".

The example search path shown is a typical setting. There are some central repositories later in the search path, where packages are released to so everyone can use them. But there is also a local package path at the front of the search path. This is where packages go that are being locally developed by a user. Having this at the start of the searchpath allows developers to resolve environments that pull in test packages in preference to released ones, so they can test a package before releasing it for general use.

You can change the packages search path in several ways. A common way is to set the REZ_PACKAGES_PATH environment variable; see Configuring Rez for more configuration options.

Package Commands

The commands section of the package definition determines how the environment is configured in order to use it. It is a python function, but note that if any imports are used, they must appear within the body of this function.

Consider this commands example:

def commands():
  env.PYTHONPATH.append("{root}/python")
  env.PATH.append("{root}/bin")

This is a typical example, where a package adds its source path to PYTHONPATH, and its tools to PATH. See here for details on what can be done within the commands section, as well as details on what order package commands are executed in.

Package Requests

A package request is a string with a special syntax which matches a number of possible package versions. You use package requests in the requires section of a package definition file, and also when creating your own configured environment directly using tools such as rez-env.

For example, here is a request (using the rez-env tool) to create an environment containing python version 2.6 or greater, and my_py_utils version 5.4 or greater, but less than 6:

]$ rez-env 'python-2.6+' 'my_py_utils-5.4+<6'

Here are some example package requests:

package request description example versions within request
foo Any version of foo. foo-1, foo-0.4, foo-5.0, foo-2.0.alpha
foo-1 Any version of foo-1[.x.x...x]. foo-1, foo-1.0, foo-1.2.3
foo-1+ foo-1 or greater. foo-1, foo-1.0, foo-1.2.3, foo-7.0.0
foo-1.2+<2 foo-1.2 or greater, but less than 2 foo-1.2.0, foo-1.6.4, foo-1.99
foo<2 Any version of foo less than 2 foo-1, foo-1.0.4
foo==2.0.0 Only version 2.0.0 exactly foo-2.0.0
foo-1.3|5+ OR'd requests foo-1.3.0, foo-6.0.0

The Conflict Operator

The '!' operator is called the conflict operator, and is used to define an incompatibility between packages, or to specify that you do not want a package version present. For example, consider the command:

]$ rez-env maya_utils '!maya-2015.6'

This specifies that you require any version of maya_utils, but that any version of maya within 2015.6 (and this includes 2015.6.1 and so on) is not acceptable.

Weak References

The '~' operator is called the weak reference operator. It forces a package version to be within the specified range if present, but does not actually require the package. For example, consider the command:

]$ rez-env foo '~nuke-9.rc2'

This request may or may not pull in the nuke package, depending on the requirements of foo; however, if nuke is present, it must be within the version 9.rc2.

Weak references are useful in certain cases. For example, applications such as nuke and maya sometimes ship with their own version of python. Their rez packages don't have a requirement on python (they have their own embedded version already). However often other python libraries are used both inside and outside of these applications, and those packages do have a python requirement. So, to make sure that they're using a compatible python version when used within the app, the app may define a weak package reference to their relevant python version, like so:

# in maya's package.py
requires = [
  "~python-2.7.3"
]

This example ensures that any package that uses python, will use the version compatible with maya when maya is present in the environment.

Implicit Packages

The implicit packages are a list of package requests that are automatically added to every rez request (for example, when you use rez-env). They are set by the configuration setting implicit_packages. The default setting looks like so:

implicit_packages = [
    "~platform=={system.platform}",
    "~arch=={system.arch}",
    "~os=={system.os}",
]

Rez models the current system - the platform, architecture and operating systems - as packages themselves. The default implicits are a set of weak requirements on each of platform, arch and os. This ensures that if any platform-dependent package is requested, the platform, architecture and/or operating system it depends on, matches the current system.

The list of implicits that were used in a request are printed by rez-env when you enter the newly configured subshell, and are also printed by the rez-context tool.

Dependency Resolving

Rez contains a solving algorithm that takes a request - a list of package requests - and produces a resolve - a final list of packages that satisfy the request. The algorithm avoids version conflicts - two or more different versions of the same package at once.

When you submit a request to rez, it finds a solution for that request that aims to give you the latest possible version of each package. If this is not possible, it will give you the next latest version, and so on.

Consider the following example (the arrows indicate dependencies):

Here we have three packages - 'foo', 'bah' and 'eek', where both foo and bah have dependencies on eek. For example, package "bah-4" might have a package definition file that looks something like this (some entries skipped for succinctness):

name = "bah"

version = "4"

requires = [
  "eek-2.6"
]

A request for "foo-1.3" is going to result in the resolve ("foo-1.3", "eek-2.7"). A request for "foo" will give the same result - we are asking for "any version of foo", but rez will prefer the latest. However, if we request ("foo", "bah"), we are not going to get the latest of both - they depend on different versions of eek, and that would cause a version conflict. Instead, our resolve is going to be ("foo-1.2", "bah-4", "eek-2.6"). Rez has given you the latest possible versions of packages, that do not cause a conflict.

Sometimes your request is impossible to fulfill. For example, the request ("foo-1.3", "bah-4") is not possible. In this case, the resolve will fail, and rez will inform you of the conflict.

Resolving An Environment

A user can create a resolved environment using the command line tool rez-env (also via the API - practically everything in rez can be done in python). When you create the environment, the current environment is not changed - you are placed into a sub-shell instead. Here is an example of using rez-env, assuming that the package repository is from our earlier (foo, bah, eek) example:

]$ rez-env foo bah

You are now in a rez-configured environment.

resolved by ajohns@14jun01.methodstudios.com, on Wed Oct 22 12:44:00 2014,
using Rez v2.0.rc1.10

requested packages:
foo
bah

resolved packages:
eek-2.6   /packages/inhouse/eek/2.6
foo-1.2   /packages/inhouse/foo/1.2
bah-4     /packages/inhouse/bah/4

> ]$ █

The output of rez-env shows the original request, along with the matching resolve. It's the resolve that tells you what actual package versions are present in the newly resolved environment. Notice the '>' character in the prompt - this is a visual cue telling you that you have been placed into a rez-resolved environment.

Putting It All Together

Let's go through what happens when an environment is resolved, using a new (and slightly more realistic) example. Let us assume that the following packages are available:

  • maya-2014.sp2;
  • nuke-8.0v3;
  • 3 versions of a maya plugin "mplugin";
  • 2 versions of a nuke plugin "nplugin";
  • 3 versions of a common base library "lib".

The following diagram shows what happens when the command "rez-env mplugin-1.3.0" is run:

The digram shows the following operations occurring:

  • Rez takes the user's request, and runs it through the dependency solver. The solver reads packages from the package repositories in order to complete the solve;
  • This results in a list of resolved packages. These are the packages that are used in the configured environment;
  • The commands from each package are concatenated together;
  • This master list of commands is then translated into the target shell language (in this example that is bash);
  • A sub-shell is created and the translated command code is sourced within this environment, creating the final configured environment.

The order of package command execution depends on package dependencies, and the order that packages were requested in. See here for more details.

You can’t perform that action at this time.