[Feature] Install as package

[j0yu]

Added -p and -P options during install to install rez as a package under the given install directory (typically a packages)

🤖 See our Experimenting with actions - PR 579 GitHub actions workflow for some experimental CI and tests! 🤖

For now, this PR will only focus on install.py to perform a production friendly install just requiring python. The rez package will be setup up with the variants:

• platform-system platform name (e.g. linux), and
• python-MAJOR.MINOR for the Python version used to run python install.py

So far, it is not intended to replace rez bind rez, but maybe at some point (see below).
It does install the rez package in a similar structure and performs package commands to setup the following:

• Add {root}/bin/rez into PATH
• Copy {root}/lib/python*//rez* into {root}/python (for clean Python imports of rez* modules)
• Add {root}/python into PYTHONPATH

e.g. for commit d987f7d

python install.py --help

usage: Rez installer [-h] [-v] [-s] [-p | -P PACKAGE] [DIR]

Install rez in a production ready, standalone Python virtual environment.

positional arguments:
  DIR                   Destination directory. If '{version}' is present, it
                        will be expanded to the rez version. Default: /opt/rez

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Increase verbosity.
  -s, --keep-symlinks   Don't run realpath on the passed DIR to resolve
                        symlinks; ie, the baked script locations may still
                        contain symlinks
  -p, --as-rez-package  Install using rez package structure as 'rez'. (DIR
                        should be a rez packages directory)
  -P PACKAGE, --as-package PACKAGE
                        Given a package name, install using rez package
                        structure. (DIR should be a rez packages directory)

python install.py -p ~/packages will install as:

~/packages
~/packages/rez
~/packages/rez/2.47.2
~/packages/rez/2.47.2/package.py
~/packages/rez/2.47.2/platform-linux
~/packages/rez/2.47.2/platform-linux/python-2.7
~/packages/rez/2.47.2/platform-linux/python-2.7/python
~/packages/rez/2.47.2/platform-linux/python-2.7/bin
~/packages/rez/2.47.2/platform-linux/python-2.7/lib
~/packages/rez/2.47.2/platform-linux/python-2.7/include
~/packages/rez/2.47.2/platform-linux/python-2.7/completion

rez view rez

URI:
/home/runner/packages/rez/2.47.2/package.py

CONTENTS:
name: rez

version: 2.47.2

description: Standalone, production-ready Rez installation

variants:
- - platform-linux
  - python-2.7

commands: "\"\"\"Setup PYTHONPATH (inspired by src/rez/bind/rez.py).\"\"\"\nimport\
  \ os\nenv.PATH.append(os.path.join('{root}', 'bin', 'rez'))\nenv.PYTHONPATH.append(os.path.join('{root}',\
  \ 'python'))\n\nif defined('SHELL'):\n    is_csh = \"csh\" in str(env.SHELL)\n \
  \   ext = \"csh\" if is_csh else \"sh\"  # Basic selection logic\n    source(os.path.join('{root}',\
  \ 'completion', 'complete.' + ext))"

base: /home/runner/packages/rez/2.47.2

This feature was originally suggest for v2.27 but has been rebase'd and adapted to the latest rez versions since. Needless to say, the various evolution and improvements to rez has put this feature in an interesting spot.

Highlights from discussions below

pip install compatible

rez now uses pip install internally in install.py! Therefore after downloading/git clone this repository, one can install rez via several methods (assuming you're running from same folder as rez's install.py/setup.py):

# 1. The OG method into a Python venv (production safe), then install rez using venv's pip install
python install.py DEST_DIR   

# 2. pip install using current folder as pip package
pip install --target DEST_DIR . 

# 3. if you have rez already installed, install rez as pip Python package into local package path
# See https://github.com/nerdvegas/rez/wiki/Configuring-Rez#local_packages_path
rez-pip --install .   

In the current state of this PR/feature, it more closely follows 1. It's able to install rez even if rez is not currently installed.

Since pip based 2 and 3 installs do not install rez into a venv, more work will need to be done to this PR to get it to be pip friendly. See #579 (comment), #579 (comment), #579 (comment)

Deprecate rez bind rez

Since this PR installs into a similar structure/package declaration as the rez package produced by rez bind rez, given some extra work, maybe even this PR can deprecate the rez rez bind.

rez bind rez currently sets up only the rez Python modules into a rez package. This is similar to the pip style installs (2 and 3), but just without the bin folder/tools.

See #579 (comment), #579 (comment)

[fnaum]

Hi @j0yu , this is pretty useful, we made rez as a package 4 years ago, but I think it was more complicated, maybe because our enrvironment with CentOS-6. I sort of remmeber that one of the problem is the python version that is shipped with rez. I'll dig in the history, but from memory and at first glace seems quite different to this.
But I guess if this is working fine for you, then it is fine. I can also give it a try at this branch

[mottosso]

Solved once wheel and pypi support get merged.

$ rez wheel --install rez==2.30.0

[nerdvegas]

Hi @j0yu, just letting you know I'm aware of this PR and I agree with the functionality. I'm just waiting til after there's some work on updating the installer, which will be happening within a few weeks. I'll either merge or cherrypick from this PR as part of that effort.

Thx
A

[j0yu]

Solved once wheel and pypi support get merged.

$ rez wheel --install rez==2.30.0

@mottosso Interesting! I'll give it a test and maybe even port this functionality over so you can directly during install using pip install --install-option="--as-package=rez"

Great stuff on #614 and #625, so right now I'm looking to run something like these?

$ pip install --user bleeding-rez  # I'll see if I can add --install-option here to do "rez wheel" after install
$ rez wheel --install rez==2.31.4

Hi @j0yu, just letting you know I'm aware of this PR and I agree with the functionality. I'm just waiting til after there's some work on updating the installer, which will be happening within a few weeks. I'll either merge or cherrypick from this PR as part of that effort.

Thanks @nerdvegas, I'm happy to rebase this branch over whenever the installer is updated. Just drop a message on here and I'll update this PR to the latest master branch.

I saw there's a lot of discussion about install.py in other PRs and I wouldn't mind porting this feature to whichever/all install methods. I'm just lazy and want to install rez as a package right-off-the-bat in a one-liner

[mottosso]

so right now I'm looking to run something like these?

Yes, that's exactly right! Except at the moment rez itself isn't on PyPI, so for a working version today till then from a fresh system, it would be:

$ pip install --user bleeding-rez
$ rez pip --install bleeding-rez  # wheel got renamed back to `pip` in the bleeding-rez version

And you'd have Rez both on your system and as a package.

I'll see if I can add --install-option here to do "rez wheel" after install

Double inception! Haven't thought of that, let me know how that goes.

[nerdvegas]

I just tested this, but it creates a very strange rez package:

• The package payload actually contains a virtualenv-based production install of rez
• The package actually then uses the rez installation if you request it.

See in its commands:

def commands():
    """Setup PYTHONPATH (inspired by src/rez/bind/rez.py)."""
    import os
    import rez  # <- this picks up installed rez!
    env.PATH.append(os.path.join('{this.root}', 'bin', 'rez'))
    source(os.path.join('{this.root}', 'completion', 'complete.sh'))
    
    rez_path = os.path.dirname(rez.__path__[0])

A rez package of rez should just be the rez API, independent of whatever the current installation is. It's entirely feasible to have (for eg) installed rez-2.40.0, but rez package ~/packages/rez/2.39.0, and so on. This is actually pretty normal at Method for eg - our installation is updated first, and the rez rez package is deployed later, once the installation has been tested for a while.

As it happens, I've just done some testing with the recent rez-pip improvements, and this actually works very well:

]$ cd <rez source clone>
]$ rez-pip --install .

I think this approach should be fine, as rez-pip is installing directly from source here, and doesn't need to download anything. There's a little massaging to do though - namely, the bin tools should be removed (because they are only guaranteed to function correctly in a production install); as well as a couple more subtleties that I won't delve into here.

So my thoughts at the moment are that we should:

1. provide python install.py -p option, much like here
2. not give the option to change the rez package name
3. if rez-pip is available, use it to do a rez-pip --install .
4. add some hardcoded stuff in pip.py to detect if rez itself is getting installed, and do some massaging as mentioned above.
5. If rez-pip is not available, then do a temp rez install, and use the rez-pip from there, to then do steps 3,4,5.

[JeanChristopheMorinPerso]

I'm inline with what @nerdvegas said. Though, I think differently of adding a -p argument to the install.py script. I would just document how someone can create a rez rez package by using rez-pip --install . from the source, or by doing rez-pip --install rez (so downloading from PyPI).

I would also add that we could make the package variant'ded (on python) so that it is only usable when using a compatible python version. That is it cannot be used with python 3 for example. Or the day we fully support python 3, then only specify the python versions with which it is compatible with (wild guess will be 2.7+|>=3.4|<=3.7).

5. If rez-pip is not available, then do a temp rez install, and use the rez-pip from there, to then do steps 3,4,5.

In which case can this happen?

Also, should we deprecate the rez bind officially (I mean the rez-bind -i rez)?

[j0yu]

Thanks for the feedback, I should have mentioned that I just did a basic rebase I haven't fully tested it with the new rez versions yet but it seems you guys have beat me to testing it out.

I agree with the comments, so I'll crack on and see how much of these I can get done:

• Fix/remove import rez
• Make sure (Docker) tests work again... CI?
• Figure out compatibility from setup.py + run time OS/platform to build variants based off it
• Switch to rez-pip based install test cd <rez source clone> && rez-pip --install .

Also, should we deprecate the rez bind officially (I mean the rez-bind -i rez)?

Not sure how/if I should do something in this PR for that @nerdvegas? Something like:

1. Deprecation warning when run
2. "Alias" it to run the rez-pip install ... command

[j0yu]

On another note, @nerdvegas is pip install rez suppose to pull down the latest 2.42.2 version? I'm only getting/seeing 2.0.ALPHA.52 on pypi.org

[instinct-vfx]

You have to build the wheel locally as builds are not yet published to pypi

[nerdvegas]

The pip installation should occur from the local source. Any option to install rez as a rez package, from install.py, I would not expect to fetch externally.

I mentioned a few caveats about installing rez via rez-pip; they're worth going over here.

The first is cli tools. The rez rez package doesn't have any, yet rez-pip will create them. So I am actually thinking... since pip-installed rez cli tools aren't safe anyway, maybe we should remove them altogether? We could do this simply by providing an empty entry point list in setup.py, unless some env-var is set (which only install.py would set). This seems clean to me - it removes potentially unsafe cli tools, and it means one less caveat for rez-pip-installing rez itself. It also means we don't need any warnings in the docs/tools about those tools potentially misbehaving.

The second is varianting. The rez-pip tool variants on python-MAJOR.MINOR, even if it could theoretically variant on a looser python version range. The reasons for this have been gone over elsewhere, but the short of it is, due to python "environment markers", it's possible to end up with odd/overlapping python version ranges. However, we know that rez doesn't need to be varianted on MAJOR.MINOR - in fact, pretty soon (when py3 port is finished) it won't need to be varianted at all, and instead could just require python-2.7+. I think we need to address this, otherwise newer rez installs will install to narrower python version variants than previously, and that could break existing studios' processes.

To address the varianting issue, I wonder if we could do it in a general way. Is it possible to author metadata in a python package, that could communicate this (and potentially other) info to rez? Ie, can we embed into into a native python package, about how it should be varianted in rez? I think this would be the best approach, as it would solve an issue specific to this PR, as well as being a generally useful new feature.

[j0yu]

So I started to get back onto this again but I've decided to go step by step and return the scope of this PR to the original, simpler quest of:

Can I run python install.py -p ~/packages to install rez as a production, venv based rez package?

I feel there's more opportunities to get rez-pip to work too but I feel like I need re-think about this PR since rez has evolved a lot since 2.27.0.

I've also update the PR description too with some bits I picked out from the discussions

[nerdvegas]

See #845