Add Artifact Repository

[ttrently]

This PR is the same as #1380. 1380 was closed by accident but it contains comments that are still valid for this PR.

UPDATE: Latest commit implements an artifact repository based on discussions found on #1393

Add support for artifact_repository plugins which allow for hosting of package payloads in remote servers. This works in conjunction with a MongoDB instance that contains the package information for querying and lookup.

Requirements

This implementation requires some additional packages that have not been included in rez.vendor.

• awscli
• pymongo

These can be installed via python -m pip install PKG --target path\to\rez\Lib\site-packages.

How To Use

In order to use you will need an existing S3 bucket and a MongoDB instance with a named database and a collection called "packages".

In your rezconfig.py you will need to set the following:

packages_path = [
    "mongo@localhost:27017/db_name",
]

artifacts_path = [
    "s3@s3://bucket-1/packages",
]

default_cachable = True
package_cache_same_device = True
package_cache_async  = False

If your MongoDB instance also requires a username / password, you can set it via:

REZ_MONGO_USERNAME=username
REZ_MONGO_PASSWORD=password

When these are set, it will format the packages path url as:

{REZ_MONGO_USERNAME}:{REZ_MONGO_PASSWORD}@localhost:27017/db_name

rez-build

The rez-build command has been extended to allow for installing to artifact repositories. This is done through the new -a --artifact-path argument from rez-build.

Example:

foo> rez-build --install -p mongo@uri:port/db_name -a s3@s3://bucket-1/packages

This build command will do the following:

1. Build the foo package normally
2. Install build results to a temporary location of $TEMP\rez\package_name\package_version, this is what will be pushed to the artifact repository.
3. Add a new variant entry to the specified mongo@ package repository
4. Push the installed build results to the s3@ artifact repository.

Missing / To-Do

• This uses a basic implementation of the artifact repository where they are defined separately from the package repository. On lookup, the artifact repositories are searched in order for the package payload, this seems to work well since the location stored in the database is the artifact_path qualifier, but may need to be more complex in the future.
• Artifact repository paths need to be extended to other cli's, such as move, release, and any others that interact with repositories.
• MongoDB authentication is still somewhat of an issue if you use multiple different databases that require different credentials.

[JeanChristopheMorinPerso]

@ttrently can you rebase your branch please? I see a bunch of changes that are already in master. Rebasing should fix that.