Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
Latest commit 62c6b16 May 30, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src Add appname config May 30, 2019
test More documentation and fixes Jul 26, 2018
util Temp commit Oct 5, 2018
.gitignore Added test file to gitignore Jul 21, 2018
CHANGELOG.md update changelog Aug 28, 2018
Readme.md Readme updates Dec 19, 2018
bundle.js Add appname config May 30, 2019
bundle.js.map Add appname config May 30, 2019
package-lock.json 1.12.0 May 30, 2019
package.json 1.12.0 May 30, 2019
rollup.config.js Added jupyter interface Aug 1, 2018

Readme.md

Rally Tools

This repository provides multiple helpful tools for working within the SDVI Rally enviornment. Some features are:

  • Preset uploader
  • Rule uploader
  • Automated deployer
  • Remote diff checker
  • Code sync checker

Installation

  • npm install -g rally-tools
  • rally (to check if it works)
  • rally config

Library usage

Most classes and functions are exposed through src/index.js. This means when you want to create a node plugin that uses this library, you should just use import {Preset, Rule, rallyFunctions, ...etc} from "rally-tools".

Development

  • Clone this repo
  • Run npm install
  • Run npm link to symlink the rally executable
  • Start rollup -cw in the background to automatically compile code
  • Test your changes
  • Commit your changes and run npm version <minor|patch> to increment the version

Config

Pass the --config option to read/write a different config file location. If including as a module, then you need to call

const rally = require("rally-tools");

rally.loadConfig(filename);
// OR
rally.setConfig({... config object here ...})

Options:

  • chalk: allow colored output.
  • restrictUAT: Only allow GET requests from UAT, not POST/PUT/etc.
  • api: Your api keys and urls
  • repodir: The directory of your repository. Should have 3 folders: silo-presets, silo-rules, silo-metadata.
  • defaultEnv: Your development enviornment, usually DEV.

Usage

Use rally help or rally help [command] to see all public commands and basic documentation.

rally preset

This command deals with preset actions such as creating, uploading, and downloading.

rally preset create can be used to create a preset. When run without arguments, a command line UI will be given. If you wish to script this, the flags --provider, --ext, and --name can be given.

The basic download usage is rally preset list, which lists all presets. Giving the --resolve flag will internall resolve the dynamic references in an object. You can then add these to the output using --attach. Ex. rally preset list -e PROD --resolve --attach

rally preset upload -e [env] -f [preset] can be used to upload a file to a remote env. You can specify multiple -f arguments to upload multiple files. If the - argument is given (rally preset upload -) then the files are read from stdin. For example: git diff HEAD..UAT --name-only | grep silo-presets | rally preset upload - will upload all changed files using git as the reference.

rally preset diff -f [preset] can be used to view the differences between a local file and a remote one. --command can be used to run a command other than diff. For example, rally preset diff -f abc.xyz --command vimdiff -e PROD would compare the local file abc.xyz to the remote version on prod using vimdiff. (make sure that zbc.xyz has a proper rally header/metadata or this will fail).

rally preset grab -f [preset] will attempt to download the metadata file for this asset. The --full argument can be given to also download the code, too.

rally rule

See all rules. rally rule list. --raw available.

rally provider

See all providers. rally provider list. --raw available.

rally asset

This command allows you to create and launch workflows on assets.

The first part of the command will be getting an asset context. You can either:

  • Use an asset id (ex. discovery.sdvi.com/content/[id]).
  • add the --id [id] argument
  • rally asset --id 12345 launch ...
  • Use an asset name
  • add the --name [asset name] argument
  • rally asset --name 1232345_004_TCCS_123456_2 launch ...
  • Create a new asset
  • add the argument "create"
  • supply a name using --name
  • # will be replaces with a random number.
  • rally asset create --name "TEST_FILE_#" launch

rally supply

This is probably the most complex command mechanically.

rally supply calc [starting rule] will create a supply chain object in memory. Then, using other flags you can do something with this chain.

  • --to [env] will copy the supply chain onto the env, creating new rules and presets as needed.
  • --check [env] will do a diff on each file in the chain to the remote given

rally supply make -f [files]

rally conifg

This command manages the "~/.rallyconfig" file, so that you dont need to edit it manually. rally config simply creates a new config walking through all the options.

rally config [key] gives the config interactor for a single key. rally config chalk would bring up y/n menu for color. rally config api would bring up the configuration for all all the enviornments, but rally config api.DEV would let you modify just the DEV credentials.

rally config --raw prints out the current config including configs changed by command line options

Deployments

Deployments using this tool are based around supply chains. At their core, supply chains are simply a group of rally objects, where an object is either a rule, preset, or notification.

Although you can only deploy supply chains, there are many ways to construct the deployment you want.

The first, and easiest way you'll probably encounter is through rally supply calc [starting rule] [ending rule]. This does the heavy lifting of parsing rules, finding notifications, linking the presets, creating metadata, etc.

Using the E2 Supply chain as an example...

$ rally supply calc R1000 -e DEV
...
Calculating Supply chain...  R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow
Done!
Required notifications: 
N-21: SNS All - test
Required rules: 8
 R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow
 R-DEV-618: NL R2001 - MP - Make EST Media Articrafts by Split
 R-DEV-621: NL R3012 - MP - Make EST Media File using Media Convert
 R-DEV-622: NL R4001 - MP - Make EST Closed Caption File
 R-DEV-627: NL R3013 - MP - QC EST Media File Launcher
 R-DEV-623: NL R5001 - MP - Make EST Media ArtiCrafts by Join
 R-DEV-628: NL R3014 - MP - QC EST Media File using SimpleSDVIQC
 R-DEV-623: NL R5001 - MP - Make EST Media ArtiCrafts by Join
Required presets: 10
 P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow
  P-DEV-18: Fail
 P-DEV-283: NL - EST - Util Library
 P-DEV-284: NL - MP - Util  Library
 P-DEV-286: NL P2001 - MP - Make EST Media Articrafts by Split
 P-DEV-289: NL P3012 - MP - Make EST Media File using MediaConvert
 P-DEV-290: NL P4001 - MP - Make EST Closed Caption File
 P-DEV-306: NL P3013 - MP - QC EST Media File Launcher
 P-DEV-291: NL P5001 - MP - Make EST Media ArtiCrafts by Join
 P-DEV-307: NL P3014 - MP - QC EST Media File using SimpleSDVIQC

This internally creates a supply chain object, which we can then apply an action to.

An example action is sync, which is given by the --to arg. rally supply calc R1000 -e DEV --to LOCAL would sync this supply chain (based on DEV) to LOCAL. In order to move it to a protected envornment, add --no-protect.

However, calc is limited by the fact that it is very rigid. Its best use is the inital setup of an environment, or to mass move supply chains. To fix this, lets move to rally supply make

make takes a list of identifiers and constructs a supply chain. Identifiers are passed in by the this.files array. From the command line this can be given by suppling -f arguments or reading from stdin.

Lets say you edited these 3 objects in DEV.

$ cat > changes.txt
 P-DEV-283: NL - EST - Util Library
 P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow
 R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow

$ cat changes.txt | rally supply make -
Reading from stdin
Required notifications: 
Required rules: 1
 R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow
Required presets: 2
 P-DEV-283: NL - EST - Util Library
 P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow

Now you can treat this like any other supply chain, and deploy it. Remote to remote, or remote to local. However, this tool is built to integrate directly with git on your local filesystem.

If you edited those 3 files locally, then commited to git, you should be able to see the diff with the git command git diff HEAD HEAD^. We are only interested in the names, so lets get those.

Quick note: The shorthand for rally supply make - is rally @.

$ git diff HEAD HEAD^ --name-only
silo-presets/NL - EST - Util Library
silo-presets/NL P1000 - MP - Non Linear Media Preparation Workflow
silo-rules/NL R1000 - MP - Non Linear Media Preparation Workflow

$ #Passing those to make will produce a supply chain based on LOCAL
$ git diff HEAD HEAD^ --name-only | rally @
Reading from stdin
Required notifications: 
Required rules: 1
 R-LOCAL: NL R1000 - MP - Non Linear Media Preparation Workflow
Required presets: 2
 P-LOCAL: NL - EST - Util Library
 P-LOCAL: NL P1000 - MP - Non Linear Media Preparation Workflow

To give a generic approach, in order to deploy all the changes between 2 commits, run git diff featureCommit baseCommit --name-only | rally @ --to DEV. rally is currently stateless: It does not remember what is deployed, who deployed it or when. All this should be tracked through git. Therefore, tagging releases or using a release branch would allow for basic version control.

Automated deployments

Automated deployments should be constructed telling rally tools the list of changed presets and rules. Silo constants, notification presets, and silo metadata should be changed manually before an automatic deploy, as these can not be stored in source control.

For example, if the previous deployment to prod was the tagged commit v1.2.3, and the new deployment will be version v2.0.0, then the deployment script will be git diff v2.0.0 v1.2.3 --name-only | rally @ --to PROD --no-protect

This will need to be run on a computer with a .rallyconfig in the home directory, otherwise the config should be given by the --config flag to the rally command.

Examples

Heres some other examples of common usage:

Upload a preset rally preset upload -e DEV -f "~/ORP/silo-presets/Audio Metadata Conditioner.py"

Look at all ffmpeg jobs rally rule list -e DEV --resolve --attach | grep ffmpeg

Clone some ffmpeg jobs you want to edit (vipe optional) rally rule list -e DEV --resolve --attach | grep ffmpeg | vipe | rally supply make - --to LOCAL

Create a new supply chain and print it rally supply calc ORHIVE

Header Parsing

Rally tools supports a standard header format. Preset name

'''
name: (Name of the preset as it exists on the silo)
autotest: (Name of movie for testing purposes)
autotest: (Name of movie, supports and arbitrary nubmer)
autotest: (Name of movie, ....)
autotest: id: (id of movie to test)
'''

This can also exist using any other comment symbol except - directly preceding a key. For this reason, - can be used to disable autotests temporarily.

# name: ok
# autotest: ok
// name: also works
-autotest: will not run
-- autotest: this will run

Atom integration

Rally tools now supports a basic amount of atom integration including testing, uploading, downloading, and rule managment. Two plugins are used for this: process-pallete, and optionally, file-watcher. Please see the file process-pallete.json in jderby/ONRAMP_WORKFLOW_PYTHON.

This should be copied into your base directory (same level as the silo-* folders)

This is still early in testing and does not support features like diffs and inline live test results

Troubleshooting

Cannot acclimatize shelled preset

Solution: Create the preset on the remote enviornment manually, or run rally preset create

Under normal usage, presets will have an associated metadata file saved. This contains information like its provider type, input and output settings, or timestamps. Preset#acclimatize attempts to take this data from a generic format into an environment specific format so that it can be accuractly created when uploading. A file without any metadata is marked as "Shelled" and given some dummy data while limiting functionality. This functionality includes updating the code of a preset, or viewing the metadata of an enviornment.

CLI Aborted: Protected enviorment

Solution: Add the --no-protect flag, or run rally config restrictUAT to unprotect UAT (if the error is on UAT).

Protected enviorments cannot recieve anything but get requests, so any kind of POST/PUT/PATCH will fail with this error. Internally, --no-protect is sets the --protect flag to false instead of true, which in turn sets the configObject.dangerModify flag to true. So if you really wish, you could add "dangerModify": true, to your config to allow unrestricted UAT/PROD posts, then use the --protect flag when you want safe calls.

API Error

Sometimes, the rally API simply wont work. Verify that all endpoints are active by running rally. Under normal circumstances, they should return a 2xx response.

If that is ok, read the data that is returned by the API to see if it is a fixable error: ex. 401 Unauthorized probably means that you have a bad API key, so run rally config api or rally config api.UAT.

My problem isn't in the list

Ask me on SDVI or discocomm slack @John Schmidt

You can’t perform that action at this time.