Add basic limits functionality

[gregdenton]

Fixes #196

This change adds the ability to specify arbitrary limits to be placed on job layers. Basic limits implemented here will count the number of running frames on the cue and ensure the dispatcher does not dispatch any additional frames containing those limits if the limit max value has been met.

This change includes database schema changes to add new tables and new functions to clean up limit records when a layer has been deleted. Database changes are added as a new migration.

This change also includes updates to cuegui to add a new limit plugin for creating, editing, and deleting limits, as well as changes to view and edit limits assigned to layers.

Cuesubmit and pyoutline have also been updated to allow new jobs to be submitted with limits.

[linux-foundation-easycla]

The committers are authorized under a signed CLA.

• ✅ Greg Denton (91f0be7, 20e68d9, ceddca4, c08088b, dff20d7, bbd09fa, 43128c3, 8dc1d35, 0d85512, 0215175, f797a4d, dee766d, 0b900b0, 7640551, 25e41c1, 60d5ba3, 12b1a09, 1fce3d7)

[bcipriano]

LGTM!

Could you please also prep a message that we'll send out to the users list which covers a) explain the new feature and how to use it, and b) how to upgrade your database with the new migration?

We might want to actually add an Admin guide page on applying migrations, and link to that instead.

Happy to help review any/all of those materials if you'd like.

[gregdenton]

Thanks @bcipriano. Please see the introduction and user guide below. I plan on emailing this out to the user group immediately after release. Let me know what you think.

This change introduces a new functionality called Limits. Limits allow users to specify a maximum number of concurrently running frames associated with that Limit. Limits are specified on the Job Layer and all Frames within that Layer are considered to take 1 limit count. Once the total limit count reaches the configured max value, the dispatcher will stop any additional frames from running until a frame with that limit has completed.

Start by configuring a new Limit.

1. Open CueGui
2. Load "Limits" view from the "Views/Plugins->Cuecommander" menu.
3. Click the "Add Limit" button in the Limits view.
4. Enter a name for the new limit and press "OK".
5. Right click on the newly created limit and select "Edit Max Value".
6. Enter the desired maximum number of concurrently running frames for this limit.

Your limit is now configured and ready for use. New submissions will be able select this limit in CueSubmit UI when creating a new job.

Limits can also be added/removed from existing layers by:

1. Open CueGui
2. Load the "Monitor Jobs" view from the "Views/Plugins->Cuetopia" menu.
3. Double click on a job to load its layers.
4. Right click on the job you'd like to modify limits for.
5. Click "Properties".
6. Check/Uncheck limits you'd like enable/disable for this layer.

To update your install with change, you'll need to reinstall all OpenCue components. If you are installing from source be sure to update and recompile the proto files. You will also need to update your database with the latest schema migration. The process for doing this is described here, https://www.opencue.io/docs/other-guides/applying-database-migrations (link not live yet).

We plan on adding additional functionality for "limiting by host count" shortly.

Please reach out on the email group if you have any questions.

[bcipriano]

That text is great.

The only thing is, I really wish we could provide a full "Upgrade your OpenCue deployment" guide. What do you think? Maybe @sharifsalah has some time to help out with that?

[sharifsalah]

Hi both, Brian's comment slipped my attention for some reason. Sure I can help with this. I think we've actually made a start and I have some other suggestions. I'll add an issue to opencue.io to publish a guide to upgrade a deployment. I'll definitely have some questions for you both about approach though.

[sharifsalah]

The following guide is now published:

https://www.opencue.io/docs/other-guides/applying-database-migrations/

[bcipriano]

Published doc LGTM too. I think we're good to merge?

[larsbijl]

Anything holding this up? I have this running locally and migration worked as expected.

[gregdenton]

I was waiting to publish an additional doc on upgrading a deployment, but I believe we can probably merge this in and and just publish that doc when the next release is pushed.

[gregdenton]

@larsbijl, Thanks for the note and testing this branch and sorry it had been sitting for awhile.