feature: store "proposed" and "golden" job versions in job-associated variables

[philrenaud]

When a Nomad job gets planned, the plan results are ephemeral and not stored anyplace: they exist for the user to validate a dry-run of a job submission and to see diffs, and if they look good, then to run the job in earnest.

This workflow depends on the nomad job plan-ing user to be the who actually approves the job for running. In some scenarios, they may want to submit the job for consideration, whereby another operator approves and ultimately runs the job. Nomad could support this workflow by use of Variables.

We could provide a flag for the Plan command (and associated web UI and CLI components) along the lines of --save-on-success that might save an automatically accessible variable at nomad/jobs/$id/_versions/$n

Subsequent nomad job status $id commands could indicate a pending job version is awaiting approval, and viewing the job page in the web UI could indicate the same, letting a cluster operator bump the version as they see fit.

[tgross]

This would be a nice external / third-party tool (or Nomad Enterprise feature), but yeah we could totally bootstrap this on top of Variables. You'd need to keep the HCL and the check index, so that you can re-parse the HCL and submit that with the check index set on approval. You'd probably also want the tool to expire stale plans when the job modify index increments, so that way you're not keeping around job versions that can never work because a newer version of the job was submitted since the plan was done.