Formalizing and documenting the JobSpec in digger

[motatoes]

Why is this needed?

The concept of the JobSpec has been grown organically in digger as we started to build the orchestrator. Before the orchestrator digger was running as a cli tool that parses and reacts to action events. After the introduction of the orchestrator that piece was now reacting to webhooks (instead of doing it in the cli). So to keep things DRY we have extracted the common piece of logic into a library and we used it in the backend as well.

There was one problem here. The parsing library has been creating a golang struct called Job that was not really serializable into a format that can be sent over an API such as JSON. In order to solve for that we created an object called JobJson that can serialize into JSON and then be consumed by the cli (deserialized). This object has mostly matching fields as the Job struct but overtime has also included more fields that are not in the Job. One such example is the BackendHostname, BackendToken and BackendOrg which are metadata fields to allow the cli to communicate with the backend.

As we have added more features and are starting to think about extendability points of digger (#1526) and also about more multi-ci backend support (#1531) its important to have a clear idea of what is included and what is not included in the JobSpec. It is also important to document the schema of this JSON for third parties to be able to consume it as necessary.

Therefore it is important to be able to document and formalize this spec since it will be the point of extensibility for the cli. Furthermore it is a good way to refactor the backendless and orchestrator modes to utilise the same JobSpec as it will lead to more reliability in the branching of behaviours. Currently the backendless relies on the Job while the orchestrator mode first creates a Job and then converts it to a JobSpec to send it to cli for then converting it to a Job again. This chain of conversions has lead to some bugs for example #1536. So instead of having this conversion of Job -> JobSpec -> Job we should have only one spec in both branches (JobSpec) and rely on it for all of our executions in both orchestrator and backendless modes.