Explore deployment service and management API and configuration store options

[christoferlof]

This issue is to explore how to implement three components which are interconnected namely the:

• Deployment Service - reliable and auditable deployments of workspaces and services (aka resources)
• Configuration Store - metadata and desired configuration for resources
• Management API - API surface for apps and scripts to invoke the deployment service and read (and update) the configuration store.

High-level requirements

• Repeatable and reliable deployment runs - Ability to re-run and retry in cases of intermittent errors etc.
• Auditable deployment runs - Need to know who initiated a deployment run when.
• Declarative resource definitions
• Open for extensibility - Allow others to extend with additional service types etc.
• Management API needs to retrieve configuration data to serve requests from callers - i.e List services in workspace 'a'
• Handle operations across several control planes such as identity provider, resource providers, resource data plane.
• Clean up and removal - i.e delete a workspace
• Deployment services are managed by an Azure operations generalist - monitor health and troubleshoot deployments.
• Also see Design Management API surface #12

Edge cases we need to cater for

• Simultaneous updates of a shared resource - Example: Two workspace deployment runs are trying to update the same shared Firewall with their required rules
• Hide eventual consistency of the system to the users - show the actual resource state and not the desired state. Example: resource being updated or deleted.

Implementation

• Review past implementations
• Create spike

Ideas

Given above requirements, Terraform and GitHub actions would probably take us a long way. Let's discuss.

[jjcollinge]

A few other requirements:

• Must support a mechanism to expose the deployment outputs back to the management API. Or atleast support a method for reading them from another source from the management API.
• Must support a mechanism to handle secrets securely i.e. return secret references rather than the secret in plaintext.
• Must provide a feedback mechanism for seeing the progress and status of async operations.
• Must not drop operations - all operations should finalize (Succeed or Fail).
• Must try to finalize to a state that is consistent with the APIs view of the world.
• Must support operations (create, update, delete) on resources concurrently.
• Must be resilient to component failures (inc. API throttling).
• Must support referencing dependent resource operation output data.
• Must scale to support *many services within a workspace.

I'll probably add a few more as I think about them.

[TessFerrandez]

Decision was made that at a high level we will re-use a lot of the design from an internal engagement by the Clarke team - but remove dependencies on CAF and moving to python where possible