DeploymentTemplate controller technical design #63
Conversation
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
|
|
||
| This design proposes the creation of two new components of the Radius control plane: the ApplicationDeployment reconciler (part of the `radius-controller`) and a new service that is responsible for compiling ARM JSON from Bicep manifests (`radius-bicep`). | ||
|
|
||
| The `radius-controller` will be responsible for reconciling ApplicationDeployment resources. When an ApplicationDeployment resource is created or updated, the controller will read the Bicep template and parameters from the ApplicationDeployment resource and send them to the `radius-bicep` service. The `radius-bicep` service will compile the Bicep and bicepparam files into ARM JSON and return the ARM JSON to the `radius-controller`. The `radius-controller` will then send the ARM JSON payload to UCP to deploy the resources. |
There was a problem hiding this comment.
Why does radius-bicep need to be a separate service? We can go from templates to ARM JSON using a bicep build command so can this be rolled into the radius-controller?
There was a problem hiding this comment.
+1. Turning bicep build into a service feels complex.
|
|
||
| The `radius-controller` will be responsible for reconciling ApplicationDeployment resources. When an ApplicationDeployment resource is created or updated, the controller will read the Bicep template and parameters from the ApplicationDeployment resource and send them to the `radius-bicep` service. The `radius-bicep` service will compile the Bicep and bicepparam files into ARM JSON and return the ARM JSON to the `radius-controller`. The `radius-controller` will then send the ARM JSON payload to UCP to deploy the resources. | ||
|
|
||
| ### Architecture Diagram |
There was a problem hiding this comment.
A little confused by second half of the diagram. Is the MissingDependency the environment that hasn't been deployed when you create the AppDeployment resource for the application?
There was a problem hiding this comment.
exactly. I'll make it more clear in the diagram
| This design proposes the creation of two new components of the Radius control plane: the ApplicationDeployment reconciler (part of the `radius-controller`) and a new service that is responsible for compiling ARM JSON from Bicep manifests (`radius-bicep`). | ||
|
|
||
| The `radius-controller` will be responsible for reconciling ApplicationDeployment resources. When an ApplicationDeployment resource is created or updated, the controller will read the Bicep template and parameters from the ApplicationDeployment resource and send them to the `radius-bicep` service. The `radius-bicep` service will compile the Bicep and bicepparam files into ARM JSON and return the ARM JSON to the `radius-controller`. The `radius-controller` will then send the ARM JSON payload to UCP to deploy the resources. |
There was a problem hiding this comment.
Do we need a webhook? For example, a validating webhook that will validate the changes (create/update/delete) to the ApplicationDeployment resources before being picked up by the reconciler? We have a validating webhook for our Recipe resources btw, if you need an example.
Ref: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/.
|
|
||
| #### ApplicationDeployment Custom Resource Definition | ||
|
|
||
| The `ApplicationDeployment` resource will be a new Kubernetes CRD that will be used to contain the Bicep template and parameters for deploying Radius resources. The CRD will have the following fields: |
There was a problem hiding this comment.
We have RadiusDeployment resource, I think, if I am not mistaken. How different are they going to be?
There was a problem hiding this comment.
I will check with Ryan about this
|
|
||
| #### Bicep Service | ||
|
|
||
| The `radius-bicep` service will be a new service that is responsible for compiling ARM JSON from Bicep manifests. The service will expose an HTTP endpoint that accepts a payload containing a Bicep template and bicepparam file and returns the compiled ARM JSON. The service will use the Bicep compiler to compile the Bicep manifest and will return the ARM JSON to the caller. The service will be stateless and will not store any data. |
There was a problem hiding this comment.
We may have something like this in our code. You can check rad bicep publish command, I think.
There was a problem hiding this comment.
|
|
||
| ### Alternatives Considered | ||
|
|
||
| We could package the Bicep compiler into the `radius-controller` service instead of creating a separate service. |
There was a problem hiding this comment.
I think that service should be separate since it could be used by some other components in future.
|
|
||
| #### ApplicationDeployment Controller | ||
|
|
||
| - If the Bicep template or parameters file is invalid, the controller will mark the ApplicationDeployment resource as "Invalid" and will not deploy the resources. |
There was a problem hiding this comment.
ValidatingWebhook can do this part.
|
|
||
| #### CLI | ||
|
|
||
| There will be a new CLI command `rad applicationdeployment generate` that will generate an ApplicationDeployment resource from a Bicep template and parameters file. |
There was a problem hiding this comment.
Can we use rad deploy command by any chance?
There was a problem hiding this comment.
We could do something like rad deploy --generate app.bicep. Let's discuss in design meeting
| design document, if one exists. | ||
| --> | ||
|
|
||
| This design proposes the creation of two new components of the Radius control plane: the ApplicationDeployment reconciler (part of the `radius-controller`) and a new service that is responsible for compiling ARM JSON from Bicep manifests (`radius-bicep`). |
There was a problem hiding this comment.
This paragraph names the component "ApplicationDeployment reconciler", but I don't see a component with that name in the diagrams or detailed design.
There was a problem hiding this comment.
Or is there a process called "reconciliation"? It may help to identify the process by name.
| @@ -0,0 +1,370 @@ | |||
| # ApplicationDeployment Controller | |||
There was a problem hiding this comment.
One high level question have I ....
Is ApplicationDeployment the right thing? ApplicationDeployment implies that it's for using Bicep to deploy Radius applications.
What about other usecases for Bicep that aren't an application?
There was a problem hiding this comment.
(I know I was the one that created ApplicationDeployment in my prototype. I'm not 100% sure that's the right terminology or design.)
|
|
||
| **Goal: Radius resources can be deployed by authoring a Kubernetes CRD** | ||
|
|
||
| **Goal: Radius resources can be deployed without using the Radius CLI or directly calling Radius APIs** |
There was a problem hiding this comment.
Should rephrase to say something about "Radius users will not have to call Radius APIs directly". We will still need to as part of the controller implementation.
|
|
||
| **Goal: Radius resources can be deployed without using the Radius CLI or directly calling Radius APIs** | ||
|
|
||
| **Users have a simple script available to generate the necessary CRD** |
There was a problem hiding this comment.
This is optional, but makes the user story easier.
|
|
||
| #### ApplicationDeployment Custom Resource Definition | ||
|
|
||
| The `ApplicationDeployment` resource will be a new Kubernetes CRD that will be used to contain the Bicep template and parameters for deploying Radius resources. The CRD will have the following fields: |
There was a problem hiding this comment.
We might want to change the name of this resource
| **Non-goal: Full support for GitOps** | ||
| - We will not yet be implementing automatic generation of ApplicationDeployment resources from Bicep manifests or querying Git repositories. This design will be covered in a separate design document. | ||
|
|
||
| **Non-goal: Deletion of resources** |
There was a problem hiding this comment.
I'd suggest we bring this in. I think it's going to be more complicated to leave out.
| priority decisions, and how we will determine success. | ||
| --> | ||
|
|
||
| **Goal: Radius resources can be deployed by authoring a Kubernetes CRD** |
There was a problem hiding this comment.
What about non-Radius usecases for Bicep? in-scope? out of scope?
There was a problem hiding this comment.
What about non-application scenarios - like updating the configuration of an environment?
| --> | ||
|
|
||
| **Non-goal: Full support for GitOps** | ||
| - We will not yet be implementing automatic generation of ApplicationDeployment resources from Bicep manifests or querying Git repositories. This design will be covered in a separate design document. |
There was a problem hiding this comment.
This is smart. Can we draw a clear line around what the CRD contains?
| using './env.bicep' | ||
|
|
||
| param k8sNamespace string = 'demo' | ||
| ``` |
There was a problem hiding this comment.
This is a great start, but it's missing a few things. Let's discuss
|
|
||
| #### ApplicationDeployment Controller | ||
|
|
||
| - If the Bicep template or parameters file is invalid, the controller will mark the ApplicationDeployment resource as "Invalid" and will not deploy the resources. |
There was a problem hiding this comment.
Are we going to do validation using Bicep/as part of the Bicep service? i.e. we try to build the ARM JSON and if there's errors, return
There was a problem hiding this comment.
nvm I just read the next part :)
| The `ApplicationDeployment` resource will be a new Kubernetes CRD that will be used to contain the Bicep template and parameters for deploying Radius resources. The CRD will have the following fields: | ||
|
|
||
| - `spec.template`: The Bicep template file that defines the resources to deploy | ||
| - `spec.parameters`: The Bicep parameters file that defines the parameters for the template |
There was a problem hiding this comment.
Can I have multiple parameter files?
| - `status`: The status of the deployment, which will be one of the following values: | ||
| - `Pending`: The deployment is pending | ||
| - `Deployed`: The deployment has been completed | ||
| - `Missing Dependencies`: The deployment is missing dependencies |
There was a problem hiding this comment.
We'll want to put more in status. Kubernetes controllers don't get private state, they use the status to store state.
We also need to implement some standard boilerplate for interoperability with other tools:
- The "Phrase" field
observedGeneration- A "Ready" condition
|
|
||
| #### `rad applicationdeployment generate` Command | ||
|
|
||
| The `rad applicationdeployment generate` command will be a new command that will generate an ApplicationDeployment resource from a Bicep template and parameters file. The command will take the following arguments: |
There was a problem hiding this comment.
Could we nest this under rad bicep somewhere? It's not clear to me that applicationdeployment should be a top level noun. I think understanding what other commands we'd add might change my mind.
| The `rad applicationdeployment generate` command will be a new command that will generate an ApplicationDeployment resource from a Bicep template and parameters file. The command will take the following arguments: | ||
|
|
||
| - `-t, --template`: The path to the Bicep template file | ||
| - `-p, --parameters`: The path to the Bicep parameters file |
There was a problem hiding this comment.
Take a look at the options we support for rad deploy and how parameters are specified there.
|
|
||
| The `radius-controller` will be updated to include a new controller that watches for ApplicationDeployment resources. When an ApplicationDeployment resource is created or updated, the controller will read the Bicep template and parameters from the ApplicationDeployment resource and send them to the `radius-bicep` service. The `radius-bicep` service will compile the Bicep and bicepparam files into ARM JSON and return the ARM JSON to the `radius-controller`. | ||
|
|
||
| For each of the resources in the template, the `radius-controller` will determine if the dependencies for the resource already exist by querying UCP. If they do not, then the controller will mark the ApplicationDeployment resource as "Missing Dependencies" with the resource that is missing and will not deploy the resource. If the dependencies do already exist for each of the resources in the template, then the controller will send the ARM JSON payload to UCP to deploy the resources. Once the deployment is complete, The controller will trigger a re-deployment of resources with status "Missing Dependencies" that depend on the newly deployed resources. The controller will update the status of the ApplicationDeployment resource to "Deployed" once all resources have been deployed. |
There was a problem hiding this comment.
I think the design can be simpler than this.
For each of the resources in the template, the
radius-controllerwill determine if the dependencies for the resource already exist by querying UCP.
If we needed to do this it would actually be really hard. It's hard to just look at a Bicep template and compute exactly the resource IDs of the resources.
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
| - Check if the template contains a `param application`. If it does, the controller will query the Radius API to get an application ID (must have exactly one application if left unspecified like this) and deploy the template using that application. | ||
| 2. The controller will send the ARM JSON to the Radius API to deploy the resources. If there are any errors during the deployment, the controller will update the `status.phrase` and the `status.description` of the BicepDeployment resource to indicate that the deployment has failed and include the error message. | ||
|
|
||
| > TODO: We need to figure out deletion of specific resources within the template. Let's discuss in the design meeting. |
There was a problem hiding this comment.
This is a very important detail that I've left out of the design because I haven't thought of a good solution yet. I'd like to discuss in the meeting.
There was a problem hiding this comment.
Let's dig into this. It's not too scary.
There was a problem hiding this comment.
| } | ||
| ``` | ||
|
|
||
| #### `rad bicep generate bicepdeployment` Command |
There was a problem hiding this comment.
This name is very up for debate.
There was a problem hiding this comment.
suggestion rad bicep generate-krm or rad bicep generate-kubernetes or similar.
Additional "words" adds to the command hierarchy. I'm not sure we're going to have additional rad bicep generate <something else> commands in the future.
| - Check if the template contains an empty `param environment`. If it does, the controller will query the Radius API to get an environment ID (must have exactly one environment if left unspecified like this) and deploy the template using that environment. | ||
| - Check if the template contains a `param application`. If it does, the controller will query the Radius API to get an application ID (must have exactly one application if left unspecified like this) and deploy the template using that application. | ||
|
|
||
| The controller will also be shipped with a Bicep binary which will allow it to compile Bicep templates and parameters into ARM JSON. The Bicep binary will be included in the `radius-controller` container image. |
There was a problem hiding this comment.
This will require us to change the base container image to something like ubuntu which has the right libraries for running the bicep binary.
There was a problem hiding this comment.
There are other ways to solve this problem 😆
| 3. This may require us to start a PUT operation. After that we can continue polling. | ||
|
|
||
| 1. The controller will perform some simple lookups to Radius to determine if the provided template can be deployed. It will make two checks: | ||
| - Check if the template contains an empty `param environment`. If it does, the controller will query the Radius API to get an environment ID (must have exactly one environment if left unspecified like this) and deploy the template using that environment. |
There was a problem hiding this comment.
This is the kind of thing that the rad CLI does today and I figured it'd be easy enough to implement. let me know what y'all think
There was a problem hiding this comment.
where do you think we'll get the application/environment names + scope?
| priority decisions, and how we will determine success. | ||
| --> | ||
|
|
||
| **Goal: Users can use Kubernetes GitOps tools (Flux, ArgoCD) to deploy and manage resources defined in Bicep manifests** |
There was a problem hiding this comment.
do we feel like this is feasible? are there any edge cases with Bicep-defined resources where we feel like we would not be able to manage them all with one CRD?
There was a problem hiding this comment.
For the Bicep controller, users can use any Kubernetes tooling. GitOps is our motivation for the project overall, but for this component we don't have a bias towards any specific tool. eg: Helm, Kubectl, Kustomize, KeptN, etc.
Kubernetes is an API, and adding a controller for Bicep allows the API to manage anything Bicep can manage.
There was a problem hiding this comment.
are there any edge cases with Bicep-defined resources where we feel like we would not be able to manage them all with one CRD?
I'm not worried about this. Goal goal.
Useful thought experiment... if there were Bicep things that can't work - would be a design issue or "just" a bug?
|
|
||
| #### Bran can use GitOps to deploy and manage a Radius application in production | ||
|
|
||
| Bran is a developer who uses GitOps to automate the deployment of Radius applications to production Kubernetes clusters. His infrastructure team has set up a Radius installation and environment on the production cluster, and Bran wants to deploy his application using Radius. He has authored a Bicep manifest that defines the resources for his application and a Bicep parameters file that specifies the parameters for the manifest. He runs the `rad bicep generate bicepdeployment app.bicep --parameters app.bicepparam` command on his machine to create an BicepDeployment resource that contains the template and parameters specified. He then commits the BicepDeployment resource to the Git repository. |
There was a problem hiding this comment.
What is the "BicepDeployment resource" that is created? Is it a CRD?
There was a problem hiding this comment.
I'll update this to be more clear that rad bicep generate bicepdeployment creates a Kubernetes custom resource
|
|
||
| Bran is a developer who uses GitOps to automate the deployment of Radius applications to production Kubernetes clusters. His infrastructure team has set up a Radius installation and environment on the production cluster, and Bran wants to deploy his application using Radius. He has authored a Bicep manifest that defines the resources for his application and a Bicep parameters file that specifies the parameters for the manifest. He runs the `rad bicep generate bicepdeployment app.bicep --parameters app.bicepparam` command on his machine to create an BicepDeployment resource that contains the template and parameters specified. He then commits the BicepDeployment resource to the Git repository. | ||
|
|
||
| When the GitOps tool watches the repository and applies the BicepDeployment resource to the production cluster, the Radius resources defined in the `app.bicep` Bicep manifest are deployed. |
There was a problem hiding this comment.
Would Bran be using the production cluster or a dev cluster that was created in the previous scenario?
There was a problem hiding this comment.
In this case, a production cluster
|
|
||
| #### Sam can use GitOps to patch a Radius application in production | ||
|
|
||
| Sam is an SRE who is responsible for maintaining a Radius application in production. He notices that the application is experiencing performance issues and the number of application container replicas needs to be increased. He updates the Bicep parameters file for the application to specify the new number of replicas and runs the `rad bicep generate bicepdeployment app.bicep --parameters app.bicepparam` command on his machine to create an BicepDeployment resource that contains the updated parameters. He then commits the BicepDeployment resource to the Git repository. |
There was a problem hiding this comment.
Would replica count be defined with the application bicep?
There was a problem hiding this comment.
in the .bicepparams file
|
|
||
| // Parameters is the ARM JSON parameters for the template. | ||
| Parameters string `json:"parameters"` | ||
| } |
There was a problem hiding this comment.
We need to include the provider configuration here as well. This is where the scopes for providers like radius, azure, aws get passed in.
There was a problem hiding this comment.
Take a look at existing CLI code where we do rad deploy
\
|
|
||
| // Message is a human-readable description of the status of the Bicep Deployment. | ||
| Message string `json:"message,omitempty"` | ||
| } |
There was a problem hiding this comment.
Where are errors reported?
There was a problem hiding this comment.
update what gets described on kubectl get and kubectl describe
There was a problem hiding this comment.
Do we store UCP-structured error messages or parse it
|
|
||
| 1. Check if there is an in-progress operation. If so, check its status: | ||
| 1. If the operation is still in progress, then queue another reconcile operation and continue processing. | ||
| 2. If the operation completed successfully, then update the `status.phrase` field as `Ready`. Create `BicepResource` resources for each resource in the `outputResources` field returned by the Radius API. and set their OwnerReferences fields to the current `BicepDeployment` resource and their `status.phrase` fields to `Ready`. |
| 2. If the `BicepDeployment` is being deleted, then process deletion: | ||
| 1. Set the `status.phrase` for the BicepDeployment to `Deleting`. | ||
| 2. Check the cluster for and `BicepResource` resources that have this `BicepDeployment` as their owner. If found, set their `status.phrase` to `Deleting`. | ||
| 3. If there are no `BicepResource` resources found, then set the `status.phrase` for the BicepDeployment to `Deleted`. |
There was a problem hiding this comment.
Not sure if it was omitted as a simplification or not... What's the role of Kubernetes finalizers in this process?
There was a problem hiding this comment.
I can add it here
| rad bicep generate-kubernetes app.bicep --parameters @app.bicepparam --parameters tag=latest --parameters appnetworking.bicepparam --outfile app.yaml | ||
|
|
||
| Generating BicepDeployment resource... | ||
| BicepDeployment resource generated at app.yaml |
There was a problem hiding this comment.
Should we call this ARMDeployment instead of BicepDeployment? It contains ARM compilation from Bicep, but it is not Bicep.
There was a problem hiding this comment.
Please no references to a cloud provider-specific components in a cross-platform tool.
There was a problem hiding this comment.
Ideas:
ARMDeployment
TemplateDeployment
RadiusDeployment
BicepDeployment
There was a problem hiding this comment.
DeploymentTemplate
|
|
||
| ### Architecture Diagram | ||
|
|
||
|  |
There was a problem hiding this comment.
it took a while for me to see the difference between the create/update and delete operations in the diagram. The call to create the my-app-container. Do we want to call it out?
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
willdavsmith
changed the title
rynowak
left a comment
rynowak
left a comment
There was a problem hiding this comment.
One minor API design comment.
| The Radius Flux Controller will be a Kubernetes controller, based on the [Flux source watcher](https://fluxcd.io/flux/gitops-toolkit/source-watcher/) pattern to watch for changes to `.bicep` and `.bicepparam` files in the Flux source controller. When a change is detected, the controller will compile the Bicep manifests and create or update `BicepDeployment` resources on the Kubernetes cluster. The controller is responsible for compiling Bicep manifests, so it will need to construct a "file system" that the Bicep compiler can use to resolve dependencies. | ||
|
|
||
| Some edge cases to consider: | ||
| * If the bicep file was there, and now it's not, delete the exising CRD |
There was a problem hiding this comment.
This is supposed to delete the whole deployment and its resources, right?
|
|
||
| ## Today's Status | ||
|
|
||
| Today, Radius users are unable to deploy resources defined in Bicep manifests using Kubernetes tooling. This is because Kubernetes tools do not understand Bicep manifests. Users must use the Radius CLI to deploy resources defined in Bicep manifests. |
There was a problem hiding this comment.
This actually opens a new door for almost all users who deploy resources using Bicep files on to Kubernetes clusters.
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
Signed-off-by: Will Smith <willdavsmith@gmail.com>
Signed-off-by: willdavsmith <willdavsmith@gmail.com>
11 participants
Uh oh!
There was an error while loading. Please reload this page.