Radius Flux Controller Technical Design #79

willdavsmith · 2025-02-06T18:27:46Z

No description provided.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>

sk593 · 2025-02-06T19:13:36Z

tools/2025-01-gitops-technical-design.md

+
+1. The controller lists all `GitRepository` resources in the cluster.
+1. For each `GitRepository`, the controller parses the contents of the Git repository.
+1. The controller checks for a `radius-config.yaml` file in the repository. If the file is not present, then the controller skips the repository.


Will this be logged to the user that the repo was skipped?

It will be logged in the radius controller logs

sk593 · 2025-02-06T19:16:05Z

tools/2025-01-gitops-technical-design.md

+
+WORKDIR /
+
+RUN curl -Lo bicep https://github.com/Azure/bicep/releases/latest/download/bicep-linux-x64 \


Have we thought about supporting different architectures? This is scoped to only Linux images

yes, the design should support all architectures. this is a typo

Or cases where the user may want to bring their own specific version of Bicep?

willtsai · 2025-02-06T19:16:46Z

tools/2025-01-gitops-technical-design.md

+
+- Supporting non-Flux GitOps tools (e.g., ArgoCD).
+- Supporting non-Bicep IaC formats.
+- Radius Dashboard support


Can you elaborate on what specifically won't be supported in Radius Dashboard? I'm assuming gitops "clickops" via dashboard UI won't be supported, but things like App Graph will still get updated when resources are changed by Flux-triggered operations?

I'll elaborate here. I meant that "Radius GitOps" (status of DeploymentTemplate resources, GitRepositories, etc.) will not be tracked in the dashboard as part of this work. Everything Radius users have today will still be supported.

kachawla · 2025-02-06T19:17:41Z

tools/2025-01-gitops-technical-design.md

+
+The Radius Flux Controller will be implemented as a new controller in the `controller` service. Here is the high-level control loop of the new controller:
+
+1. The controller lists all `GitRepository` resources in the cluster.


Why does it need to list all the resources?

There's no way of knowing if the git repository contains Radius definitions (Bicep files) without listing them. If the controller doesn't have access to a GitRepository on the cluster, then that sounds like user error.

I think Kubernetes API Server will trigger this controller once there is a creation or an update of a GitRepository resource. Please correct me if I am wrong.

~~Does the controller act like a cron job and lists all the GitRepository resources to check if there are differences?~~

I should update this to be more clear.

kachawla · 2025-02-06T19:19:28Z

tools/2025-01-gitops-technical-design.md

+1. The controller compiles the Bicep file using the Bicep CLI, which is installed on the controller.
+1. The controller creates (or updates) a `DeploymentTemplate` resource on the cluster with the compiled Bicep file as the template.
+1. The controller lists all `DeploymentTemplate` resources in the cluster that have been created from this Git repository and deletes any resources that are not present in the Git repository.
+1. The controller repeats this process at a regular interval to ensure that the resources in the Git repository are accurately reflected in the Kubernetes cluster.


What is the value for the interval?

I should be more clear about what's happening here. We don't define the interval - it's handled by the kubernetes controller internals. this controller is set up to watch changes on GitRepository resources on the cluster, so it will trigger a Reconcile if one of these resources changes.

ytimocin · 2025-02-06T19:16:46Z

tools/2025-01-gitops-technical-design.md

+1. The controller reads the `radius-config.yaml` file and extracts the Bicep compilation instructions, such as the Bicep file path and parameters.
+1. The controller compiles the Bicep file using the Bicep CLI, which is installed on the controller.
+1. The controller creates (or updates) a `DeploymentTemplate` resource on the cluster with the compiled Bicep file as the template.
+1. The controller lists all `DeploymentTemplate` resources in the cluster that have been created from this Git repository and deletes any resources that are not present in the Git repository.


Why do we need to list all DeploymentTemplate resources for the given Git repository? Aren't just the last one and the one before that enough?

There may be multiple DeploymentTemplates that get created from one git repository (multiple root bicep files). So we need to handle the case where 2 are created on the first commit, then one is removed.

ytimocin · 2025-02-06T19:20:32Z

tools/2025-01-gitops-technical-design.md

+  providers:
+    - radius:
+        scope: /planes/radius/local/resourcegroups/default
+    - aws:
+        scope: /planes/aws/aws/accounts/000000000/regions/us-west-2
+    - azure:
+        scope: 	/planes/azure/azurecloud/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-group


How do we access the credentials for the providers? Does the code go and check the Radius environment for that?

if an environment is present already with the right configuration (credentials, etc.) the deployments will use them as normal.

ytimocin · 2025-02-06T19:21:36Z

tools/2025-01-gitops-technical-design.md

+
+The controller will list all `DeploymentTemplate` resources in the cluster that have been created from this Git repository and delete any resources that are not present in the Git repository:
+```go
+	// List all DeploymentTemplates on the cluster that are from the same git repository


Are we talking about older versions of a DeploymentTemplate for a given repository?

Can a repository have multiple DeploymentTemplates?

Yes, it can have multiple.

ytimocin · 2025-02-06T19:22:38Z

tools/2025-01-gitops-technical-design.md

+	// For all of the DeploymentTemplates on the cluster, check if the bicep file
+	// that it was created from is still present in the git repository. If not, delete the
+	// DeploymentTemplate from the cluster.


If this the case, we only keep the last DeploymentTemplate that is applied to the cluster, right?

in this case (the bicep file is deleted from the git repository), we should delete the deploymentTemplate which will delete the resources.

willtsai · 2025-02-06T19:24:36Z

tools/2025-01-gitops-technical-design.md

 -->

-!Architecture Diagram
+![Architecture Diagram Part 1](2025-01-gitops-technical-design/rfc-1.png)


are there changes needed to be made to the Flux codebase for detecting *.bicep file changes, or can this be done through configuration?

We handle this in the controller code by detecting the radius-gitops-config.yaml

zachcasper · 2025-02-06T19:15:03Z

tools/2025-01-gitops-technical-design.md

+-->
+
+- **GitOps**: A set of practices to manage infrastructure and application configurations using Git.
+- **Bicep**: A domain-specific language (DSL) for deploying Azure resources declaratively.


Suggested change

- **Bicep**: A domain-specific language (DSL) for deploying Azure resources declaratively.

- **Bicep**: A domain-specific language (DSL) for managing resources; originally developed to deploy Azure resources declaratively, Radius uses Bicep to model applications and resources agnostic of a cloud provider.

zachcasper · 2025-02-06T19:16:46Z

tools/2025-01-gitops-technical-design.md

+-->
+
+- Enable seamless integration of GitOps workflows with Radius + Bicep. Users should be able to specify their Bicep files and parameters in a Git repository and have the resources within them automatically deployed to their Kubernetes cluster.
+- Ensure that the infrastructure defined in the Git repository is accurately reflected in the Kubernetes cluster. This includes creating, updating, and deleting resources as necessary, based on the changes in the Git repository.


Very quickly "the Kubernetes cluster" will be not precise enough as we will support deploying to external Kubernetes clusters. This fact made me question where the CRs are created, in the cluster hosting Radius or in the cluster hosting the application.

the CRs should be created on the cluster hosting radius. this should enable us to do things like multicluster or other platforms (serverless)

zachcasper · 2025-02-06T19:18:04Z

tools/2025-01-gitops-technical-design.md

+-->
+
+- Supporting non-Flux GitOps tools (e.g., ArgoCD).
+- Supporting non-Bicep IaC formats.


Radius only supports Bicep for application definitions but supports Bicep and Terraform for recipes. Does this solution support Terraform recipes?

true, I'll re-word this

zachcasper · 2025-02-06T19:23:28Z

tools/2025-01-gitops-technical-design.md

+Include separate high level architecture diagram and component specific diagrams, wherever appropriate.
+-->
+
+![Architecture Diagram Part 1](2025-01-gitops-technical-design/rfc-1.png)


A proper sequence diagram would be easier to parse.

brooke-hamilton

This is super 💯

brooke-hamilton · 2025-02-06T19:16:24Z

tools/2025-01-gitops-technical-design.md

+
+#### Helly can use Radius and Bicep to simplify application and infrastructure management
+
+Helly is an infrastructure operator for an enterprise company. Her team manages a production application that is deployed on Kubernetes and uses AWS resources. To simplify the management of the application and infrastructure, Helly decides to use Radius and Bicep. She writes Bicep manifests that define the AWS resources required by the application, such as S3 buckets and RDS databases, as well as Radius resources that define container applications. Helly then uploads these Bicep files to a Git repository. She then writes a Flux configuration which includes Flux installation, Radius installation, and Flux setup for tracking her Git repository. Finally, Helly applies the Flux configuration to her Kubernetes cluster, and Radius deploys the resources defined in the Bicep manifests to the cluster. Now, any changes to the Bicep files in the Git repository are automatically reflected in the Kubernetes cluster, enabling a seamless GitOps workflow.


Bootstrapping in which Flux deploys Radius, which then receives notifications from Flux to deploy Bicep: is this a supported scenario?

yes, this is already supported because Radius can be installed by a Helm chart

brooke-hamilton · 2025-02-06T19:21:07Z

tools/2025-01-gitops-technical-design.md

+considered during the design process.
+-->
+
+The Radius Flux Controller will be implemented as a new controller in the `controller` service. Here is the high-level control loop of the new controller:


I used the Radius Support Agent to generate a mermaid diagram for this because it helps me visualize the flow. Do not feel compelled to use it, but here it is:

sequenceDiagram participant Controller participant GitRepository participant BicepCLI participant DeploymentTemplate Controller->>GitRepository: List all GitRepository resources loop For each GitRepository Controller->>GitRepository: Parse contents of the Git repository alt radius-config.yaml present Controller->>GitRepository: Read radius-config.yaml Controller->>BicepCLI: Extract Bicep compilation instructions BicepCLI->>Controller: Compile Bicep file Controller->>DeploymentTemplate: Create/Update DeploymentTemplate resource Controller->>DeploymentTemplate: List all DeploymentTemplate resources Controller->>DeploymentTemplate: Delete resources not present in Git repository else radius-config.yaml not present Controller->>GitRepository: Skip repository end end Controller->>Controller: Repeat at regular intervals

Loading

brooke-hamilton · 2025-02-06T19:23:48Z

tools/2025-01-gitops-technical-design.md

+
+The controller is set up to watch `GitRepository` resources in the cluster:
+```go
+func (r *FluxController) SetupWithManager(mgr ctrl.Manager) error {


How is this code implementing or using the Flux GitOps toolkit referenced above?

the ArchiveFetcher is from the flux gitops toolkit. that's the integration point with the flux source controller

brooke-hamilton · 2025-02-06T19:24:48Z

tools/2025-01-gitops-technical-design.md

+```
+
+When a `GitRepository` resource is created or updated, the controller will get the object and read the contents of the Git repository:
+```go 


Is this part of the SetupWithManager function?

this is in the Reconcile() function. I'll make this more clear

kachawla · 2025-02-06T19:28:09Z

tools/2025-01-gitops-technical-design.md

+-->
+
+- **Unit Tests**: Test the individual components of the Radius Flux Controller. There are many external dependencies that will need to be mocked, such as GitRepository objects and Bicep compilation.
+- **Functional Tests**: Test the end-to-end functionality of the Radius Flux Controller together with the DeploymentTemplate controller. This will involve creating GitRepository objects on a real Kubernetes cluster and verifying that the resources in the Bicep templates are created and updated correctly.


Thanks for including test plan section. Do we need to think about any error scenarios and performance issues for large git repos?

this is a good callout - In the controller code we can configure the MaxDownloadSize and MaxUntarSize both currently set to Unlimited by default. I can do some more testing on this

kachawla · 2025-02-06T19:29:12Z

tools/2025-01-gitops-technical-design.md

+with the instrumentation.
+-->
+
+The Radius Flux Controller will be setup as another controller in the Radius controller process. It will use standard Radius and Kubernetes logging and monitoring facilities, such as Kubernetes events and logs, to provide visibility into its actions. Users will be able to troubleshoot the Radius Flux Controller by examining the logs and events generated by the controller, as is the case with other Kubernetes controllers.


Users will be able to troubleshoot the Radius Flux Controller by examining the logs and events generated by the controller

Will be helpful if we document this.

Added to the docs

willdavsmith · 2025-02-06T19:51:16Z

tools/2025-01-gitops-technical-design.md

+them here. Provide a brief explanation on why this is a non-goal.
+-->
+
+- Supporting non-Flux GitOps tools (e.g., ArgoCD).


Non-goal for Radius helm support

brooke-hamilton

This is super 💯

kachawla · 2025-02-06T20:04:42Z

tools/2025-01-gitops-technical-design.md

+  providers:
+    - radius:
+        scope: /planes/radius/local/resourcegroups/default
+    - aws:
+        scope: /planes/aws/aws/accounts/000000000/regions/us-west-2
+    - azure:
+        scope: 	/planes/azure/azurecloud/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-group


What's the motivation for adding providers here, given that they are already expected to be configured in env.bicep?

good point, removed

willdavsmith · 2025-02-06T20:07:09Z

tools/2025-01-gitops-technical-design.md

+  }
+```
+
+The `radius-config.yaml` file will have the following structure:


we can rename this... probably something like "radius" "gitops" "flux"

renamed to radius-gitops-config.yaml

github-actions · 2025-02-21T00:21:43Z

This pull request is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions · 2025-02-28T00:22:08Z

This pull request has been closed due to inactivity. Feel free to reopen if you are still working on it.

Signed-off-by: willdavsmith <willdavsmith@gmail.com>

brooke-hamilton

# Description This PR adds a pull and download of the Bicep CLI into a container as part of the work to enable the Radius + GitOps workflow. Design notes: radius-project/design-notes#79 ## Type of change  - This pull request adds or changes features of Radius and has an approved issue (issue link required).  Part of: #6689 ## Contributor checklist Please verify that the PR meets the following requirements, where applicable: - An overview of proposed schema changes is included in a linked GitHub issue. - [ ] Yes - [x] Not applicable - A design document PR is created in the [design-notes repository](https://github.com/radius-project/design-notes/), if new APIs are being introduced. - [x] Yes - [ ] Not applicable - The design document has been reviewed and approved by Radius maintainers/approvers. - [x] Yes - [ ] Not applicable - A PR for the [samples repository](https://github.com/radius-project/samples) is created, if existing samples are affected by the changes in this PR. - [ ] Yes - [x] Not applicable - A PR for the [documentation repository](https://github.com/radius-project/docs) is created, if the changes in this PR affect the documentation or any user facing updates are made. - [ ] Yes - [x] Not applicable - A PR for the [recipes repository](https://github.com/radius-project/recipes) is created, if existing recipes are affected by the changes in this PR. - [ ] Yes - [x] Not applicable --------- Signed-off-by: willdavsmith <willdavsmith@gmail.com>

# Description * Add Radius Flux controller and tests * Update test workflows to install and use Flux and Gitea * Add bicep container and changes to allow for Bicep to run server-side in `controller` pod Note: I expect the functional tests to fail for this PR because I'm updating the functional test workflow. Here's a link to a successful action run: https://github.com/radius-project/radius/actions/runs/14074009492 Design PR: radius-project/design-notes#79 Docs PR: radius-project/docs#1408 ## Type of change  - This pull request fixes a bug in Radius and has an approved issue (issue link required). - This pull request adds or changes features of Radius and has an approved issue (issue link required). - This pull request is a minor refactor, code cleanup, test improvement, or other maintenance task and doesn't change the functionality of Radius (issue link optional).  Fixes: #6689 ## Contributor checklist Please verify that the PR meets the following requirements, where applicable: - An overview of proposed schema changes is included in a linked GitHub issue. - [ ] Yes - [x] Not applicable - A design document PR is created in the [design-notes repository](https://github.com/radius-project/design-notes/), if new APIs are being introduced. - [x] Yes - [ ] Not applicable - The design document has been reviewed and approved by Radius maintainers/approvers. - [x] Yes - [ ] Not applicable - A PR for the [samples repository](https://github.com/radius-project/samples) is created, if existing samples are affected by the changes in this PR. - [ ] Yes - [x] Not applicable - A PR for the [documentation repository](https://github.com/radius-project/docs) is created, if the changes in this PR affect the documentation or any user facing updates are made. - [x] Yes - [ ] Not applicable - A PR for the [recipes repository](https://github.com/radius-project/recipes) is created, if existing recipes are affected by the changes in this PR. - [ ] Yes - [x] Not applicable --------- Signed-off-by: willdavsmith <willdavsmith@gmail.com>

Signed-off-by: willdavsmith <willdavsmith@gmail.com>

# Description This PR adds a pull and download of the Bicep CLI into a container as part of the work to enable the Radius + GitOps workflow. Design notes: radius-project/design-notes#79 ## Type of change  - This pull request adds or changes features of Radius and has an approved issue (issue link required).  Part of: #6689 ## Contributor checklist Please verify that the PR meets the following requirements, where applicable: - An overview of proposed schema changes is included in a linked GitHub issue. - [ ] Yes - [x] Not applicable - A design document PR is created in the [design-notes repository](https://github.com/radius-project/design-notes/), if new APIs are being introduced. - [x] Yes - [ ] Not applicable - The design document has been reviewed and approved by Radius maintainers/approvers. - [x] Yes - [ ] Not applicable - A PR for the [samples repository](https://github.com/radius-project/samples) is created, if existing samples are affected by the changes in this PR. - [ ] Yes - [x] Not applicable - A PR for the [documentation repository](https://github.com/radius-project/docs) is created, if the changes in this PR affect the documentation or any user facing updates are made. - [ ] Yes - [x] Not applicable - A PR for the [recipes repository](https://github.com/radius-project/recipes) is created, if existing recipes are affected by the changes in this PR. - [ ] Yes - [x] Not applicable --------- Signed-off-by: willdavsmith <willdavsmith@gmail.com>

# Description * Add Radius Flux controller and tests * Update test workflows to install and use Flux and Gitea * Add bicep container and changes to allow for Bicep to run server-side in `controller` pod Note: I expect the functional tests to fail for this PR because I'm updating the functional test workflow. Here's a link to a successful action run: https://github.com/radius-project/radius/actions/runs/14074009492 Design PR: radius-project/design-notes#79 Docs PR: radius-project/docs#1408 ## Type of change  - This pull request fixes a bug in Radius and has an approved issue (issue link required). - This pull request adds or changes features of Radius and has an approved issue (issue link required). - This pull request is a minor refactor, code cleanup, test improvement, or other maintenance task and doesn't change the functionality of Radius (issue link optional).  Fixes: #6689 ## Contributor checklist Please verify that the PR meets the following requirements, where applicable: - An overview of proposed schema changes is included in a linked GitHub issue. - [ ] Yes - [x] Not applicable - A design document PR is created in the [design-notes repository](https://github.com/radius-project/design-notes/), if new APIs are being introduced. - [x] Yes - [ ] Not applicable - The design document has been reviewed and approved by Radius maintainers/approvers. - [x] Yes - [ ] Not applicable - A PR for the [samples repository](https://github.com/radius-project/samples) is created, if existing samples are affected by the changes in this PR. - [ ] Yes - [x] Not applicable - A PR for the [documentation repository](https://github.com/radius-project/docs) is created, if the changes in this PR affect the documentation or any user facing updates are made. - [x] Yes - [ ] Not applicable - A PR for the [recipes repository](https://github.com/radius-project/recipes) is created, if existing recipes are affected by the changes in this PR. - [ ] Yes - [x] Not applicable --------- Signed-off-by: willdavsmith <willdavsmith@gmail.com>

willdavsmith added 2 commits February 3, 2025 13:32

Thank you Copilot

6415d14

Signed-off-by: willdavsmith <willdavsmith@gmail.com>

wip

fa54e8e

Signed-off-by: willdavsmith <willdavsmith@gmail.com>

willdavsmith requested review from a team as code owners February 6, 2025 18:27

sk593 reviewed Feb 6, 2025

View reviewed changes

willtsai reviewed Feb 6, 2025

View reviewed changes

kachawla reviewed Feb 6, 2025

View reviewed changes

ytimocin reviewed Feb 6, 2025

View reviewed changes

willtsai reviewed Feb 6, 2025

View reviewed changes

zachcasper reviewed Feb 6, 2025

View reviewed changes

brooke-hamilton reviewed Feb 6, 2025

View reviewed changes

kachawla reviewed Feb 6, 2025

View reviewed changes

willdavsmith commented Feb 6, 2025

View reviewed changes

brooke-hamilton approved these changes Feb 6, 2025

View reviewed changes

kachawla reviewed Feb 6, 2025

View reviewed changes

willdavsmith commented Feb 6, 2025

View reviewed changes

github-actions bot added the Stale label Feb 21, 2025

github-actions bot closed this Feb 28, 2025

design review comments

b644d96

Signed-off-by: willdavsmith <willdavsmith@gmail.com>

willdavsmith reopened this Mar 27, 2025

willdavsmith added 2 commits March 27, 2025 09:37

Merge branch 'main' into fluxcontroller

24a1153

spelling

c3acec0

Signed-off-by: willdavsmith <willdavsmith@gmail.com>

willdavsmith removed the Stale label Mar 27, 2025

willdavsmith mentioned this pull request Mar 27, 2025

Add Radius Flux Controller for GitOps support radius-project/radius#8784

Merged

12 tasks

brooke-hamilton approved these changes Mar 27, 2025

View reviewed changes

sk593 approved these changes Mar 27, 2025

View reviewed changes

willdavsmith mentioned this pull request Mar 31, 2025

Add build and publish for Radius Bicep container image radius-project/radius#9039

Merged

12 tasks

sylvainsf approved these changes Mar 31, 2025

View reviewed changes

sylvainsf merged commit 4f7d842 into radius-project:main Mar 31, 2025
2 checks passed

brooke-hamilton pushed a commit to brooke-hamilton/radius-design-notes that referenced this pull request Apr 4, 2025

Radius Flux Controller Technical Design (radius-project#79)

3c953ca

Signed-off-by: willdavsmith <willdavsmith@gmail.com>


		WORKDIR /

		RUN curl -Lo bicep https://github.com/Azure/bicep/releases/latest/download/bicep-linux-x64 \


		The Radius Flux Controller will be implemented as a new controller in the `controller` service. Here is the high-level control loop of the new controller:

		1. The controller lists all `GitRepository` resources in the cluster.

	- Bicep: A domain-specific language (DSL) for deploying Azure resources declaratively.
	- Bicep: A domain-specific language (DSL) for managing resources; originally developed to deploy Azure resources declaratively, Radius uses Bicep to model applications and resources agnostic of a cloud provider.


		#### Helly can use Radius and Bicep to simplify application and infrastructure management

		Helly is an infrastructure operator for an enterprise company. Her team manages a production application that is deployed on Kubernetes and uses AWS resources. To simplify the management of the application and infrastructure, Helly decides to use Radius and Bicep. She writes Bicep manifests that define the AWS resources required by the application, such as S3 buckets and RDS databases, as well as Radius resources that define container applications. Helly then uploads these Bicep files to a Git repository. She then writes a Flux configuration which includes Flux installation, Radius installation, and Flux setup for tracking her Git repository. Finally, Helly applies the Flux configuration to her Kubernetes cluster, and Radius deploys the resources defined in the Bicep manifests to the cluster. Now, any changes to the Bicep files in the Git repository are automatically reflected in the Kubernetes cluster, enabling a seamless GitOps workflow.

Radius Flux Controller Technical Design #79

Radius Flux Controller Technical Design #79

Uh oh!

Conversation

willdavsmith commented Feb 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ytimocin Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brooke-hamilton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ytimocin Feb 6, 2025 •

edited

Loading