kargo sharding configuration

[prudnitskiy]

Hello community,

I'm evaluating the Kargo for our setup and got the elephant in the room: In most cases, a sharding configuration is required, but unfortunately, it is not widely documented. The kargo itself moves freight through steps, including analysis to verify if the step was successful or not. To be able to confirm if the step was successful, the Kargo should be able to run a job on a target cluster. Do I understand the setup considerations correctly here?

• The API server manages jobs and distributes them over shards (including validation)
• The shard controller (let's call it satellite) must access the API server to get tasks scheduled and post status updates on task progress
• The satellite controller uses the kubeconfig file provided via deployment to access the API cluster. The credentials in kubeconfig must provide the necessary permissions to read CRDs on the project in the main cluster.
• The one and only way to provide information to the satellite on how to access API server is the kubeconfig file.
• In general, it means the main cluster KubeAPI should be accessible from the satellite, and the SA logs into the satellite should have enough permissions to operate with CRDs on the main server

Or am I missing something? I did some research, and it looks like some people achieved sharded installation, but I do not understand what the reference design here is.

Thank you in advance!

[krancour]

The API server manages jobs and distributes them over shards (including validation)

Not quite true. The API server has no agency. It's just an bridge from the UI/CLI to Kargo resources that doesn't require giving out cluster credentials to every user.

The shard controller (let's call it satellite) must access the API server to get tasks scheduled and post status updates on task progress

Correct. From other statements (below), I think you understand this already, but a minor point of clarification for the benefit of other readers. It is the Kubernetes API server it communicates with, and not the Kargo API server.

The satellite controller uses the kubeconfig file provided via deployment to access the API cluster. The credentials in kubeconfig must provide the necessary permissions to read CRDs on the project in the main cluster.

The one and only way to provide information to the satellite on how to access API server is the kubeconfig file.

In general, it means the main cluster KubeAPI should be accessible from the satellite, and the SA logs into the satellite should have enough permissions to operate with CRDs on the main server

All of these are correct.

it looks like some people achieved sharded installation, but I do not understand what the reference design here is.

Your understanding of how it's done is entirely correct. It is, however, not easy to achieve this at scale. That's not really a Kargo problem so much as it is simply a pain in the butt to manage and distribute the appropriate kubeconfigs. It's deliberate that we don't explicitly document how to do this or how to automate it at scale. We built a SaaS that does that. 😉