Redefining Credential Management: Implementing a Pull-Model Credentials Service for Enhanced Security and Scalability

[drpaneas]

Summary

The proposal is to implement a new Credentials Service in ArgoCD to improve the management of repository credentials. This would transition the repo-server from a "push-model" approach to a "pull-model" approach.

Motivation

Currently, repository credentials are handled in a "push-model" approach, where credentials are sent to the repo-server as needed. However, this approach isn't scalable, and it doesn't treat the external plugins as first-class citizens in the ArgoCD ecosystem.

This becomes especially problematic when we consider diverse tools like Helm, Kustomize, and others that could potentially be integrated through the same plugin mechanism, thus not considered anymore as "core management tools" for ArgoCD.

see also: #10265

Proposal

The proposed solution is to implement a new gRPC-based "credentials" service within ArgoCD. This service would be responsible for managing and providing repository credentials to plugins based on requests.

We need to set up a system where our repo-server can pull creds instead of being spoon-fed. Picture this: our plugin goes over to Mr. Repo-Server and politely requests, "Hey, I need access to this $PRIV_REPO. You got the keys or what?" No more playing favorites with the argocd native management tools.

Implementation would follow these steps:

1. Get the gRPC service and messages set up: define the gRPC service and messages through .proto files, generating Go code to define our service, messages, etc.
2. Implement the credentials service: handle authentication, authorization, and the logic for secret-fetching (that means: retrieving and providing the right credentials to the plugins based on a provided JWT(.
3. Integrate the service with the repo-server: get the repo-server to register our brand new thing: the credentials service. This will allow to handle gRPC requests from the plugins.
4. Authorize the plugins: Implement JWT-based authentication to ensure only the rightful plugins access the necessary credentials, using a reliable JWT library (such as: https://github.com/golang-jwt/jwt)
5. Find the secret: Write code within the credentials service to access the Secrets stored in the argocd namespace, employing the Kubernetes client library for this purpose.
6. Add some QA: Finally, test the Credentials Service thoroughly in multiple scenarios to ensure its reliability and efficiency.

We could go for an alternative implementation, where we just dump all creds into a directory and let the plugin deal with them. Easy? Yes. Secure? No. We don't want to make it too easy and end up being a demo for defcon, do we?

Suggestions

1. We already have a git-centric credential helper that offers gRPC API, but doesn't provide any AuthN or AuthZ. Maybe there is some code there which can be borrowed instead of starting completely from scratch.
2. The JWTs would have to be issued by both the API server and the application controller. This is because both request manifests. Thankfully both have access to secrets, so they should be able to sign the JWT based on a shared secret.
3. The mechanism for exposing credentials via the plugins needs to be easy to use. That means you don't provide different documentation and support for each and every kind of helm-like took out there, but hide/abstract the backend through a a dumb CLI. The plugin author can use the CLI to "ask" for a specific credential. Then it's up to them to place that credential where it's needed.
4. The repo-server has no access to Kubernetes cluster, so how are you going to search for the secret? Well, the credential service should be on its own pod/SA, so that only this one has access to k8s secrets. Not the repo-server. I repeat: do not give access to the repo-server to talk to the cluster.

I know you might don't like proposal this because it introduces yet another new deployment, which is somewhat complex, but provides improved security and scalability over the current implementation. The timeline for the proposal's implementation is approximately three months, allowing for design, prototyping, integration, testing, and documentation.

Credits

Although I am the one creating this enhancement proposal, I would like to give credits to @crenshaw-dev for his insights and ideas and to @jannfis for bringing up great questions concerning the security aspect of it.

[Joibel]

I could like to use this for crumbhole/argocd-lovely-plugin#36.