Make sidecar CMPs easier to configure and develop

[crenshaw-dev]

Summary

argocd-cm plugins were awesome, because you could simply modify the ConfigMap and immediately use the plugin, as configured. They sucked because they offered no filesystem isolation from the repo-server and therefore were relatively insecure.

Sidecar plugins are awesome, because they're isolated and much more secure. They suck because 1) they're difficult to initially set up, and 2) you have to restart the sidecar to load a new plugin config (makes development/iteration a pain).

We should make it easy to set up a basic plugin.

Motivation

Installing and developing sidecar CMPs is painful.

Proposal

1. let the user define the CMP config in argocd-cm, just like they could with old-style plugins
2. send that config to repo-server when the application-controller or api-server request manifests
3. send that config from the repo-server to the sidecar
4. have the sidecar use that config to generate manifests
5. add a hash of the config to the manifests cache key, so there's no need to hard-refresh after changing the plugin config

Bonus points:

Create an opt-in Kustomize patch in the official manifests that adds a basic, properly-configured sidecar using the Argo CD image. Best practices defined in the CMP docs can be applied. We could even set up the basic Helm env vars by default, since a lot of CMP users want to use Helm.

[rufreakde]

Linking here:
argoproj/argocd-example-apps#235

[mhjacks]

I'm working on updating CMP support to 2.0 in the Validated Patterns effort (https://validatedpatterns.io), and I was encouraged to comment here.

Validated Patterns are opinionated, declarative reference architectures. The idea is that we provide a way of expressing reference architectures that can be run through CI/CD because the primary deliverable is a gitops repo, that can then be further customized or tailored.

One of our released Validated Patterns uses a custom CMP (which combines helm and kustomize), that is currently configured using the old, argocd-cm mechanism. I personally dislike putting things in argocd-cm, because caring about some of it makes you care about quite a lot if it if your initial installation is via helm and you want it to be declarative.

Otherwise, the CMP 1.0 approach was a bit problematic because, if I understand it right, custom tooling required a custom image in place of the repo server, which is undesirable for a lot of reasons.

The CMP 2.0 mechanism is superior in that it removes the CMP plugin config from argocd-cm, but currently mandates an image to be specified, which the old mechanism couldn't do.

When the use case is to combine helm and kustomize in new ways, you don't necessarily need anything outside of the standard repo-server image; and so to me it would be nice if the operator could infer this by default for plugins. Obviously, if the user needs something that isn't provided in the repo-server image, they would need to supply their own image, and that is perfectly reasonable.

I have heard also that there are discussions about factoring out helm and kustomize from the repo-server image and making them plugins. This is an interesting proposal; as a user of the feature I would just ask that the tooling image be easily and readily accessible for customization (or straightforwardly to include implicitly as a sidecar). In the OpenShift GitOps productization, the argocd kind has a field, sidecarContainers, for specifying custom plugins. I think it would cause more angst if the semantics of that field were to change (i.e. to add helm/kustomize plugins there by default, where they were implicit before).

[crenshaw-dev]

Well said! Will respond to just a couple pieces:

it would be nice if the operator could infer this by default for plugins

Agreed, that would be much more user-friendly. In fact, auto-configuring the sidecars in general would be a great help. To make things GitOps-friendly, I don't think we should have some process configuring the repo-server deployment at runtime to add more sidecars. But I do think a CLI tool to patch your argocd-repo-server Deployment manifest to add a plugin would be nice.

I have heard also that there are discussions about factoring out helm and kustomize from the repo-server image and making them plugins.

I do think that this is currently a long way out. But we definitely need to carefully consider the UX if/when we make that decision.

[mhjacks]

Thanks for the response! I'll post a link to our updated framework with support for both mechanisms here when that's done. I expect to have it done in the next week or so.

Agreed, that would be much more user-friendly. In fact, auto-configuring the sidecars in general would be a great help. To make things GitOps-friendly, I don't think we should have some process configuring the repo-server deployment at runtime to add more sidecars. But I do think a CLI tool to patch your argocd-repo-server Deployment manifest to add a plugin would be nice.

There is a bit of repo-server "magic" going on that one definitely needs to understand, it seems, to make effective use of this feature.

I do think that this is currently a long way out. But we definitely need to carefully consider the UX if/when we make that decision.

Thanks, that's good to hear. I understand the security concerns and I'm sensitive to the notion of separating concerns; it's also important (I think) to be able to unwind and deconstruct how the manifest are actually being rendered in the case of surprises.

I very much like the fact that this functionality is possible with ArgoCD; it would be great if we could make it easier/more accessible; and I think that would be a function of understanding the common use cases and optimizing for them (assuming we could seamlessly enable the more complex stuff that would involve all the elements). The complexities are greater in the OpenShift productization because we gate the downloads of the repo-server image, so making it implicit eliminates several potential pain points for customers who would use it.

[crenshaw-dev]

it's also important (I think) to be able to unwind and deconstruct how the manifest are actually being rendered in the case of surprises.

This should be carved in stone somewhere. :-)