Kustomize/Helm Cleanup / Support Multiple ImagePullSecrets

[toddbaert]

As discussed in #655, the fact we use both Kustomize and Helm causes us some pain and limitations. We should see if we can completely remove kustomize in favor of pure Helm, and as a first-fruit of this effort, allow multiple ImagePullSecrets to be defined (instead of the current limitation to one).

One request I'll have is to add supporting the array of imagePullSecrets to the backlog for OpenFeature. Inevitably the build process will change and evolve, and when it does, this use-case will just become part of it.

Originally posted by @cpitstick-latai in #655 (comment)

[cpitstick-latai]

What would the removal of the kustomize layer look like? I feel like it's there for a reason, and there will likely be dragons in that removal process...

[toddbaert]

@thisthat and I spend some time looking into this. I actually don't think it's reasonably possible right now unless we want to manually maintain a chart (which I'm really reluctant to do).

The main issue is that kubebuilder is the thing that generates our kustomize layer from go-code (which is really useful). There's no way to have kubebuilder generate helm YET, but there is a feature/plugin in discussion.

I think for now, we may have to settle for my most recent hack, which doesn't solve the root of the mess but at least allows for multiple ImagePullSecrets.