[feat req] process crds before any helm installs

[jr0dd]

This is more of a feature request than an issue. I have been using a separate crds directory to make sure the crds exist, in a fresh cluster, prior reconciling helmrelease files. This can get very cumbersome, having to create kustomizations & git repositories resources for crd manifests in addition to the helmreleases, etc etc. The best example for this is the prometheus-operator CR's. If servicemonitors and alert rules are defined in helmrelease values, they won't reconcile unless kube-prometheus-stack crd's get installed first. What would be a cleaner approach is having flux install all crd's first, if defined in the helmrelease like below, before doing any helm installs.

current approach

cluster/crds
├── external-dns
│   ├── crds.yaml
│   └── kustomization.yaml
├── kube-prometheus-stack
│   ├── crds.yaml
│   └── kustomization.yaml
└── kustomization.yaml

helm-release.yaml

  install:
    crds: CreateReplace
  upgrade:
    crds: CreateReplace

[kingdonb]

The Flux approach uses a Kustomization to install a HelmRelease, and those are two separate transactions in terms of server side apply. Since Flux added support for server side apply, Kustomize controller itself no longer has the problem of not being able to apply CRDs and CRs of those definitions at the same time due to validation issues. As you are experiencing the same issue in a different way, this is slim consolation and I'm sorry for the poor experience you're having!

Since Flux v0.18, the Kustomize controller is smart enough to apply CRDs using server-side, keeping the apply operation open, then using server-side to apply the instances of CRs, in the proper order so that validation succeeds where it might have failed like as in this case.

Kustomize controller however is not smart enough (nor is there a simple design or change which I can imagine that does not radically alter the way Flux works) to install a Kustomization, then wait for HelmRelease to become ready, and apply Custom Resources finally after that. Well, maybe there could be such an optional behavior added without compromising too much, speculating, but I think the complexity that it adds is not worth the benefit of being able to manage CRDs from within Helm.

I understand you're requesting a particular feature and you may be aware of workarounds already, but as an issue report it should be in scope to reply with workarounds... feature requests should properly go in flux2/discussions and are not issue reports. That does not preclude the possibility this might also be a valid issue report worthy of keeping open as an issue.

---

Here's my cleanest workaround / alternative configuration approach suggested example based on cert-manager: https://github.com/kingdonb/bootstrap-repo/blob/staging/apps/cert-manager/kustomization.yaml

Prom-operator doesn't publish the CRD as a separate release artifact so it's not so easy.

(Come to think of it, this strategy that I'm using to grab from releases might just as easily get rate limited too, and I think it has occurred at least once in a high-profile outage recently... it has worked for me and rarely contributes much noise to my Flux slack notifications. What would be better than using the artifact at an HTTP URL is using it from a GitRepository source, so that Flux can cache it and avoid fetching them again when the source hasn't changed.)

This specific approach will not work for prometheus-stack for prom-operator I'm afraid because they do not publish the CRDs in the same way at a release URL. You can still reference them directly as files from a raw github HTTP URL in a git branch or tag, but this is sub-optimal and seems likely to incur the wrath of rate limiting as it will not be cached if it's done this way in the Kustomize overlay.

This approach is IMHO the best for solving this particular problem:

https://github.com/kingdonb/bootstrap-repo/blob/staging/infrastructure/crds/traefik-crds.yaml

Create the CRDs in a Kustomization by themselves, then include them from the git repository source. You can use this example directly from Traefik, combine it with the CRDs from prometheus-operator that are published in their git repository:

https://github.com/prometheus-operator/prometheus-operator/tree/main/example/prometheus-operator-crd

Apply them in a Flux Kustomization by itself, then apply that Kustomization from another one called infrastructure as in the example of my bootstrap-repo. There is honestly quite a lot going on in my bootstrap-repo example repo, maybe it would be better if I linked to this version with a lot less going on in it, (still has the same example, we're installing Traefik CRDs in advance, then waiting for them to be ready before making downstreams apply): https://github.com/jakubhajek/traefik-gitops

I think the best thing to do if you don't like having two separate dependsOn Kustomizations, is to disregard the CRD installation that happens inside of the helm chart altogether and handle CRD installation in the Kustomize overlay instead.

tl;dr: cert-manager publishes the CRD as part of the release artifacts, which makes this really easy for us:

https://github.com/kingdonb/bootstrap-repo/blob/staging/apps/cert-manager/kustomization.yaml#L4

... but prometheus-operator does not. With a complete understanding of how Flux caches artifacts, it should be clear enough that the "traefik-crd" approach above is better than the "cert-manager" one in terms of caching and avoiding failed applies due to rate limiting or other temporary/intermittent kinds of remote availability failures.

[kingdonb]

I was going to lead with this info, but I wanted to say the workaround more concisely and get to the point, in case you're actively struggling with this, and that bifurcated into two ideas, so unfortunately this wall of text. Maybe better to start with: the classic solution to this issue uses Flux Kustomization's dependsOn: first install the HelmRelease, including CRDs there with install.crds and upgrade.crds set to CreateReplace just as you have done. In the Kustomization which installs it, set Kustomization.spec.wait so that Kustomization waits for the HelmRelease to become ready. This will signal that CRDs are applied and a second Kustomization can dependsOn the first one, now it will be able to apply its own CRs reliably.

Personally I think that while it's nice Helm Controller supports CRD operations that even Helm itself does not, it's working against the grain of the tool, and it's better to acknowledge that Helm itself just isn't very good at managing CRDs at all. If we can use a different tool to manage them it would be better, and realistically though I have an idea for how it could work, I think it unlikely to be adopted and more likely that Flux will never reach through Kustomization and wait for HelmReleases as I've described before, it's not currently smart enough for that, and would require Flux to treat HelmReleases specially again.

The approach is not much different, same as above: first install the CRDs, then install the instances, except that when it is all handled by Kustomize controller, you don't need a separate one or any dependsOn just for applying CRDs and CRs in the right order. So when the CRDs are applied by Kustomize Controller, the HelmRelease can still manage the helm installation, there is only one Flux Kustomization needed, and you can apply the CRs at the same time as the CRDs and HelmRelease.

(That strategy won't work if HelmRelease installs any admission webhooks that might have been required to validate the CRs. I don't think that's a concern for any of these, but maybe others truly need HelmRelease to be ready before CRs can be installed, else they fail validation by admission webhook... that's another complicated issue outside of what you asked.)

I have some ideas about how prometheus-operator could publish their CRDs differently to make it easier for Flux users to do this in whatever way suits them best; in my opinion the example of cert-manager is top notch, but that idea is not required if you use the solution above. Basically we need an HTTP url for the CRD so we can include it in a Kustomize Overlay, then the configuration required can be drastically simplified compared to above. This is harder than it needs to be, but maybe better this way for caching opportunities, and to avoid being rate limited later by Kustomize pulling the same HTTP url every time.

Utilities like prometheus-operator should provide better guidance about which CRDs to apply at install time and how to handle this. They do acknowledge the Helm chart for kube-prometheus-stack has CRDs which will helm does not manage with a note in the docs: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#upgrading-chart – I feel like this is one place where deferring to the Helm docs is quite a bit less than helpful, given Helm basically leaves it up to the chef in how to handle installing CRDs and upgrading them; Helm Controller users have the privilege of using CreateReplace strategy for both install and upgrade, but, prometheus-stack surely doesn't assume its users will be using Flux v2 and Helm Controller. It's not clear what other options there are, and it would be better if prom-operator explained specifically how to go about managing the CRDs as it suggests. (It seems they assumed you'd be smart enough to know that Helm's doc means you should apply the CRD at the latest version before or during every upgrade.)

The cert-manager installation provides the best example model for how prometheus-operator could change their docs here IMHO:
https://cert-manager.io/docs/installation/helm/#option-1-installing-crds-with-kubectl

They actually include this step in their Helm installation docs as Option 1; as I have suggested, they also suggest, don't even bother using Helm's support for CRDs. You can use it, but with Flux you're currently stuck with this issue as it's not smart enough to reach through the HelmRelease, and as I mentioned it probably never will be. (I should take a highlighter and note any forward-looking statements are my opinion and could be up for debate! It's certainly not up to me to decide...)

In Flux, if you take that option, just manage the CRDs outside of Helm altogether, you can get away with using a single Kustomization and a single HelmRelease. I don't have any complete example handy that actually uses HelmRelease too, maybe I should add one, but since we're just handling the CRD for ourselves that doesn't change much.

Hopefully this adds some additional color about the dimensions of this problem, and known possible solutions that are already extant for this issue. 👍 thanks for your report @jr0dd

[jr0dd]

@kingdonb I appreciate the thoroughness in your replies! I was reorganizing my repo last night and just thought no better time than now to bring for discussion. CRD management in k8s is just a mess in general.

After moving most out of the CRD directory in my repo last night I'm debating moving them all back for consistency. I just find it a little cumbersome and then having to create a regexManager for renovate to keep them up-to-date when they don't follow semver.

I didn't even realize flux had that CRD install/upgrade option until last night. Which is great if they are actually located in the crds dir and if no other helm releases need to install their CR's beforehand...

[kingdonb]

@jr0dd I should highlight this: https://github.com/jakubhajek/traefik-gitops/blob/18c3373233fb26997849d335e179623809d98393/infrastructure/crds/traefik-crds.yaml#L11-L16

spec:
  interval: 30m
  url: https://github.com/traefik/traefik-helm-chart.git
  ref:
    tag: v10.3.0
  ignore: |
    # exclude all
    /*
    # path to crds
    !/traefik/crds/

This is really the best example, when GitRepository has in spec.ref a semver tag, hypothetically you can even combine this with ImageUpdateAutomation so that CRD changes always get a commit in the repo. That would be a good example... 👍

This example also only filters the GitRepository source down to a single path, the contents of which are the CRDs to install. So what I was really suggesting as a solution for prometheus-operator or kube stack chart, is to use this same approach. Since the helm chart is published not only from a HelmRepo but also in the GitRepo, you can even use envsubst and a configmap to DRY up the repetition of the version number for semver tag.

So, something like this:

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: prom-stack-crds
  namespace: flux-system
spec:
  interval: 30m
  url: https://github.com/prometheus-community/helm-charts.git
  ref:
    tag: kube-prometheus-stack-${kube_prometheus_stack_ver}
  ignore: |
    # exclude all
    /*
    # path to crds
    !/charts/kube-prometheus-stack/crds/
---
apiVersion: v1
data:
  kube_prometheus_stack_ver: "26.1.0" # {"$imagepolicy": "magic"}
kind: ConfigMap
metadata:
  name: flux-config-postbuild-prom-stack-data
  namespace: flux-system

Or even more simply tag: kube-prometheus-stack-26.1.0 - how often you are updating the CRDs anyway, you probably want to do this explicitly and on purpose, without magic 👍

[jr0dd]

@kingdonb That is an interesting approach with the configmap and image automation controller. I'll play around with it some more this weekend!

[kingdonb]

Yeah as an approach it would be more usable if charts and app version were the same, but they're not likely to be the same in most cases, so you can't really use image automation for that... except in certain cases where maybe you can.

If it looks like it won't work, don't strain too hard at it. But the configmap + envsubst to avoid writing the same version number twice in two different manifest files (and inevitably forgetting to update one of them along the way somewhere) is definitely a winning strategy 👍

[kingdonb]

I think there's a pretty strong argument in terms of usability that if HelmReleases are going to be able to manage CRDs, then there ought to be a special case of Kustomization which waits for a HelmRelease before applying from a bucket of CRs.

It's not that different than what the Flux Kustomization does now, it enables a common pattern, and it is a use case that would allow Flux to shine more brightly and send fewer users away complaining about a tedious but necessary workaround.

Anyway, I think feature requests, along with Q&A and RFCs/Proposals are all meant to start out on the Flux2 Discussions page, based on our support policy around feature requests, this should be a discussion.

I'm not sure if it's possible to migrate an Issue to a Discussion, but I'm afraid that it isn't within my power at least, and we'll have to close it instead. You're welcome to open up a discussion and refer back to this one though! Or chime in on this related discussion I just found

Closing, (it's labeled as a feature request and the issues queue is meant for bug reports)