Fix Helm Chart remove Capabilities.APIVersions for Kustomize to parse file

[stoupance]

What this PR does / why we need it:

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation only

Which issue/s this PR fixes

This PR fixes the issue initially described here. The Helm chart provided could not customized by Kustomize to build the YAML section for serviceMonitor. Indeed this Helm template file has the following test as first line.

{{- if and ( .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" ) .Values.controller.metrics.enabled .Values.controller.metrics.serviceMonitor.enabled -}}

So Kustomize with Helm (kubectl kustomize . --enable-helm=true) build locally the YAML from this template and could not pass the first test ( .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" ) for obvious reasons. Test failed and then did not output, as expected, the final YAML section for serviceMonitor.

This test irrelevant and is not present in other template files, for example like the one regarding Prometheus to build the prometheusRule.

This issue is not presents with vanilla Helm and only appears when Kustomize is used on top of Helm

How Has This Been Tested?

1. I fork this repo
2. I made the change into controller-servicemonitor.yaml file and pushed it into my forked repo
3. I change my code to pull files from my forked repo as a Helm repo
4. I ran the kustomize command provided previously (kubectl kustomize . --enable-helm=true)
5. Results: it outputs the YAML section for serviceMonitor as expected

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I've read the CONTRIBUTION guide
• I have added tests to cover my changes.
• All new and existing tests passed.

[k8s-ci-robot]

Welcome @stoupance!

It looks like this is your first PR to kubernetes/ingress-nginx 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/ingress-nginx has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @stoupance. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[stoupance]

/assign @ChiefAlexander

[ChiefAlexander]

/ok-to-test

[ChiefAlexander]

@rikatz for the release process of the helm chart do we need to have PR's bump the chart version?

This looks good to me otherwise.

[cpanato]

/lgtm
/hold for @rikatz

[cpanato]

@ChiefAlexander @rikatz i think we decide when we have enough things to release or when is a critical fix/feature

[ChiefAlexander]

I disagree. I would rather we release fixes as they happen to get them out as quickly as possible. There should be no harm in bumping a bugfix version and turning that around. Even minor fixes.

[cpanato]

I disagree. I would rather we release fixes as they happen to get them out as quickly as possible. There should be no harm in bumping a bugfix version and turning that around. Even minor fixes.

i agree with you @ChiefAlexander I was just explaining how was before, not sure if we want to continue having that

[tao12345666333]

This check is added here to prevent incorrect configuration from causing exceptions.
And this is a Helm chart, I don’t think it is reasonable to delete this condition.

/hold

[ChiefAlexander]

This check is added here to prevent incorrect configuration from causing exceptions. And this is a Helm chart, I don’t think it is reasonable to delete this condition.

/hold

It is not used in the other CRD's for this. Given that the option .Values.controller.metrics.serviceMonitor.enabled is tied directly to the CRD in this case this seems totally fine.

[cpanato]

i agree with @ChiefAlexander or I'm missing something that I cannot see/understand

[rikatz]

• You folks are the helm expert, not me. You can approve and release as you wish :P
• I think it's fine to release helm untied, what we need is actually something that defines when to bump or not. If we bump too much because of typos or new features to a really small subset of users, we may lose users opportunity to update as they think all the releases are small ones :)

[tao12345666333]

The problem I am considering here is the following situation:

➜  ingress-nginx git:(stoupance/main) ✗ helm install ingress -n 7829-ingress --set controller.metrics.enabled=true --set controller.metrics.serviceMonitor.enabled=true charts/ingress-nginx
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

If we assume that the user knows what they are doing, then this condition can be removed. (Feel free to remove the hold and merge it. )

[longwuyuan]

Personally, servicemonitor is just too complicated. Unless the docs can be updated to teach and support service monitors, this may not end up being an improvement. Thanks, ; Long

…

On Mon, 25 Oct, 2021, 9:14 AM Jintao Zhang, ***@***.***> wrote: The problem I am considering here is the following situation: ➜ ingress-nginx git:(stoupance/main) ✗ helm install ingress -n 7829-ingress --set controller.metrics.enabled=true --set controller.metrics.serviceMonitor.enabled=true charts/ingress-nginx Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" If we assume that the user knows what they are doing, then this condition can be removed. (Feel free to remove the hold and merge it. ) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7829 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWXSID536IB2YDEF44TUITHAXANCNFSM5GMCNBHQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[stoupance]

The problem I am considering here is the following situation:

➜  ingress-nginx git:(stoupance/main) ✗ helm install ingress -n 7829-ingress --set controller.metrics.enabled=true --set controller.metrics.serviceMonitor.enabled=true charts/ingress-nginx
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

If we assume that the user knows what they are doing, then this condition can be removed. (Feel free to remove the hold and merge it. )

@tao12345666333 Well, this error is thrown by your cluster because I assume you don't have the proper (Prometheus) CRDs installed? Which is, in fact, an expected and "normal" error. It goes the same way with any operator, if it's not installed, then you cannot use its CRDs. Nothing can protect a user from this, this is the default behaviour.

There is actually 3 conditions in the template file

1. .Capabilities.APIVersions.Has "monitoring.coreos.com/v1"
2. .Values.controller.metrics.enabled
3. .Values.controller.metrics.serviceMonitor.enabled

If you want to keep condition 1, I can understand this, then it must be also added in this other template file because both are using the same CRDs (keep in mind doing this will then prevent any usage of this Helm chart with Kustomize (which is the case of maybe only few users, but which is my case 😄 ))

Indeed, condition 1 cannot be present in one file, and not in another template file which both are tied to the same CRDs. This is not logical. And this condition is actually causing an issue when used by Kustomize.

That being said, the third condition (.Values.controller.metrics.serviceMonitor.enabled) also ensure the proper CRDs are deployed into the cluster and prevents any wrong situations for users who don't know what they are doing (this is because of this you got the error output and preventing you to deploy the serviceMonitor).

---

Personally, servicemonitor is just too complicated. Unless the docs can be updated to teach and support service monitors, this may not end up being an improvement. Thanks, ; Long

@longwuyuan I don't understand your point. The serviceMonitor is already present in the helm template file and the issue is not related to the serviceMonitor. Nor it is related to Prometheus/serviceMonitor documentation. Nor an improvement. It is a bug fix about **conditions on how to deploy ** the serviceMonitor to avoid any error.

In this case, when using Kustomize to customize a Helm chart the serviceMonitor is not deployed because the condition .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" cannot be verified by Kustomize.
As said by @ChiefAlexander the option .Values.controller.metrics.serviceMonitor.enabled is already tied directly to this same CRDs so bascially it means in this Helm template file there is twice the same check, but written differently, and the first one causing an issue in this particular case of using Kustomize. See my answer just above for more details.

So, it's not an improvement, but a bug fix while some users (me for example) cannot use this Helm chart because of this "double checks" when one is totally irrelevant.

[tao12345666333]

@tao12345666333 Well, this error is thrown by your cluster because I assume you don't have the proper (Prometheus) CRDs installed? Which is, in fact, an expected and "normal" error. It goes the same way with any operator, if it's not installed, then you cannot use its CRDs. Nothing can protect a user from this, this is the default behaviour.

Sure. The situation I mentioned above is that when the user did not install the relevant CRD, the option was turned on by mistake.

So adding this condition is equivalent to a stricter restriction.

Strict restrictions have not caused any bugs, on the contrary, it reduces the probability of users seeing errors and can bring a better user experience.

I think the reason here is that kustomize does not fully support all the capabilities of Helm (and this is not the native usage of Helm). ref: kubernetes-sigs/kustomize#3815 (comment)

The safest, most performant, and completely unlimited way to use kustomize + helm to generate config is

1. manually inflate the helm chart via helm template --foo --bar ... (with whatever risky options one wants),
2. commit the resulting plain old k8s YAML to a repo.
3. use kustomize to operate on that repo, treating it as a base, adding some last mile config changes.

From time to time, capture the latest chart, and repeat steps 1 and 2. It's dangerous to set up a production stack that relies on kustomize to freshly download a helm chart with each build.

[rikatz]

How is this going?

[iamNoah1]

@longwuyuan @tao12345666333 friendly ping to follow up here :)

[iamNoah1]

/kind bug
/triage accepted
/priority important-longterm

[longwuyuan]

@stoupance , any chance you can create a issue and link it here.
Such a complicated discussion requires at least a issue to be describing what is the current problem this PR wants to solve, in such a way that all normal users of the ingress-nginx controlller, can understand and track

[afirth]

/lgtm

[rikatz]

/lgtm
/approve
We discussed this in ingress-nginx dev meeting and this makes sense.

Let's move forward with this

/hold cancel

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afirth, cpanato, rikatz, stoupance

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• charts/ingress-nginx/OWNERS [cpanato,rikatz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment