[stable/nginx-ingress] Fix for "spec.clusterIP: Invalid value" error on upgrade #13646
Conversation
helm-bot
added
Contribution Allowed
|
Hi @max-rocket-internet. Thanks for your PR. I'm waiting for a helm member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
the
needs-ok-to-test
|
/assign @jackzampolin |
|
@max-rocket-internet: GitHub didn't allow me to assign the following users: jackzampolin. Note that only helm members and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/assign mgoodness |
| @@ -19,7 +19,9 @@ metadata: | |||
| release: {{ .Release.Name }} | |||
| name: {{ template "nginx-ingress.controller.fullname" . }}-metrics | |||
| spec: | |||
| {{- if not .Values.controller.metrics.service.omitClusterIP }} | |||
There was a problem hiding this comment.
Instead of a new value that can be set lets have these check if the current value of clusterIP contains anything.
{{- if .Values.controller.metrics.service.clusterIP }}
https://github.com/helm/helm/blob/master/docs/chart_template_guide/control_structures.md#ifelse
There was a problem hiding this comment.
Hey @ChiefAlexander
Of course that's the obvious solution but it's not usable here. This is all covered in #9227
This was tried in #7933 and reverted in #10993
All users who installed the chart between version 0.31.0 and 1.2.2 cannot upgrade.
There was a problem hiding this comment.
This whole thing is a disaster. Add a section in the readme to document this and how users need to use it to upgrade from those versions.
There was a problem hiding this comment.
OK done
|
/assign |
…on upgrade Signed-off-by: Max Williams <max.williams@deliveryhero.com>
Signed-off-by: Max Williams <max.williams@deliveryhero.com>
Signed-off-by: Max Williams <max.williams@deliveryhero.com>
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ChiefAlexander, max-rocket-internet The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
…on upgrade (helm#13646) * [stable/nginx-ingress] Fix for "spec.clusterIP: Invalid value" error on upgrade Signed-off-by: Max Williams <max.williams@deliveryhero.com> * update README and values.yaml Signed-off-by: Max Williams <max.williams@deliveryhero.com> * adding note to readme Signed-off-by: Max Williams <max.williams@deliveryhero.com>
…on upgrade (helm#13646) * [stable/nginx-ingress] Fix for "spec.clusterIP: Invalid value" error on upgrade Signed-off-by: Max Williams <max.williams@deliveryhero.com> * update README and values.yaml Signed-off-by: Max Williams <max.williams@deliveryhero.com> * adding note to readme Signed-off-by: Max Williams <max.williams@deliveryhero.com>
|
I installed 1.6.0 and am now upgrading to 1.10.1 and get this same error. How am I suppose to configure this, because the readme.md is not clear to me. This is the error I get: On the cmdline I have add this: But that makes no different. I also use a values-yaml which has these fields (partial): Also doesn;t help. This makes it impossible to upgrade. All I can do is remove the chart en reinstall. |
|
@kwaazaar AFAIK version 1.6 wasn't affected by this issue. Can you paste helm diff output? |
|
@naseemkullah I tried all combinations of At least I could get out of the situation: What is funny: I looked at the output of |
Just to confirm did you try commenting out/removing |
|
A similar issue is also occurring for |
|
Hi I just want to report that I'm experiencing exactly the same issue as described by @heidemn I'm trying to upgrade from the chart v1.6.4 |
|
I'm experiencing exactly the same issue as described by @heidemn . I tried to upgrade from Chart 1.24.7 to 1.26.2. |
|
From what I gather only fresh installs, either as of My understanding is, and I may be wrong but: If you can afford downtime (probably not) uninstall and reinstall the latest version of the chart without any clusterIP related values (Not even If you cannot afford downtime, no choice but to hardcode all clusterIP values, until you do get some allocated downtime, at which point do the above. (I'm curious as to why clusterIPs were ever made configurable in the first place?) |
|
This is even a problem from 1.25.0 -> 1.26.2 for me now. I had this problem before when I had to purge the nginx-ingress release and redeploy it. Now even setting service.omitClusterIP: true for controller, controller.metrics and default-backend does not help for this upgrade. Also setting ClusterIP to "" does not help for a migration as mentioned in the README
I had to manually get the ClusterIP of each of the 3 services and manually set them into my values. With 1.25.0 I did not have to do this, so this seems like a regression to me. Am I missing something? The goal (for migrating from 1.25.0 to 1.26.2) should still be, to not need to set manually the ClusterIP of existing services. |
The README is unclear then, it states to not set any clusterIP values, (not even Please note, I was able to run Followed up with a sucessfully. However if omitClusterIP was not already being used in the pre 1.25.0 version, the upgrade failed with the infamous clusterIP error. In other words, when you initially redeployed, had you set the Both
My understanding is that same thing would happen today if So to sum up, a freshly installed release as of this commit should set the |
|
I still have this problem. The only solution to update is with these instructions: I'm using chart 0.26.1. |
…grade see - helm#13646 Signed-off-by: Jyrno Ader <jyrno42@gmail.com>
…grade see - helm#13646 Signed-off-by: Jyrno Ader <jyrno42@gmail.com>
…grade see - helm#13646 Signed-off-by: Jyrno Ader <jyrno42@gmail.com>
…grade see - helm#13646 Signed-off-by: Jyrno Ader <jyrno42@gmail.com>
…grade see - helm#13646 Signed-off-by: Jyrno Ader <jyrno42@gmail.com>
Yes, I tried that. Might it be possible to edit the internal state of Helm2/Tiller to get rid of the offending |
|
We're also experiencing problems with upgrading from v0.22.1. We have |
|
Hi. |
|
I've been trying to figure out a way to upgrade without hard coding a clusterIP, and it looks like the solution is upgrading the chart with "--force". Initial attempt the normal way. First install 0.22.1 like it is currently: Then upgrade, including Getting the familiar error. After cleanup, do the same, except upgrade using --force. Much better! Not sure if there are any downsides to this yet. This works even without adding |
|
--force didn't work for me, but this did (#9227 (comment)): patch_nginx_release() {
set -v
local secret=$1
# patch ClusterIP
patched=$(kubectl get secrets $secret -o jsonpath='{.data.release}' | base64 -d | base64 -d | gzip -d - | gsed 's# clusterIP: \\"\\"\\n##g' | gzip - | base64 | base64)
kubectl patch secret $secret -p "{\"data\":{\"release\":\"$patched\"}}" --type=merge
# patch rbac v1beta deprecation needed for 1.29.x
patched=$(kubectl get secrets $secret -o jsonpath='{.data.release}' | base64 -d | base64 -d | gzip -d - | gsed 's#rbac.authorization.k8s.io/v1beta1#rbac.authorization.k8s.io/v1#g' | gzip - | base64 | base64)
kubectl patch secret $secret -p "{\"data\":{\"release\":\"$patched\"}}" --type=merge
set +v
}
# Convert helm2 to helm3
kubectl get cm -l NAME=MYHELMRELEASE -o yaml -n kube-system > backup-MYHELMRELEASE-helm2.yaml
helm3 2to3 convert --delete-v2-releases MYHELMRELEASE
# get latest helm3 release via: kubectl get secrets
patch_nginx_release sh.helm.release.v1.MYHELMRELEASE.REVISION
# See that there should be no ClusterIP change
helm3 diff upgrade --namespace NAMESPACE -f values.yaml MYHELMRELEASE stable/nginx --version 1.29.6
# Update labels for helm3 and update from 1.25 to 1.29
helm3 upgrade --namespace NAMESPACE -f values.yaml MYHELMRELEASE stable/nginx --version 1.29.6I used this like 10-20 times to upgrade from nginx-ingress 1.25.0 to 1.29.6 now. |
|
Good to have another alternative! |
The downsides with this approach is that there will be downtime, since a load balancer will be recreated for the nginx service. |
|
I made this utility to be able to change helm 2 charts in their configmaps, if you want to remove ClusterIP: "" from the old manifests. |
|
I tested out removing the ClusterIP: "" from all the services in the old helm2 release using the utility above, and the upgrade worked for me. |
Not sure I understand. Do you mean if you have a service of type LoadBalancer? |
What this PR does / why we need it:
Fixes this error by allowing the user to set
xxx.service.omitClusterIP=truein their values:The workaround provided in #9227 where you need to retrieve the
ClusterIPvalue from your cluster and insert into your values is not adequate 😃Which issue this PR fixes
Fixes: #9227
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]
[stable/chart]