Broken Helm Rollback Command in v3.11.0-rc.1 #11712

bschoenbach · 2023-01-11T11:11:27Z

Output of helm version:
version.BuildInfo{Version:"v3.11.0-rc.1", GitCommit:"9d8fee155bd7e7d3c1390f4076d9271a1147dce5", GitTreeState:"clean", GoVersion:"go1.18.9"}

Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.12", GitCommit:"c6939792865ef0f70f92006081690d77411c8ed5", GitTreeState:"clean", BuildDate:"2022-09-21T12:13:07Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): On-Premise nutanix hardware

Problem:
helm rollback cmd to latest release breaks the deployment. Hence, corresponding deployments are deleted...everythins is gone after rollback. Using older helm version v3.10.3 works picture perfect.....Below you will find the trerminal output

BEFORE Rollback:

/$:~/helm_3.11/linux-amd64$ helm history rost-webui -n dns
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
...
258 Wed Jan 11 09:28:02 2023 superseded rost-webui-0.1.0 prod-8f7e8a0e-742694699 Upgrade complete

/$ kubectl get pods -n dns
NAME READY STATUS RESTARTS AGE
rost-webui-764f4d6c84-6xdkx 1/1 Running 0 9m1s

PERFORM Rollback

/$:~/helm_3.11/linux-amd64$ ./helm rollback rost-webui 258 -n dns
Rollback was a success! Happy Helming!

AFTER Rollback:

/$:~/helm_3.11/linux-amd64$ helm history rost-webui -n dns
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
...
258 Wed Jan 11 09:28:02 2023 superseded rost-webui-0.1.0 prod-8f7e8a0e-742694699 Upgrade complete
259 Wed Jan 11 09:34:07 1025 superseded rost-webui-0.1.0 prod-8f7e8a0e-742694699 Rollback to 258

/$ kubectl get pods -n dns
No resources found in dns namespace.

The text was updated successfully, but these errors were encountered:

mattfarina · 2023-01-11T18:19:40Z

Yikes. Thanks for providing those details. We will attempt to verify the issue.

mattfarina · 2023-01-11T19:16:30Z

I have been able to verify this is happening. A simple script to reproduce.

$ helm create foo
$ helm install f foo
$ helm upgrade f foo
$ helm rollback f 1

This is enough to trigger the problem described here.

mattfarina · 2023-01-11T20:13:14Z

The commit that caused this to happen is 6f8c178 which came in as part of #10912. Unfortunately, this appear to be a merge commit from main so it's not obvious what's causing it at first glance.

mattfarina · 2023-01-11T20:33:31Z

I've been able to trace this to #10912 where TransformRequests(transformRequests). was introduced. Commenting that out fixes things. This is the source of the issue.

If you look at the http requests between 3.11.0-rc.1 and 3.10.3 on rollback with -v 10 as a flag you'll see the HTTP requests have changed. They are being transformed in a way that's causing problems.

mattfarina · 2023-01-11T22:22:51Z

I have a fix I'm working on. The could should be available for review on Thursday.

Fixes helm#11712 A change was made that when validation was turned off the Kubernetes packages were building objects as a Table type. This was done for display purposes. When details about the objects was going to be printed as part of helm#10912. This broke rollback, and possibly other functionality, as a Table type was returned in some cases that needed the regular object. This caused things to break silently. The fix involved adding in a new Function (and interface) to query for tables instead of the objects themselves. There was not a clean way to add it to the existing function that covered all cases. A second problem was noticed along the way. When data was output via status as YAML or JSON it was in the form of a table rather than the objects themselves. This did not reflect expectations and did not match the functionality in kubectl. The code was updated to return a table when that was presented and the objects when they are being output for YAML or JSON. The API also supports this handling to SDK users can replicate this functionality. API changes made here were never released. The functions were developed for this release of Helm and only ever appeared in an RC. In this case, they can be changed. Signed-off-by: Matt Farina <matt.farina@suse.com>

Fixes #11712 A change was made that when validation was turned off the Kubernetes packages were building objects as a Table type. This was done for display purposes. When details about the objects was going to be printed as part of #10912. This broke rollback, and possibly other functionality, as a Table type was returned in some cases that needed the regular object. This caused things to break silently. The fix involved adding in a new Function (and interface) to query for tables instead of the objects themselves. There was not a clean way to add it to the existing function that covered all cases. A second problem was noticed along the way. When data was output via status as YAML or JSON it was in the form of a table rather than the objects themselves. This did not reflect expectations and did not match the functionality in kubectl. The code was updated to return a table when that was presented and the objects when they are being output for YAML or JSON. The API also supports this handling to SDK users can replicate this functionality. API changes made here were never released. The functions were developed for this release of Helm and only ever appeared in an RC. In this case, they can be changed. Signed-off-by: Matt Farina <matt.farina@suse.com> (cherry picked from commit 36e18fa)

mattfarina · 2023-01-13T20:41:27Z

@bschoenbach https://github.com/helm/helm/releases/tag/v3.11.0-rc.2 is out which fixes the regression. Thank you for catching it and providing details for us to fix it.

Fixes helm#11712 A change was made that when validation was turned off the Kubernetes packages were building objects as a Table type. This was done for display purposes. When details about the objects was going to be printed as part of helm#10912. This broke rollback, and possibly other functionality, as a Table type was returned in some cases that needed the regular object. This caused things to break silently. The fix involved adding in a new Function (and interface) to query for tables instead of the objects themselves. There was not a clean way to add it to the existing function that covered all cases. A second problem was noticed along the way. When data was output via status as YAML or JSON it was in the form of a table rather than the objects themselves. This did not reflect expectations and did not match the functionality in kubectl. The code was updated to return a table when that was presented and the objects when they are being output for YAML or JSON. The API also supports this handling to SDK users can replicate this functionality. API changes made here were never released. The functions were developed for this release of Helm and only ever appeared in an RC. In this case, they can be changed. Signed-off-by: Matt Farina <matt.farina@suse.com>

bkthong · 2023-11-01T12:41:06Z

I am getting the same behavior on helm 3.11 installed on fedora 38 via dnf:

The following sequence triggers the issue consistently:

$ helm create testchart
$ helm install myapp testchart
$ help upgrade myapp testchart
$ helm rollback myapp

--> pods and deployments are all gone (deleted) after a rollback!!

Manually installing 3.10.3 (https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz) currently solves the problem (as in rollback works properly)

$ helm version (install via dnf on Fedora 38)
version.BuildInfo{Version:"v3.11", GitCommit:"", GitTreeState:"", GoVersion:"go1.20.1"}

$ dnf info helm
Last metadata expiration check: 0:00:24 ago on Wed 01 Nov 2023 08:25:30 PM +08.
Installed Packages
Name : helm
Version : 3.11.1
Release : 1.fc38
Architecture : x86_64
Size : 59 M
Source : golang-helm-3-3.11.1-1.fc38.src.rpm
Repository : @System
From repo : fedora
Summary : The Kubernetes Package Manager
URL : https://github.com/helm/helm
License : Apache-2.0
Description :
: Helm is a tool for managing Charts. Charts are packages of pre-configured
: Kubernetes resources.
:
: Use Helm to:
: - Find and use popular software packaged as Helm Charts to run in Kubernetes
: - Share your own applications as Helm Charts
: - Create reproducible builds of your Kubernetes applications
: - Intelligently manage your Kubernetes manifest files
: - Manage releases of Helm packages

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"846602e50eb29ecdc7441ebed2fc846048959eaa", GitTreeState:"clean", BuildDate:"2023-03-08T15:10:20Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6+73ac561", GitCommit:"cbbd0bdba10e0b612e32cdeb6462daa6909df4be", GitTreeState:"clean", BuildDate:"2023-07-14T11:07:31Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}

mattfarina added the bug Categorizes issue or PR as related to a bug. label Jan 11, 2023

mattfarina added this to the 3.11.0 milestone Jan 11, 2023

mattfarina self-assigned this Jan 11, 2023

mattfarina mentioned this issue Jan 12, 2023

Fix improper use of Table request/response to k8s API #11720

Merged

3 tasks

mattfarina closed this as completed in #11720 Jan 13, 2023

sollidy mentioned this issue Jan 3, 2024

Broken Helm Rollback Command #12681

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Broken Helm Rollback Command in v3.11.0-rc.1 #11712

Broken Helm Rollback Command in v3.11.0-rc.1 #11712

bschoenbach commented Jan 11, 2023 •

edited

Loading

mattfarina commented Jan 11, 2023

mattfarina commented Jan 11, 2023

mattfarina commented Jan 11, 2023

mattfarina commented Jan 11, 2023

mattfarina commented Jan 11, 2023

mattfarina commented Jan 13, 2023

bkthong commented Nov 1, 2023

Broken Helm Rollback Command in v3.11.0-rc.1 #11712

Broken Helm Rollback Command in v3.11.0-rc.1 #11712

Comments

bschoenbach commented Jan 11, 2023 • edited Loading

mattfarina commented Jan 11, 2023

mattfarina commented Jan 11, 2023

mattfarina commented Jan 11, 2023

mattfarina commented Jan 11, 2023

mattfarina commented Jan 11, 2023

mattfarina commented Jan 13, 2023

bkthong commented Nov 1, 2023

bschoenbach commented Jan 11, 2023 •

edited

Loading