[IMPROVEMENT] Remove privilege requirement from lifecycle jobs #5862

innobead · 2023-05-05T05:03:58Z

          Good question. It was enabled by this commit https://github.com/longhorn/longhorn/commit/eb767033390cc31c2a01ec3f93dfd2eb3ff05b1a

But I just walked through the post-upgrade and uninstall job. They are just doing operations against kubeAPI. Not sure why we need them to be privileged

Originally posted by @PhanLe1010 in #5567 (comment)

The text was updated successfully, but these errors were encountered:

innobead · 2023-05-05T05:05:14Z

@mantissahz Try to fix this together with upgrade enforcement. Thanks.

cc @PhanLe1010

longhorn-io-github-bot · 2023-05-08T07:08:51Z

Pre Ready-For-Testing Checklist

Where is the reproduce steps/test steps documented?
The reproduce steps/test steps are at:

Install Longhorn by kubectl
Uninstall Longhorn by kubectl
Uninstall successfully
Install Longhorn by Helm
Upgrade Longhorn by Helm
Upgrade successfully
(for downgrade cases, you could use jamesluhz/lh-manager:v1.4.x-upgrade-path and jamesluhz/lh-manager:v1.3.x-upgrade-path) for downgrade from 1.5.0 to 1.4.x or 1.3.x)

Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
The PR for the YAML change is at:
The PR for the chart change is at:
fix: remove privileged from lifecycle jobs #5881
fix: remove privileged from lifecycle jobs #5885
fix(upgrade): missing command pre-upgrade longhorn-manager#1892
Which areas/issues this PR might have potential impacts on?
Area
Issues
If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
The documentation issue/PR is at
doc: upgrade path enforcement website#699

innobead · 2023-05-08T07:47:37Z

@mantissahz need to check if we need to update the doc.

chriscchien · 2023-05-09T08:02:37Z

Hi @mantissahz ,

From test steps, I can install/uninstall Longhorn by kebectl, but when I try upgrade Longhorn from v1.4.2-rc1 to master by helm, I got below error

it checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
root@ip-172-31-81-81:/home/ubuntu/k9s/longhorn# helm upgrade longhorn ./chart/ -n longhorn-system
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* job failed: BackoffLimitExceeded

pre upgrade pod log

│ panic: unrecognized command: pre-upgrade                                                                                                                                                                │
│                                                                                                                                                                                                         │
│ goroutine 1 [running]:                                                                                                                                                                                  │
│ main.cmdNotFound(0xc00065eba0?, {0x7ffd4ed1b0fc?, 0xc00065ebc8?})                                                                                                                                       │
│     /go/src/github.com/longhorn/longhorn-manager/main.go:15 +0x67                                                                                                                                       │
│ github.com/urfave/cli.ShowCommandHelp(0xc0000fdb80, {0x7ffd4ed1b0fc, 0xb})                                                                                                                              │
│     /go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/help.go:213 +0x404                                                                                                        │
│ github.com/urfave/cli.glob..func1(0x1?)                                                                                                                                                                 │
│     /go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/help.go:21 +0x34                                                                                                          │
│ github.com/urfave/cli.HandleAction({0x1d4a180?, 0x225bf18?}, 0xc00051fa40?)                                                                                                                             │
│     /go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/app.go:524 +0x50                                                                                                          │
│ github.com/urfave/cli.(*App).Run(0xc00051fa40, {0xc000136000, 0x2, 0x2})                                                                                                                                │
│     /go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/app.go:286 +0x7db                                                                                                         │
│ main.main()                                                                                                                                                                                             │
│     /go/src/github.com/longhorn/longhorn-manager/main.go:65 +0x83b                                                                                                                                      │
│ Stream closed EOF for longhorn-system/longhorn-pre-upgrade-mtbjp (longhorn-post-upgrade)

mantissahz · 2023-05-09T09:32:19Z

@chriscchien
The command pre-upgrade is not really added into longhorn-manager image now.
longhorn/longhorn-manager#1892 will do.

chriscchien · 2023-05-09T10:05:01Z

Verified in longhorn master (longhorn-manager c05bf1) with test steps
Result Pass

Install / uninstall Longhorn by kubectl success
Upgrade Longhorn from v1.4.1 to master by helm success
Upgrade Longhorn from v1.4.2-rc1 to master by helm success
After upgrade, uninstall Longhorn by helm success

innobead · 2023-05-09T10:07:47Z

@chriscchien can we test failed case? like 1.3 to 1.5/master? also after it failed, we should be able to recover it by following #5131 (comment).

chriscchien · 2023-05-09T12:52:21Z

Hi @mantissahz , I tried upgrade Longhorn from v1.3.x to master-head by kubectl, I saw daemonset components(longhorn-manager, longhorn-csi-plugin, engine-image) not upgraded and not reboot. Images kept in v1.3.x.

But I saw below deployment components had upgraded to newer version as the setting in master-head manifest

longhorn-ui
longhorn-driver-deployer
csi-attacher
csi-provisioner
csi-resizer
csi-snapshotter

I can recover them by apply v1.3.x manifest again, not sure if this is the correct behavior, thank you.

mantissahz · 2023-05-09T13:15:20Z

@chriscchien
The deployment will be updated by manifest, but the new longhorn-driver-deployer pod will be waiting for new longhorn-manager pod running successfully.
And old pods should be working as usual.

innobead · 2023-05-09T13:20:04Z

Reopened first to let QA finish the complete testing before closing this issue.

@mantissahz Do we have the doc for upgrade enforcement? The issue has require/doc, as we discussed before, need to have a doc for that to explain all upgrade & recovery cases.

innobead · 2023-05-09T13:46:06Z

@chriscchien

chriscchien · 2023-05-10T03:34:29Z

Hi @mantissahz, I can do

Upgrade fail from v.1.3.x to master and can roll back by kubectl
Upgrade fail from v.1.3.x to master and can roll back by helm

But when I try downgrade form master to v1.3.x by kubectl. I can not see notification message related to upgrade deny from console and the longhorn-manger imagre changed to v1.3.x-head (Happened when downgrade from master to v1.4.x as well)

chriscchien · 2023-05-12T07:34:38Z

Verify pass with longhorn-manager (master 784f5a, v1.4.x jamesluhz/lh-manager:v1.4.x-upgrade-path, v1.3.x jamesluhz/lh-manager:v1.3.x-upgrade-path)

Install / uninstall Longhorn by kubectl and helm success
Upgrade Longhorn from v1.3.x to v1.4.x by kubectl and helm success
Upgrade Longhorn from v1.4.x to master by kubectl and helm success
Upgrade Longhorn from v1.3.x to master by kubectl and helm not success
Downgrade Longhorn from master to v1.4.x by kubectl and helm not success
Downgrade Longhorn from master to v1.3.x by kubectl and helm not success

Downgrade failure case can see below log and longhorn-manager pod kept in crashloopbackoff state. Can be recovery by kubectl apply manifest or helm upgrade again

longhorn-manager W0512 06:30:26.625927       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.              │
 longhorn-manager time="2023-05-12T06:30:26Z" level=info msg="Checking if the upgrade path from v1.5.0-dev to v1.3.3 is supported"                                                    │
 longhorn-manager time="2023-05-12T06:30:26Z" level=fatal msg="Error starting manager: failed to upgrade since downgrading from v1.5.0-dev to v1.3.3 is not supported"

May need test again after code of jamesluhz/lh-manager:v1.4.x-upgrade-path and jamesluhz/lh-manager:v1.3.x-upgrade-path merged into related branch

cc @innobead , @mantissahz

innobead · 2023-05-12T07:45:28Z

May need test again after code of jamesluhz/lh-manager:v1.4.x-upgrade-path and jamesluhz/lh-manager:v1.3.x-upgrade-path merged into related branch

@mantissahz What else we need to update/fix? I assumed all changes have been merged, right?

mantissahz · 2023-05-22T08:09:22Z

Yes, there is nothing needed to fix or update.
We could close it if verified.

chriscchien · 2023-05-22T08:13:09Z

After discussed with @mantissahz , we can follow test result here and close this ticket, thank you.

innobead assigned mantissahz May 5, 2023

innobead added area/install-uninstall-upgrade Install, Uninstall or Upgrade related area/security System or volume data access security labels May 5, 2023

innobead added this to the v1.5.0 milestone May 5, 2023

innobead added component/longhorn-manager Longhorn manager (control plane) priority/0 Must be fixed in this release (managed by PO) labels May 5, 2023

innobead mentioned this issue May 5, 2023

feat: introduce helm hook pre-upgrade #5567

Merged

mantissahz mentioned this issue May 8, 2023

fix: remove privileged from lifecycle jobs #5881

Merged

innobead added the require/doc Require updating the longhorn.io documentation label May 8, 2023

mantissahz mentioned this issue May 8, 2023

fix: remove privileged from lifecycle jobs #5885

Merged

innobead assigned chriscchien May 8, 2023

mantissahz mentioned this issue May 9, 2023

fix(upgrade): missing command pre-upgrade longhorn/longhorn-manager#1892

Merged

longhorn deleted a comment from longhorn-io-github-bot May 9, 2023

chriscchien closed this as completed May 9, 2023

innobead reopened this May 9, 2023

chriscchien closed this as completed May 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IMPROVEMENT] Remove privilege requirement from lifecycle jobs #5862

[IMPROVEMENT] Remove privilege requirement from lifecycle jobs #5862

innobead commented May 5, 2023

innobead commented May 5, 2023

longhorn-io-github-bot commented May 8, 2023 •

edited by mantissahz

innobead commented May 8, 2023

chriscchien commented May 9, 2023

mantissahz commented May 9, 2023

chriscchien commented May 9, 2023

innobead commented May 9, 2023 •

edited

chriscchien commented May 9, 2023 •

edited

mantissahz commented May 9, 2023

innobead commented May 9, 2023 •

edited

innobead commented May 9, 2023 •

edited

chriscchien commented May 10, 2023 •

edited

chriscchien commented May 12, 2023 •

edited

innobead commented May 12, 2023

mantissahz commented May 22, 2023

chriscchien commented May 22, 2023

[IMPROVEMENT] Remove privilege requirement from lifecycle jobs #5862

[IMPROVEMENT] Remove privilege requirement from lifecycle jobs #5862

Comments

innobead commented May 5, 2023

innobead commented May 5, 2023

longhorn-io-github-bot commented May 8, 2023 • edited by mantissahz

Pre Ready-For-Testing Checklist

innobead commented May 8, 2023

chriscchien commented May 9, 2023

mantissahz commented May 9, 2023

chriscchien commented May 9, 2023

innobead commented May 9, 2023 • edited

chriscchien commented May 9, 2023 • edited

mantissahz commented May 9, 2023

innobead commented May 9, 2023 • edited

innobead commented May 9, 2023 • edited

chriscchien commented May 10, 2023 • edited

chriscchien commented May 12, 2023 • edited

innobead commented May 12, 2023

mantissahz commented May 22, 2023

chriscchien commented May 22, 2023

longhorn-io-github-bot commented May 8, 2023 •

edited by mantissahz

innobead commented May 9, 2023 •

edited

chriscchien commented May 9, 2023 •

edited

innobead commented May 9, 2023 •

edited

innobead commented May 9, 2023 •

edited

chriscchien commented May 10, 2023 •

edited

chriscchien commented May 12, 2023 •

edited