[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy #4003

innobead · 2022-05-19T02:29:09Z

Is your improvement request related to a feature? Please describe

PodSecurityPolicy has been deprecated and will be removed from K8s 1.25, so we need to find an alternative way to resolve the need for PSP in Longhorn to support 1.25.

Also, some deprecated resource versions are also removed from 1.25. Need to resolve this via #4239 or even consider to backport this to 1.3 & 1.2 via an adaptive way to determine the K8s version of the cluster to use which API resource version if possible (except PSP, because it's totally removed instead of version bump).

Cronjob v1beta1 -> v1
EndpointSlice v1beta1 -> v1
Event v1beta1 -> v1
HorizontalPodAutoscaler v2beta1 -> v2
PodDisruptionBudget v1beta1 -> removed

Note: client-go is backward compatible with K8s any version.

Compatibility: client-go <-> Kubernetes clusters
Since Kubernetes is backwards compatible with clients, older client-go versions will work with many different Kubernetes cluster versions.

Describe the solution you'd like

Deprecate PSP if it's not needed. Otherwise, we need an alternative solution like https://kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/.

Describe alternatives you've considered

N/A

Additional context

https://www.kubernetes.dev/resources/release/#timeline
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-25
#4239

innobead · 2022-07-29T09:23:46Z

This needs to consider to backport.

mattmeye · 2022-09-13T06:49:44Z

This was released in Longhorn v1.2.5-rc1 release but not in Longhorn v1.2.5 release. Is there an ETA for an official release?

innobead · 2022-09-13T07:06:16Z

This was released in Longhorn v1.2.5-rc1 release but not in Longhorn v1.2.5 release. Is there an ETA for an official release?

It was wrongly mentioned in the rc release note. It is planned for 1.4.0 instead, the end of this year.

daimoniac · 2022-09-13T09:01:50Z

dont' forget CronJobs #4490 (comment)

innobead · 2022-09-13T09:09:21Z

dont' forget CronJobs #4490 (comment)

Thanks 👍 , all deprecated resources will be taken care of together. cc @PhanLe1010

skandragon · 2022-09-14T12:18:46Z

I managed to upgrade to kubernetes 1.25 on several nodes before I noticed the issue, so I'm hoping a release comes out that will handle that version as rolling back isn't an option.

mattmeye · 2022-09-14T13:26:36Z

@skandragon I have exactly the same problem too. However, I cannot wait until the end of the year and unfortunately have to set up the cluster again.

…4003

PurpleBooth · 2022-09-21T06:49:57Z

It was wrongly mentioned in the rc release note. It is planned for 1.4.0 instead, the end of this year.

Are you sure about this scheduling, by the end of the year, that is Tuesday 6th December 2022, 1.26 will be out?

innobead · 2022-09-21T07:10:15Z

Knowing people are eager for this, but for now, the plan is still the same, because Longhorn has its release scope and cadence.

Please keep following up on the update here, and we will work on this soon or later.

skandragon · 2022-09-27T19:39:45Z

@innobead what is required to do beyond just changing the API calls? I know that means version 1.3 would be locked to 1.25 or higher; the better option is likely to try to use the proper API calls depending on what version was in use, I suppose?

I know I was able to just change the betav1 magic to v1 and it just worked as far as I could tell.

A bigger issue for me with longhorn is unrelated, and that's that it won't run under Talos. :/

BjoernPetersen · 2022-09-28T06:46:06Z

Just want to mention that the wording of the issue is misleading.

PodSecurityPolicy will be deprecated from K8s 1.25 [...]

PodSecurityPolicy has been deprecated since K8s 1.21 and will be removed in 1.25.

kjaleshire · 2022-09-29T02:39:54Z

That's really unfortunate, Longhorn will be unusable by the most recent Kubernetes release for the next two months.

I encourage you to reconsider this release plan.

dominch · 2022-10-05T07:59:54Z

I just hit this issue on k3s (latest channel) and needed to revert to 1.24 (where other coredns bug is present)
This should happen ASAP because 1.25 is on the way and any service based on longhorn will fail :(

MohammedNoureldin · 2022-10-26T00:25:45Z

(subjected to change any time)

@PhanLe1010 I know that you mean to an earlier date ;)

Maybe not really relevant question, but adding a GitHub repo and installing this chart is not really something that works out of the box, and no link online helped so far. Could you summarize that in 2 lines of bash commands?

PhanLe1010 · 2022-10-26T00:30:12Z

The bash command would be like
git clone https://github.com/longhorn/longhorn.git && cd longhorn && helm install longhorn ./chart --namespace longhorn-system --create-namespace

Another alternative is kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

MohammedNoureldin · 2022-10-26T00:31:39Z

Oh, thanks! I thought it is possible to install it using Helm from a GitHub repo on the fly, but it turned out that I need to clone it. I will try it now. Thanks a lot!

dkrizic · 2022-10-26T22:39:31Z

I just want to confirm that the helm chart @ master works excellently on Kubernetes 1.25.3 with the default settings. I am now waiting for the release 1.4.0

chriscchien · 2022-10-28T01:08:38Z

Verified in longhorn master-head 777bc5 with test steps
Result Pass

cc @PhanLe1010 , @khushboo-rancher

Case 1: within a Kubernetes version

Test basic recurring snapshot/backup jobs are working correctly

Kubernetes Version	Pipeline Link	Verify Result
v1.25.3+k3s1	Verified on local environment	✔️
v1.24.7+k3s1	https://ci.longhorn.io/job/private/job/longhorn-tests-regression/2239/	✔️
v1.23.1+k3s2	https://ci.longhorn.io/job/public/job/master/job/sles/job/amd64/job/longhorn-tests-sles-amd64/292/testReport/tests/test_recurring_job/	✔️
v1.22.15+k3s1	https://ci.longhorn.io/job/private/job/longhorn-tests-regression/2238/	✔️
v1.21.10+k3s1	https://ci.longhorn.io/job/private/job/longhorn-tests-regression/2237/	✔️

Test generating support bundle generation. Make sure that the above jobs are in /yamls/kubernetes/cronjobs folder

Kubernetes Version	Verify Result
v1.25.3+k3s1	✔️
v1.24.7+k3s1	✔️
v1.23.13+k3s1	✔️
v1.22.15+k3s1	✔️
v1.21.10+k3s1	✔️

Test upgrade Longhorn from 1.3.2 or v1.2.5 to master

Test basic functionalities (workload, upgrade engine image, replica rebuilding)

Kubernetes Version	Upgrade from v1.3.2	Upgrade from v1.2.5
v1.25.3+k3s1	NA	NA
v1.24.7+k3s1	✔️	✔️
v1.23.13+k3s1	✔️	✔️
v1.22.15+k3s1	✔️	✔️
v1.21.10+k3s1	✔️	✔️

Verify CSI snapshot upgrade (by pytest test_csi_snapshotter.py after upgrade)

Kubernetes Version	Upgrade from v1.3.2	Upgrade from v1.2.5
v1.25.3+k3s1	NA	NA
v1.24.7+k3s1	✔️	✔️
v1.23.13+k3s1	✔️	✔️
v1.22.15+k3s1	✔️	✔️
v1.21.10+k3s1	✔️	✔️

Test zone and topology,

Kubernetes Version	Verify Result
v1.25.3+k3s1	✔️
v1.24.7+k3s1	✔️
v1.23.13+k3s1	✔️
v1.22.15+k3s1	✔️
v1.21.10+k3s1	✔️

Test uninstallation

Kubernetes Version	Verify Result
v1.25.3+k3s1	✔️
v1.24.7+k3s1	✔️
v1.23.13+k3s1	✔️
v1.22.15+k3s1	✔️
v1.21.10+k3s1	✔️

Case 2: Test a Kubernetes version upgrade

Update from v1.24.7+k3s1 to v1.25.3+k3s1 and workload still working properly

Install Longhorn master in Kubernetes v1.24 cluster
Create some workload, Longhorn volumes, cronjobs, etc...
Upgrade Kubernetes to v.125
Verify that Longhorn is running ok. Basic functionality is working ok

khushboo-rancher · 2022-10-28T18:30:44Z

@chriscchien Could you just test one more scenario, deploying Longhorn on a hardened cluster?

abderraxim · 2022-10-29T21:56:45Z

Head Chart installs perfectly and everything looks fine
N.B.: please include the dev chart in your repository

MicroK8s 1.25.3

nabeelshaikh7 · 2022-11-02T18:52:20Z

and some how installing longhorn crashed treafik in my k3s cluster

chriscchien · 2022-11-04T05:52:47Z

Hi @khushboo-rancher ,

I tried deploy Longhorn master on hardened cluster but the deploy not scuccessed.
Deployment longhorn-driver-deployer raised error and most Longhorn component pod not deployed

[FailedCreate] pods "longhorn-driver-deployer-f9546bd9b-" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.initContainers[0].securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden spec.containers[0].securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden]

yangchiu · 2022-11-04T06:18:45Z

Hi @khushboo-rancher ,

I tried deploy Longhorn master on hardened cluster but the deploy not scuccessed. Deployment longhorn-driver-deployer raised error and most Longhorn component pod not deployed
[FailedCreate] pods "longhorn-driver-deployer-f9546bd9b-" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.initContainers[0].securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden spec.containers[0].securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden]

@chriscchien Would you mind to share the test steps here?

PhanLe1010 · 2022-11-04T18:07:17Z

Probably need to enable the PSP on this hardened cluster. Can you switch this field to true

longhorn/chart/values.yaml

Lines 253 to 255 in 23d2641

    
           #  For Kubernetes < v1.25, if your cluster enables Pod Security Policy admission controller, 
        
           #  set this to `true` to ship longhorn-psp which allow privileged Longhorn pods to start 
        
           enablePSP: false

chriscchien · 2022-11-05T00:37:50Z

Hi @PhanLe1010

Using helm to install Longhorn master head by command helm install longhorn ./chart -n longhorn-system which enablePSP set to true can deploy/uninstall Longhorn and workload worked well on the hardened cluster

For use kubectl to deploy Longhorn, I tried add below section into longhon.yaml to deploy but still have problem in longhorn-driver-deployer

---
# Source: longhorn/templates/psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: longhorn-psp
  labels:
    app.kubernetes.io/name: longhorn
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/version: v1.4.0-dev
spec:
  privileged: true
  allowPrivilegeEscalation: true
  requiredDropCapabilities:
  - NET_RAW
  allowedCapabilities:
  - SYS_ADMIN
  hostNetwork: false
  hostIPC: false
  hostPID: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - secret
  - projected
  - hostPath
---

cc @khushboo-rancher, @yangchiu

PhanLe1010 · 2022-11-05T00:50:20Z

longhorn.yaml is generated by running helm template . The default Helm value enablePSP is false so Helm template does not generate PSP yaml into longhorn.yaml. This behavior is expected because we are moving away from PSP. However, it is a bit inconvenient for the user using old hardened clusters with PSP enabled, therefore we have 2 options:

Document and pointing the user to PSP yaml
Put the PSP yaml under the comment section into the longhorn.yaml. This method would require us to modify the generate-longhorn-yaml.sh

cc @innobead Which option would you suggest from PM perspective?

PhanLe1010 · 2022-11-05T00:56:03Z

For use kubectl to deploy Longhorn, I tried add below section into longhon.yaml to deploy but still have problem in longhorn-driver-deployer

You also need to add PSP into longhorn-service-account. AKA, you need to put this whole section into longhorn.yaml

chriscchien · 2022-11-05T01:24:37Z

Thank you @PhanLe1010 ,

Deploy Longhorn on hardened cluster by kubectl and helm all worked well( including steps install, create/scale workload and uninstall).

innobead · 2022-11-07T03:41:43Z

longhorn.yaml is generated by running helm template . The default Helm value enablePSP is false so Helm template does not generate PSP yaml into longhorn.yaml. This behavior is expected because we are moving away from PSP. However, it is a bit inconvenient for the user using old hardened clusters with PSP enabled, therefore we have 2 options:

Document and pointing the user to PSP yaml

Put the PSP yaml under the comment section into the longhorn.yaml. This method would require us to modify the generate-longhorn-yaml.sh

cc @innobead Which option would you suggest from PM perspective?

I think we need to do both for users who use kubectl to apply the manifest directly. If in the future, it becomes complicated, we can use kustomize instead, but right now it's enough to have a separate manifest for < 1.25.

@PhanLe1010 Could you help with this? Thanks.

…ecated PodSecurityPolicy on k3s v1.25 (not yet fixed - longhorn/longhorn#4003)

longhorn issues Issue: longhorn/longhorn#4003 (comment)

See longhorn/longhorn#4003

chriscchien · 2022-12-14T07:53:15Z

In addition, tested upgrade Longhorn from old release to v1.40-rc1 then upgrade Kubernetes version to v1.25
Below scenario were all passed

In v1.24.7+k3s1

Upgrade Longhorn from v1.2.6 to v1.4.0-rc1 then upgrade kubernetes to v1.25.4+k3s1
Upgrade Longhorn from v1.3.2 to v1.4.0-rc1 then upgrade kubernetes to v1.25.4+k3s1

In v1.23.11+k3s1

Upgrade Longhorn from v1.2.6 to v1.4.0-rc1 then upgrade kubernetes to v1.25.4+k3s1
Upgrade Longhorn from v1.3.2 to v1.4.0-rc1 then upgrade kubernetes to v1.25.4+k3s1

innobead added area/kubernetes Kubernetes related like K8s version compatibility priority/0 Must be fixed in this release (managed by PO) kind/improvement Request for improvement of existing function backport/1.3.1 labels May 19, 2022

innobead added this to the v1.4.0 milestone May 19, 2022

innobead assigned PhanLe1010 May 19, 2022

innobead removed the backport/1.3.1 label Jun 24, 2022

innobead added backport/1.2.5 backport/1.3.2 labels Jul 29, 2022

innobead mentioned this issue Aug 10, 2022

[FEATURE] Update K8s version support and component/pkg/build dependencies #4239

Closed

6 tasks

innobead removed the backport/1.2.5 label Aug 24, 2022

mantissahz mentioned this issue Aug 26, 2022

Kubernetes v1.25 support #4490

Closed

mantissahz mentioned this issue Sep 15, 2022

[BUG] PodSecurityPolicy in the policy/v1beta1 API deprecated in Kubernetes v1.25 #4569

Closed

tyriis added a commit to tyriis/home-ops that referenced this issue Sep 15, 2022

fix(k3s): rollback due to issues with longhorn see longhorn/longhorn#…

5aabc17

…4003

c3y1huang mentioned this issue Sep 26, 2022

feat(system-backup/restore): add LEP #4388

Merged

innobead removed the backport/1.3.2 label Sep 27, 2022

innobead added the highlight Important feature/issue to highlight label Oct 5, 2022

mschirrmeister mentioned this issue Oct 27, 2022

[BUG] Failed to install Longhorn 1.3.2 on K3s 1.25.3 #4788

Open

ricsanfre mentioned this issue Oct 27, 2022

Upgrade Longhorn to 1.4., so Kubernetes 1.25 version can be installed ricsanfre/pi-cluster#82

Closed

chriscchien closed this as completed Oct 28, 2022

krittin added a commit to krittin/homeassistant-on-raspberrypi that referenced this issue Nov 11, 2022

pegged k3s at v1.24.7+k3s1, otherwise longhorn would fail due to depr…

31b3caa

…ecated PodSecurityPolicy on k3s v1.25 (not yet fixed - longhorn/longhorn#4003)

axeII added a commit to axeII/home-ops that referenced this issue Nov 28, 2022

chore: Downgrades kuberetnes version to 1.23 due

7f1ce5b

longhorn issues Issue: longhorn/longhorn#4003 (comment)

This was referenced Nov 28, 2022

Add PSP yaml #4954

Merged

Add PSP document longhorn/website#622

Merged

innobead added the release/note Note for installation, upgrade and highlighted issues label Nov 30, 2022

valtzu added a commit to valtzu/rpi-images that referenced this issue Dec 11, 2022

Downgrade k3s to v1.24.7+k3s1 due to longhorn incompatibility

dab6e49

See longhorn/longhorn#4003

innobead mentioned this issue Jan 3, 2023

[BUG] With longhorn 1.4.0, k8s 1.25, and enablePSP: false in Helm chart, still getting resource mapping not found for name: "longhorn-psp" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1" #5185

Closed

polds mentioned this issue May 18, 2023

[BUG] longhorn-manager: unable to get engine image longhornio/longhorn-engine:v1.4.2 after re-install #5963

Closed

ChanYiLin mentioned this issue May 19, 2023

[TASK] Knowledge Base for upgrading to incompatible Kubernetes Version #5964

Open

Phoenixfan mentioned this issue Jun 2, 2023

[BUG] longhorn manager fails to startup: unable to get image on v1.4.2 with K8s v1.25 #6051

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy #4003

[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy #4003

innobead commented May 19, 2022 •

edited

Loading

innobead commented Jul 29, 2022

mattmeye commented Sep 13, 2022

innobead commented Sep 13, 2022

daimoniac commented Sep 13, 2022

innobead commented Sep 13, 2022 •

edited

Loading

skandragon commented Sep 14, 2022

mattmeye commented Sep 14, 2022

PurpleBooth commented Sep 21, 2022 •

edited

Loading

innobead commented Sep 21, 2022

skandragon commented Sep 27, 2022

BjoernPetersen commented Sep 28, 2022

kjaleshire commented Sep 29, 2022

dominch commented Oct 5, 2022

MohammedNoureldin commented Oct 26, 2022 •

edited

Loading

PhanLe1010 commented Oct 26, 2022

MohammedNoureldin commented Oct 26, 2022

dkrizic commented Oct 26, 2022

chriscchien commented Oct 28, 2022 •

edited

Loading

khushboo-rancher commented Oct 28, 2022

abderraxim commented Oct 29, 2022 •

edited

Loading

nabeelshaikh7 commented Nov 2, 2022

chriscchien commented Nov 4, 2022

yangchiu commented Nov 4, 2022

PhanLe1010 commented Nov 4, 2022 •

edited

Loading

chriscchien commented Nov 5, 2022 •

edited

Loading

PhanLe1010 commented Nov 5, 2022 •

edited

Loading

PhanLe1010 commented Nov 5, 2022

chriscchien commented Nov 5, 2022 •

edited

Loading

innobead commented Nov 7, 2022

chriscchien commented Dec 14, 2022 •

edited

Loading

[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy #4003

[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy #4003

Comments

innobead commented May 19, 2022 • edited Loading

Is your improvement request related to a feature? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Additional context

innobead commented Jul 29, 2022

mattmeye commented Sep 13, 2022

innobead commented Sep 13, 2022

daimoniac commented Sep 13, 2022

innobead commented Sep 13, 2022 • edited Loading

skandragon commented Sep 14, 2022

mattmeye commented Sep 14, 2022

PurpleBooth commented Sep 21, 2022 • edited Loading

innobead commented Sep 21, 2022

skandragon commented Sep 27, 2022

BjoernPetersen commented Sep 28, 2022

kjaleshire commented Sep 29, 2022

dominch commented Oct 5, 2022

MohammedNoureldin commented Oct 26, 2022 • edited Loading

PhanLe1010 commented Oct 26, 2022

MohammedNoureldin commented Oct 26, 2022

dkrizic commented Oct 26, 2022

chriscchien commented Oct 28, 2022 • edited Loading

Case 1: within a Kubernetes version

Case 2: Test a Kubernetes version upgrade

khushboo-rancher commented Oct 28, 2022

abderraxim commented Oct 29, 2022 • edited Loading

nabeelshaikh7 commented Nov 2, 2022

chriscchien commented Nov 4, 2022

yangchiu commented Nov 4, 2022

PhanLe1010 commented Nov 4, 2022 • edited Loading

chriscchien commented Nov 5, 2022 • edited Loading

PhanLe1010 commented Nov 5, 2022 • edited Loading

PhanLe1010 commented Nov 5, 2022

chriscchien commented Nov 5, 2022 • edited Loading

innobead commented Nov 7, 2022

chriscchien commented Dec 14, 2022 • edited Loading

innobead commented May 19, 2022 •

edited

Loading

innobead commented Sep 13, 2022 •

edited

Loading

PurpleBooth commented Sep 21, 2022 •

edited

Loading

MohammedNoureldin commented Oct 26, 2022 •

edited

Loading

chriscchien commented Oct 28, 2022 •

edited

Loading

abderraxim commented Oct 29, 2022 •

edited

Loading

PhanLe1010 commented Nov 4, 2022 •

edited

Loading

chriscchien commented Nov 5, 2022 •

edited

Loading

PhanLe1010 commented Nov 5, 2022 •

edited

Loading

chriscchien commented Nov 5, 2022 •

edited

Loading

chriscchien commented Dec 14, 2022 •

edited

Loading