[BUG] Failed to install Longhorn 1.3.2 on K3s 1.25.3

[votdev]

Describe the bug

Longhorn 1.3.2 can not be installed on K3s v1.25.3. With K3s v1.24.7+k3s1 it is working.

To Reproduce

$ curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
[INFO]  Finding release for channel stable
[INFO]  Using v1.25.3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

$ k3s kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.3.2/deploy/prerequisite/longhorn-iscsi-installation.yaml
daemonset.apps/longhorn-iscsi-installation created

$ k3s kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.3.2/deploy/longhorn.yaml
namespace/longhorn-system unchanged
serviceaccount/longhorn-service-account configured
configmap/longhorn-default-setting configured
configmap/longhorn-storageclass configured
customresourcedefinition.apiextensions.k8s.io/backingimagedatasources.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/backingimagemanagers.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/backingimages.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/backups.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/backuptargets.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/backupvolumes.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/engineimages.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/engines.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/instancemanagers.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/nodes.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/orphans.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/recurringjobs.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/replicas.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/settings.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/sharemanagers.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/snapshots.longhorn.io configured
customresourcedefinition.apiextensions.k8s.io/volumes.longhorn.io configured
clusterrole.rbac.authorization.k8s.io/longhorn-role configured
clusterrolebinding.rbac.authorization.k8s.io/longhorn-bind configured
role.rbac.authorization.k8s.io/longhorn-psp-role configured
rolebinding.rbac.authorization.k8s.io/longhorn-psp-binding configured
service/longhorn-backend configured
service/longhorn-frontend configured
service/longhorn-conversion-webhook unchanged
service/longhorn-admission-webhook unchanged
service/longhorn-engine-manager configured
service/longhorn-replica-manager configured
daemonset.apps/longhorn-manager configured
deployment.apps/longhorn-driver-deployer configured
deployment.apps/longhorn-ui configured
deployment.apps/longhorn-conversion-webhook unchanged
deployment.apps/longhorn-admission-webhook unchanged
Error from server (NotFound): error when creating "https://raw.githubusercontent.com/longhorn/longhorn/v1.3.2/deploy/longhorn.yaml": the server could not find the requested resource

Expected behavior

It should install Longhorn using latest K3s.

[mschirrmeister]

Have a look at #4003

[PhanLe1010]

Longhorn 1.3.x doesn't support Kubernetes 1.25. We have to wait for Longhorn 1.4.0 release as mentioned in the issue #4003

[withinboredom]

This is mindblowing that this has happened, as there was no secret that these features were changed/deprecated, and 1.25 is now stable. I feel sorry for anyone updating their cluster (or worse, auto-updating). Even myself doing some due diligence before updating did not notice the incompatibility because I assumed it would be somewhere prominent (like on the front page or in the readme) instead of buried in issues or documentation. Now half my cluster is 1.25 and the other half 1.24. Thankfully there is still one admission controller running on the 1.24 side... while I rebuild the other half back to 1.24.

[xamindar]

This is unacceptable. Both are made by the same company, how can you let this happen? I upgraded my three etcd nodes to 1.25.3 which seemed fine (longhorn doesn't write volumes to them). But then I upgraded one of my two worker nodes and BAM! longhorn will not initialize on the one worker node. Tried going back to 1.24.3 on that worker and still broke. Now half my pods are stuck in "unknown" because none of the volumes will mount on that node.

No mention of any of this incompatibility on the upgrade page.

EDIT: #4003 has a solution that worked to fix it:
#4003 (comment)

[mschirrmeister]

Unacceptable and mindblowing is maybe a little bit harsh. What features exactly did you need from 1.25.x, that you upgraded?
Sure they can improve the documentation where there is a table that lists all dependencies to show all which versions work with what other components.

Besides that, I always recommend to people to run stage/development environment that matches production, where updates or any kind of changes are tested and verified before applied to production.
Even at home, I have a dev K3s cluster where I test everything before I apply it to my main home production K3s cluster.

[withinboredom]

I didn't mean for "mindblowing" to come across as harsh; maybe I should clarify. I found it mindblowing because of the support/documentation around configuring auto-upgrading clusters (Rancher/k3s/etc), and these projects are part of the same company. Surely someone should have realized what would happen and raised the issue internally?

Besides that, I always recommend to people to run stage/development environment that matches production

Either way, it is money/time lost, due to devs trying to save a dev/staging cluster.

[2fst4u]

I'm unashamedly calling it mind-blowing. For the same company that distributes K3S to release a stable update that causes breaking changes to their "native" storage solution has caused headaches for so many people now. K3S docs outline how to auto update your cluster on stable branch, which no doubt many people like me would have had in place. Automatically causing a breaking change.

As stated above, these breaking changes were already EOL, why were they still implemented in Longhorn up to the point of removal?

I'd implore Rancher to fast-track a fix for this. Am I getting the product I pay for? Sure, probably. I'm grateful for the product being free and open source. Do I have a right to be disappointed by the outcome of these events though? I think so. If this were a third party product I might have some understanding but for the same company to cause this is unfathomable.

@xamindar how are you finding the master branch? I don't want to screw anything up even more so I might have to wait for a "stable" branch update.

[xamindar]

@2fst4u The master branch has been fine on my setup (5 raspberry pi4 nodes). I have 10 volumes. Everything seems to be working correctly; volumes, mounts, backups to nfs. I don't think I will touch it again until the new stable version with this issue resolved is out, however.

[2fst4u]

I guess my next question is when an update eventually comes out, will upgrading from master to that new version work ok?

[excalq]

For what it's worth, Linode just upgraded our LKS cluster to 1.25, and our Longhorn statefulSets are all broken now. Hopefully I can get this figured out from the passengers seat of a car on a Saturday night!

[omidraha]

I think I have related issue,
I tried to install longhorn with helm chart on k3s .
The longhorn helm chart version is 1.5.2.
I don't have this issue #6930 when I installed longhorn on the eks.

The source code is available here:

https://github.com/omidraha/pulumi_example/blob/main/longhorn/longhorn.py
https://github.com/omidraha/pulumi_example/blob/main/longhorn/__main__.py

Info:

Diagnostics:
  kubernetes:helm.sh/v3:Release (longhorn-op):
    error: kubernetes:helm.sh/v3:Release resource 'longhorn-op': property chart value {longhorn} has a problem: chart requires kubeVersion: >=1.21.0-0 which is incompatible with Kubernetes v1.20.0; check the chart name and repository configuration.

Info:

k3s --version

k3s version v1.28.3+k3s1 (49411e70)
go version go1.20.10

Info:

$ kubectl version --output json

{
  "clientVersion": {
    "major": "1",
    "minor": "28",
    "gitVersion": "v1.28.3+k3s1",
    "gitCommit": "49411e7084f74b931882f1cf85bf2e17a5f02158",
    "gitTreeState": "clean",
    "buildDate": "2023-10-30T22:35:52Z",
    "goVersion": "go1.20.10",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.0.4-0.20230601165947-6ce0bf390ce3",
  "serverVersion": {
    "major": "1",
    "minor": "28",
    "gitVersion": "v1.28.3+k3s1",
    "gitCommit": "49411e7084f74b931882f1cf85bf2e17a5f02158",
    "gitTreeState": "clean",
    "buildDate": "2023-10-30T22:35:52Z",
    "goVersion": "go1.20.10",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

Related concept: defenseunicorns/zarf#1607

[EronWright]

I believe that the problem mentioned by @omidraha was due to an unrelated upstream issue (in Pulumi, see here) and was fixed on the Pulumi side.

[omidraha]

Yes, my issue fixed and I successfully deployed longhorn with pulumi:
https://github.com/omidraha/pulumi_example/tree/main/longhorn