[FEATURE] Talos support

[runningman84]

Is your feature request related to a problem? Please describe.
Talos does not support longhorn due to missing iscsi support:
siderolabs/talos#3057

Describe the solution you'd like
Other supported storage providers in talos use the nvmf protocol to overcome this issue.

Describe alternatives you've considered
I would really like to continue using longhorn, the other solutions have their our limitations like needing a seperated disk.

Additional context
With an uncertain future of k3os many users might take a look at talos.

[PhanLe1010]

The current Longhorn engine is dependent on iscsi.
We are working on the next generation of Longhorn engine that uses spdk for better performance. However, it may take a long time for this version to become as stable and feature-rich as the current version.

#760
#3044

I am curious why k3os's future is uncertain?

[larssb]

Sounds good @PhanLe1010 ... that works is ongoing in regards to the next generation of the Longhorn engine.

On K3OS see:

• [FEATURE] Replace K3OS harvester/harvester#581 <-- where @yasker talks about K3OS not having official support. Not the same as the project being deprecated or not developed further. But, that fact just raises peoples awareness towards K3OS maybe not being the best option if you're building a Kubernetes platform in an enterprise setting
• And as an outsider it's easy to get into your head that now that SuSe acquired RancherLabs why wouldn't they take the best of K3OS and integrate that into an OS they already have - or simply propose another SuSe OS that they already have as simply being a replacement for K3OS.

[HaveFun83]

iscsi-tools are now support via system extensions on the talos side.
Maybe the settings how longhorn communicate with the iscsi-daemon can be made configurable?

https://www.talos.dev/v1.1/kubernetes-guides/configuration/replicated-local-storage-with-openebs-jiva/

[skandragon]

I fear it's more than just the settings. I fear it's also that bash and other tools are used. There's a somewhat long list of tools:

The host filesystem supports the file extents feature to store the data. Currently we support: ext4, XFS

curl, findmnt, grep, awk, blkid, lsblk must be installed.

[innobead]

ref:

• https://www.talos.dev/v1.2/kubernetes-guides/configuration/replicated-local-storage-with-openebs-jiva/#patching-the-jiva-installation
• https://github.com/longhorn/go-iscsi-helper/blob/ebff48f3632ad7549e8ae516d730aa42321f9d53/util/process.go#L54-L65 (need to be aware of the installed iscsid daemon by Talos)

Also, need a customized tgt system extension

[c3y1huang]

Note:

• The iscsiadm can be executed using nsenter approach from its ancestor namespace.
• We should replace all dependent binary invocations. To achieve this, we need to implement namespace entering using setns or alternatives. This is also related to [REFACTOR] Use go pkg for system operation instead of relying on external system call via shell command #5193.
• We also need to replace the binary dependencies in the automation tests.

I've written a PoC to test the feasibility of the changes. And will revisit this issue later to provide a proper LEP.

[runningman84]

Cool so see my feature request implemented…

[ccben87]

I really hope this will be merged soon. Otherwise I might have to look at alternatives :(

[CRASH-Tech]

I really hope this will be merged soon. Otherwise I might have to look at alternatives :(

Same situation

[zimmertr]

If you're using Proxmox someone has recently started developing a CSI driver that works with ZFS, LVM, Local, etc. Storage types. I'm using it with ZFS quite happily now.

[ccben87]

If you're using Proxmox someone has recently started developing a CSI driver that works with ZFS, LVM, Local, etc. Storage types. I'm using it with ZFS quite happily now.

I am using Nutanix AHV which is KVM under the surface so it might work? Maybe not? What is this project?

[zimmertr]

Review the hyperlinks you quoted 😉

As for Nutanix, no. The plugin directly interacts with the Proxmox API. But if it works it works well. I'm directly using block storage in K8s now. It's awesome.

[ccben87]

Review the hyperlinks you quoted 😉

Oh, duh, it's late here.

As for Nutanix, no. The plugin directly interacts with the Proxmox API. But if it works it works well. I'm directly using block storage in K8s now. It's awesome.

That's a shame. Need this to be merged then.

[bdsoha]

@c3y1huang Any updates on the merge process?

[innobead]

@c3y1huang Any updates on the merge process?

The PRs will be released in the near future and planned for the upcoming 1.6.0 release.

[c3y1huang]

Test results (core):

• Talos (v1.5.5)
  • S3

    = 63 passed, 1 skipped, 290 deselected, 27 warnings in 7091.03s (1:58:11) =
    

  • NFS

    = 64 passed, 290 deselected, 28 warnings in 7072.83s (1:57:52) =
    

• SUSE (SLES15-SP5)
  • S3: https://ci.longhorn.io/job/private/job/longhorn-tests-regression/5232/

    = 63 passed, 1 skipped, 290 deselected, 28 warnings in 6878.20s (1:54:38) =
    

  • NFS: https://ci.longhorn.io/job/private/job/longhorn-tests-regression/5233/

    = 64 passed, 290 deselected, 28 warnings in 7014.31s (1:56:54) =
    

[c3y1huang]

Test result (full):

• Talos (v1.5.5)
  • S3

  = 336 passed, 18 skipped, 223 warnings in 50413.40s (14:00:13) =
  

  • NFS

  = 335 passed, 19 skipped, 246 warnings in 50567.20s (14:02:47) =
  

• SUSE (SLES15-SP5)
  • S3: https://ci.longhorn.io/job/private/job/longhorn-tests-regression/5234/

  = 337 passed, 17 skipped, 217 warnings in 50205.38s (13:56:45) =
  

  • NFS: https://ci.longhorn.io/job/private/job/longhorn-tests-regression/5235/

  = 336 passed, 18 skipped, 216 warnings in 50201.84s (13:56:41) =
  

[Ornias1993]

To ensure it actually makes 1.6.0, it might be best to give the PR's a bit more pro-active attention @innobead

[boixu]

Hi, I am also very much interested in this. Keep up the great work!

[c3y1huang]

Currently I am experimenting Longhorn with Talos v1.6.0. Somehow, patching the machine kubelet extra-mount isn't working as in Talos v1.5.5. cc @frezbo

machine:
  kubelet:
    extraMounts:
      - destination: /var/lib/longhorn
        type: bind
        source: /var/lib/longhorn
        options:
          - bind
          - rshared
          - rw

[frezbo]

Currently I am experimenting Longhorn with Talos v1.6.0. Somehow, patching the machine kubelet extra-mount isn't working as in Talos v1.5.5. cc @frezbo

machine:
  kubelet:
    extraMounts:
      - destination: /var/lib/longhorn
        type: bind
        source: /var/lib/longhorn
        options:
          - bind
          - rshared
          - rw

what's the error?

[c3y1huang]

what's the error?

There is no error, the mount didn't seem to apply somehow.

[frezbo]

what's the error?

There is no error, the mount didn't seem to apply somehow.

Probably check the kubelet and controller-runtime logs

talosctl logs kubelet talosctl logs controller-runtime

[c3y1huang]

Probably check the kubelet and controller-runtime logs

talosctl logs kubelet talosctl logs controller-runtime

Thank you @frezbo . It appears this was a false alarm caused by a typo in my launch template.

[c3y1huang]

Test result (full):

• Talos (v1.6.0)
  • S3

  = 2 failed, 341 passed, 22 skipped, 254 warnings in 50724.13s (14:05:24) =
  

  • NFS

  = 2 failed, 340 passed, 23 skipped, 233 warnings in 50157.04s (13:55:57) =
  

The 2 failed test cases seem unrelated to this feature.

• test_engine_image_incompatible: Failed to pull image "longhornio/longhorn-test:version-test.7-7.5-4.1-1"
• test_engine_live_upgrade_with_intensive_data_writing: already been reported by QA. https://suse.slack.com/archives/C02DR3N5T24/p1703804384321789?thread_ts=1703775609.253649&cid=C02DR3N5T24

[innobead]

The 2 failed test cases seem unrelated to this feature.

• test_engine_image_incompatible: Failed to pull image "longhornio/longhorn-test:version-test.7-7.5-4.1-1"

@chriscchien Please help with this. @longhorn/qa

[yangchiu]

Verified passed on master-head (longhorn-manager b2132e7) following longhorn/website#827 to setup Talos environment, but extra config for Pod Security is also needed:

cluster:
  apiServer:
    admissionControl:
      - name: PodSecurity
        configuration:
          apiVersion: pod-security.admission.config.k8s.io/v1alpha1
          defaults:
            enforce: privileged
          kind: PodSecurityConfiguration

Longhorn can be installed successfully in the configured Talos environment. And longhorn-tests coretest has been run based on longhorn/longhorn-tests#1501, there is no outstanding issue found.

[camaeel]

Verified passed on master-head (longhorn-manager b2132e7) following longhorn/website#827 to setup Talos environment, but extra config for Pod Security is also needed:

cluster:
  apiServer:
    admissionControl:
      - name: PodSecurity
        configuration:
          apiVersion: pod-security.admission.config.k8s.io/v1alpha1
          defaults:
            enforce: privileged
          kind: PodSecurityConfiguration

Longhorn can be installed successfully in the configured Talos environment. And longhorn-tests coretest has been run based on longhorn/longhorn-tests#1501, there is no outstanding issue found.

Instead lowering PodSecurity for whole cluster it would be better either to label longhorn-system namespace or put it into PodSecurityConfiguration as exemption (https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/#configure-the-admission-controller).

[yangchiu]

Instead lowering PodSecurity for whole cluster it would be better either to label longhorn-system namespace or put it into PodSecurityConfiguration as exemption (https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/#configure-the-admission-controller).

@c3y1huang Would you like to add this to the documentation?

[c3y1huang]

@c3y1huang Would you like to add this to the documentation?

Longhorn's documentation has links directing to the Talos documentation, which offers various approaches for handling this (including the namespace approach).