Releases: longhorn/longhorn
Release list
Longhorn v1.12.1
Longhorn v1.12.1 Release Notes
Longhorn v1.12.1 introduces V2 Data Engine fast volume cloning and storage sharding (experimental), along with important improvements and bug fixes that enhance system quality, resilience, stability, and security.
We welcome feedback and contributions to help continuously improve Longhorn.
For terminology and context on Longhorn releases, see Releases.
Breaking Changes
Deprecation of Legacy V2 Linked Clone Volumes
V2 linked-clone volumes created in v1.12.0 or earlier are marked as legacy and deprecated starting in v1.12.1. The new linked-clone architecture introduced in Issue #12552 is not compatible with the legacy design.
After upgrading to v1.12.1, legacy linked-clone volumes cannot be operated on except for detachment and deletion.
To replace, create new linked-clone volumes from the same source volumes that back the legacy ones. As long as a legacy volume exists, its source volume is guaranteed to still be present, so you can create a replacement linked clone directly; no data copy is required.
For more information, see Issue #12552.
Highlighted Features
Fast Volume Cloning
Longhorn v1.12.1 enhances fast volume cloning for the V2 Data Engine. A linked-clone volume shares data blocks with its source instead of copying data. With the new architecture, a source replica can share its data blocks with multiple linked-clone volumes, and multiple clone replicas can be created in parallel.
Linked-clone volumes now support most operations available to regular volumes, including snapshots, expansion, replica rebuilding, and use as the source of nested linked clones.
For more information, see Issue #12552 and CSI Volume Clone.
Note
V2 fast cloning does not currently support volume backup and restore. This improvement is tracked in Issue #13714.
Storage Sharding (Experimental)
Longhorn v1.12.1 introduces storage sharding as an experimental data protection and storage layout feature built on the V2 Data Engine. Instead of storing a full copy of the volume on each replica, sharding uses erasure coding to encode written data into data and parity chunks, which are distributed across multiple nodes. This allows a volume to grow beyond the capacity of a single disk or node while using less disk space to achieve the same level of fault tolerance.
Because this feature is experimental, it is intended for evaluation and testing only and is not recommended for production use.
For more information, see Issue #1061 and Sharding with Erasure Coding.
Important Improvements and Fixes
This release includes several important improvements and critical stability fixes.
Internal Network Policies
Longhorn v1.12.1 enables ingress NetworkPolicy resources for internal component endpoints and RPCs by default to improve security by restricting access to Longhorn internal services, including the instance-manager gRPC endpoint used for engine control. These policies only take effect when a NetworkPolicy provider is available in the cluster.
Longhorn has validated the internal network policies with the following Kubernetes distribution and CNI plugin combinations. See the CNI Plugin Compatibility table for the validated combinations. The minimum required Kubernetes version is v1.25.
For more information and troubleshooting guidance, see Internal Network Policies and Issue #13438.
Note
ServiceMonitor discovery does not automatically authorize network traffic. Cross-namespace Prometheus scrapers might be blocked by the Longhorn Manager's network policy. To allow this traffic, apply a scoped additive policy as detailed in the Prometheus and Grafana setup guide.
Instance Manager gRPC mTLS Coverage
In previous versions, mutual TLS (mTLS) for the instance-manager gRPC endpoint only covered the instance and proxy services when the longhorn-grpc-tls secret was configured. Other services, including the disk service and the SPDK service, accepted plaintext connections.
Longhorn v1.12.1 extends mTLS to all remaining instance-manager gRPC services, so every gRPC port now requires a valid client certificate when the longhorn-grpc-tls secret is configured.
For more information, see Issue #7787.
CPU Core Allocation with the Kubernetes CPU Manager
Longhorn v1.12.1 can allocate exclusive CPU cores to the SPDK target daemon, which runs in each V2 Instance Manager pod, through the Kubernetes CPU Manager by using the data-engine-number-of-cpu-cores setting.
The setting can be applied only when the kubelet CPU Manager policy is set to static on all worker nodes; otherwise, the update is rejected. When the value is positive, it takes precedence, and data-engine-cpu-mask is ignored.
For more information, see Issue #13248.
Host CPU Isolation
The data-engine-cpu-isolation-enabled setting now also configures host network Receive Packet Steering (RPS) to steer RX softirq processing away from the CPU cores used by the SPDK target daemon, in addition to hardware IRQs and unbound kernel workqueue workers. Without this, the kernel can distribute incoming network packets to the SPDK reactor cores, and the resulting softirq work competes with the reactor's busy-poll loop, degrading volume I/O under network load.
For more information, see Issue #13483 and Issue #13502.
V2 Data Engine SPDK iobuf Pool Size Configuration
Longhorn v1.12.1 allows tuning the SPDK iobuf buffer pools used by the V2 Data Engine. The data-engine-iobuf-large-pool-size and data-engine-iobuf-small-pool-size settings configure the large and 8 KiB small buffer pools, respectively. Increasing the small pool can relieve buffer exhaustion under high-queue-depth workloads with small I/O sizes. Because iobuf pools can only be sized at SPDK target startup, changing either setting recreates V2 Instance Manager pods that have no running instances.
For more information, see Issue #13322 and Issue #13674.
Encrypted Volume Size Correction
Longhorn reserves an additional 16 MiB of raw capacity for the LUKS2 metadata used by encrypted volumes, allowing the mapped device to expose the full capacity requested by the workload. Previously, the metadata was taken from usable capacity, so a requested 1 GiB encrypted volume exposed only 1008 MiB. This discrepancy could cause operations such as block-level copies between equally sized unencrypted and encrypted volumes to fail.
- V1 Data Engine: This correction was introduced in Longhorn v1.12.0. Existing encrypted V1 volumes created with v1.11.x or earlier receive the additional capacity automatically when their engine image is upgraded to v1.12 or later. Encrypted migratable V1 volumes cannot be live-migrated until they are upgraded to the version (>= v1.12.0).
- V2 Data Engine: Longhorn v1.12.1 applies the correction to newly created encrypted V2 volumes.
Note
Encrypted V2 volumes created before v1.12.1, and volumes restored from the backup of such volumes, do not receive the additional 16 MiB of raw capacity and continue to expose 16 MiB less than requested. Existing data is preserved.
For more information, see Issue #9205 and Issue #13163.
Installation
Important
Ensure that your cluster is running Kubernetes v1.25 or later before installing Longhorn v1.12.1.
You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.
Upgrade
Important
Ensure that your cluster is running Kubernetes v1.25 or later before installing Longhorn v1.12.1.
Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.
Post-Release Known Issues
For information about issues identified after this release, see Release-Known-Issues.
Resolved Issues in this release
Highlight
- [BACKPORT][v1.12.1][UI][FEATURE] V2 Data Engine Fast Cloning 13672 - @shuo-wu @roger-ryao
- [BACKPORT][v1.12.1][FEATURE] V2 Data Engine Sharding - Experimental 13176 - @c3y1huang @chriscchien
- [BACKPORT][v1.12.1][FEATURE] V2 Data Engine Fast Cloning 13174 - @shuo-wu @roger-ryao
Feature
- [BACKPORT][v1.12.1][FEATURE] Support Kubernetes CPU Manager for Longhorn V2 instance-mana...
Longhorn v1.12.1-rc4
DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.
Resolved Issues in this release
Highlight
- [BACKPORT][v1.12.1][UI][FEATURE] V2 Data Engine Fast Cloning 13672 - @shuo-wu @roger-ryao
- [BACKPORT][v1.12.1][FEATURE] V2 Data Engine Sharding - Experimental 13176 - @c3y1huang @chriscchien
- [BACKPORT][v1.12.1][FEATURE] V2 Data Engine Fast Cloning 13174 - @shuo-wu @roger-ryao
Feature
- [BACKPORT][v1.12.1][FEATURE] Support Kubernetes CPU Manager for Longhorn V2 instance-manager SPDK CPU assignment 13320 - @yangchiu @mantissahz @Copilot
Improvement
- [BACKPORT][v1.12.1][IMPROVEMENT] Improve V2 Engine Frontend Handoff Tolerance in Split Topology 13569 - @davidcheng0922 @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Allow configuring SPDK iobuf small pool size 13675 - @yangchiu @hookak
- [BACKPORT][v1.12.1][IMPROVEMENT] go-common-libs: kill the child process when command execution times out 13621 - @hookak
- [BACKPORT][v1.12.1][IMPROVEMENT] always setup NetworkPolicy for the internal communication 13439 - @COLDTURNIP @roger-ryao
- [BACKPORT][v1.12.1][IMPROVEMENT] Allow persisting number of hugepages using
longhornctl13535 - @chriscchien @bachmanity1 - [BACKPORT][v1.12.1][IMPROVEMENT] Steer host RPS away from SPDK reactor cores 13502 - @bachmanity1 @roger-ryao
- [BACKPORT][v1.12.1][IMPROVEMENT] updateBackupCompressionMethod may write the Volume even when the method is unchanged 13480 - @yangchiu
- [BACKPORT][v1.12.1][IMPROVEMENT] Add metrics to collect information about V2 data engine usage 13262 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Improving error transparency for volume attachment failure 13431 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Allow configuring SPDK iobuf large pool size 13415 - @chriscchien @bachmanity1
- [BACKPORT][v1.12.1][IMPROVEMENT] V2 volume write I/O stalls(~10s) when a replica is removed during migration 13310 - @hookak @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Support mTLS encrypted communication for remaining gRPC services in instance manager 13299 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.12.1][IMPROVEMENT] Add metrics to collect information about LONGHORN_DISTRO 13253 - @derekbit @chriscchien
Bug
- [BACKPORT][v1.12.1][BUG] Host OS nvmf-autoconnect connects kernel initiators to v2 replica subsystems, stalling volume attach/detach for minutes 13660 - @hookak @chriscchien
- [BACKPORT][v1.12.1][BUG] Longhorn Helm Chart NetworkPolicies do not honor new RKE2 "rke2-traefik" ingress controller 13665 - @COLDTURNIP @roger-ryao
- [BACKPORT][v1.12.1][BUG] Longhorn may try to attach volumes to a node without valid IM pod during the clone 13640 - @yangchiu @shuo-wu
- [BACKPORT][v1.12.1][BUG] Test case
test_volume_scheduling_failurefails on v2 volumes 13656 - @yangchiu @c3y1huang - [BACKPORT][v1.12.1][BUG] v2 volumes might get stuck in
deletingstate 13586 - @davidcheng0922 - [BACKPORT][v1.12.1][BUG]
Backup Listing With More Than 1000 Backupsfails on v2 volume due to an empty replica address in the backup status 13612 - @COLDTURNIP @chriscchien - [BACKPORT][v1.12.1][BUG] Encrypted rwo volume stuck in
attachedstate and cannotdetachwhen a privileged pod with host /var/run mounted exists on the same node. 13604 - - [BACKPORT][v1.12.1][BUG] Incorrect Web Link in GUI 13539 - @yangchiu @sushant-suse
- [BACKPORT][v1.12.1][BUG] Encrypted V2 volume size is 16MB short of the claimed size 13175 - @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] CSI pods do not respect anti-affinity preset update 13548 - @chriscchien @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Failed to add v2 block disk with virtio-scsi BDF path 13475 - @chriscchien @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] V2 encrypted volume keeps switching between Attaching and Detaching state after expand operation 13562 - @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] V1 volumes not rebuilding after cluster shutdown 13583 - @COLDTURNIP
- [BACKPORT][v1.12.1][BUG] GCS backup target: backup of large volume fails at final .cfg PUT with SignatureDoesNotMatch (residual of #12676 in v1.12.0) 13574 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][BUG] Can not use v2 volume as block disk 13564 - @davidcheng0922
- [BACKPORT][v1.12.1][BUG] Fail to restore a volume from a full backup if a previous backup is corrupted 13538 - @yangchiu @derekbit
- [BACKPORT][v1.12.1][BUG] Test case
Recurring Job Pod Should Not Crashfails 13568 - @yangchiu @c3y1huang - [BACKPORT][v1.12.1][BUG] Longhorn 1.12.0: AWS chunked encoding not supported with OCI S3 buckets 13478 - @derekbit @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] Adding V2 disk using /dev/disk/by-path/scsi-* path fails 13559 - @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] When using v2-data-engine, the virtio-scsi virtual disk is not recognized as a block device, returning the error: "not a block device: exit status 32". 13560 - @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Error logs in longhorn-uninstall job 13549 - @yangchiu @c3y1huang
- [BACKPORT][v1.12.1][BUG] go-spdk-helper JSON-RPC client leaks pending-request bookkeeping forever when a response never arrives after timeout 13554 -
- [BACKPORT][v1.12.1][BUG] V2 Data Engine: UBLK fails with EINVAL on Linux kernel 6.17.0 13274 - @chriscchien @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Kernel Workqueue Lockup and Unstable RKE2 Service After Enabling LH V2 in Harvester 13495 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][BUG] V2 expansion can report success while the engine remains at the old size 13380 - @davidcheng0922 @chriscchien
- [BACKPORT][v1.12.1][BUG] FilesystemReadOnly never detected on kernel >= 6.12 — ext4 reports emergency_ro, not ro; read-only auto-remount silently inoperative 13482 - @yangchiu
- [BACKPORT][v1.12.1][BUG] csi.ReplicaCount Helm values silently no-op on existing csi- deployments (only applied at first creation) 13465 - @roger-ryao
- [BACKPORT][v1.12.1][BUG] v2 volume may crash again after the auto reattachment 13337 - @shuo-wu @roger-ryao
- [BACKPORT][v1.12.1][BUG] V2 Volume Cannot Be Attached When the Storage Network Is Enabled 13490 - @c3y1huang
- [BACKPORT][v1.12.1][BUG] v2 volume repeated replica reuse failure 13336 - @shuo-wu @chriscchien
- [BACKPORT][v1.12.1][BUG] V2 Encrypted Volume Restore Fails 13365 - @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] V2 backup/snapshot can leave NVMe/TCP frontend or dm device stale, causing pod EIO on attached volumes 13332 - @davidcheng0922 @chriscchien
- [BACKPORT][v1.12.1][BUG] (chart) ArgoCD OutOfSync when using Gateway API 13446 - @yangchiu
- [BACKPORT][v1.12.1][BUG] Migration Engine Can Be Unexpectedly Deleted If the Target Node Is Still in Readiness Transition 13367 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.12.1][BUG] Recurring trim job fails with deadlock 13425 - @c3y1huang @roger-ryao
- [BACKPORT][v1.12.1][BUG] volume expansion stuck 13368 - @shuo-wu @chriscchien
- [BACKPORT][v1.12.1][BUG] pvc resize fails after iscsid restart 13412 - @yangchiu @shuo-wu
- [BACKPORT][v1....
Longhorn v1.12.1-rc3
DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.
Resolved Issues in this release
Highlight
- [BACKPORT][v1.12.1][FEATURE] V2 Data Engine Sharding - Experimental 13176 - @c3y1huang @chriscchien
- [BACKPORT][v1.12.1][FEATURE] V2 Data Engine Fast Cloning 13174 - @shuo-wu @roger-ryao
Feature
- [BACKPORT][v1.12.1][FEATURE] Support Kubernetes CPU Manager for Longhorn V2 instance-manager SPDK CPU assignment 13320 - @yangchiu @mantissahz @Copilot
Improvement
- [BACKPORT][v1.12.1][IMPROVEMENT] go-common-libs: kill the child process when command execution times out 13621 - @hookak
- [BACKPORT][v1.12.1][IMPROVEMENT] always setup NetworkPolicy for the internal communication 13439 - @COLDTURNIP @roger-ryao
- [BACKPORT][v1.12.1][IMPROVEMENT] Allow persisting number of hugepages using
longhornctl13535 - @chriscchien @bachmanity1 - [BACKPORT][v1.12.1][IMPROVEMENT] Improve V2 Engine Frontend Handoff Tolerance in Split Topology 13569 - @davidcheng0922
- [BACKPORT][v1.12.1][IMPROVEMENT] Steer host RPS away from SPDK reactor cores 13502 - @bachmanity1 @roger-ryao
- [BACKPORT][v1.12.1][IMPROVEMENT] updateBackupCompressionMethod may write the Volume even when the method is unchanged 13480 - @yangchiu
- [BACKPORT][v1.12.1][IMPROVEMENT] Add metrics to collect information about V2 data engine usage 13262 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Improving error transparency for volume attachment failure 13431 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Allow configuring SPDK iobuf large pool size 13415 - @chriscchien @bachmanity1
- [BACKPORT][v1.12.1][IMPROVEMENT] V2 volume write I/O stalls(~10s) when a replica is removed during migration 13310 - @hookak @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Support mTLS encrypted communication for remaining gRPC services in instance manager 13299 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.12.1][IMPROVEMENT] Add metrics to collect information about LONGHORN_DISTRO 13253 - @derekbit @chriscchien
Bug
- [BACKPORT][v1.12.1][BUG] Longhorn may try to attach volumes to a node without valid IM pod during the clone 13640 - @shuo-wu
- [BACKPORT][v1.12.1][BUG] v2 volumes might get stuck in
deletingstate 13586 - @davidcheng0922 - [BACKPORT][v1.12.1][BUG]
Backup Listing With More Than 1000 Backupsfails on v2 volume due to an empty replica address in the backup status 13612 - @COLDTURNIP @chriscchien - [BACKPORT][v1.12.1][BUG] Encrypted rwo volume stuck in
attachedstate and cannotdetachwhen a privileged pod with host /var/run mounted exists on the same node. 13604 - - [BACKPORT][v1.12.1][BUG] Incorrect Web Link in GUI 13539 - @yangchiu @sushant-suse
- [BACKPORT][v1.12.1][BUG] Encrypted V2 volume size is 16MB short of the claimed size 13175 - @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] CSI pods do not respect anti-affinity preset update 13548 - @chriscchien @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Failed to add v2 block disk with virtio-scsi BDF path 13475 - @chriscchien @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] V2 encrypted volume keeps switching between Attaching and Detaching state after expand operation 13562 - @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] V1 volumes not rebuilding after cluster shutdown 13583 - @COLDTURNIP
- [BACKPORT][v1.12.1][BUG] GCS backup target: backup of large volume fails at final .cfg PUT with SignatureDoesNotMatch (residual of #12676 in v1.12.0) 13574 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][BUG] Can not use v2 volume as block disk 13564 - @davidcheng0922
- [BACKPORT][v1.12.1][BUG] Fail to restore a volume from a full backup if a previous backup is corrupted 13538 - @yangchiu @derekbit
- [BACKPORT][v1.12.1][BUG] Test case
Recurring Job Pod Should Not Crashfails 13568 - @yangchiu @c3y1huang - [BACKPORT][v1.12.1][BUG] Longhorn 1.12.0: AWS chunked encoding not supported with OCI S3 buckets 13478 - @derekbit @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] Adding V2 disk using /dev/disk/by-path/scsi-* path fails 13559 - @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] When using v2-data-engine, the virtio-scsi virtual disk is not recognized as a block device, returning the error: "not a block device: exit status 32". 13560 - @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Error logs in longhorn-uninstall job 13549 - @yangchiu @c3y1huang
- [BACKPORT][v1.12.1][BUG] go-spdk-helper JSON-RPC client leaks pending-request bookkeeping forever when a response never arrives after timeout 13554 -
- [BACKPORT][v1.12.1][BUG] V2 Data Engine: UBLK fails with EINVAL on Linux kernel 6.17.0 13274 - @chriscchien @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Kernel Workqueue Lockup and Unstable RKE2 Service After Enabling LH V2 in Harvester 13495 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][BUG] V2 expansion can report success while the engine remains at the old size 13380 - @davidcheng0922 @chriscchien
- [BACKPORT][v1.12.1][BUG] FilesystemReadOnly never detected on kernel >= 6.12 — ext4 reports emergency_ro, not ro; read-only auto-remount silently inoperative 13482 - @yangchiu
- [BACKPORT][v1.12.1][BUG] csi.ReplicaCount Helm values silently no-op on existing csi- deployments (only applied at first creation) 13465 - @roger-ryao
- [BACKPORT][v1.12.1][BUG] v2 volume may crash again after the auto reattachment 13337 - @shuo-wu @roger-ryao
- [BACKPORT][v1.12.1][BUG] V2 Volume Cannot Be Attached When the Storage Network Is Enabled 13490 - @c3y1huang
- [BACKPORT][v1.12.1][BUG] v2 volume repeated replica reuse failure 13336 - @shuo-wu @chriscchien
- [BACKPORT][v1.12.1][BUG] V2 Encrypted Volume Restore Fails 13365 - @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] V2 backup/snapshot can leave NVMe/TCP frontend or dm device stale, causing pod EIO on attached volumes 13332 - @davidcheng0922 @chriscchien
- [BACKPORT][v1.12.1][BUG] (chart) ArgoCD OutOfSync when using Gateway API 13446 - @yangchiu
- [BACKPORT][v1.12.1][BUG] Migration Engine Can Be Unexpectedly Deleted If the Target Node Is Still in Readiness Transition 13367 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.12.1][BUG] Recurring trim job fails with deadlock 13425 - @c3y1huang @roger-ryao
- [BACKPORT][v1.12.1][BUG] volume expansion stuck 13368 - @shuo-wu @chriscchien
- [BACKPORT][v1.12.1][BUG] pvc resize fails after iscsid restart 13412 - @yangchiu @shuo-wu
- [BACKPORT][v1.12.1][BUG] expanding the volume fails 13384 - @chriscchien
- [BACKPORT][v1.12.1][BUG] Test case
test_rwx_delete_share_manager_podfails because it's unable to find the exported volume in share manager pod after it's deleted and restarted 13226 - @davidcheng0922 @roger-ryao - [BACKPORT][v1.12.1][BUG] System Backup RecurringJob retention prunes newest CR — sorts by Status.CreatedAt (zero for Error/racing CRs) 13209 - @roger-ryao
- [BACKPORT][v1.12.1][BUG] CSI components may have 0 running replica during upgrade 13348 - @yangchiu @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Node update forces a complete rebuild 13357 - @mantissahz
- [BACKPORT][v1.12.1][BUG] C...
Longhorn v1.12.1-rc1
DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.
Resolved Issues in this release
Highlight
- [BACKPORT][v1.12.1][FEATURE] V2 Data Engine Sharding - Experimental 13176 - @c3y1huang @chriscchien
- [FEATURE] Support v2 Data Engine (GA) 6229 - @derekbit
- [BACKPORT][v1.12.1][FEATURE] V2 Data Engine Fast Cloning 13174 - @shuo-wu
Feature
- [BACKPORT][v1.12.1][FEATURE] Support Kubernetes CPU Manager for Longhorn V2 instance-manager SPDK CPU assignment 13320 - @yangchiu @mantissahz @Copilot
Improvement
- [BACKPORT][v1.12.1][IMPROVEMENT] Allow persisting number of hugepages using
longhornctl13535 - @bachmanity1 - [BACKPORT][v1.12.1][IMPROVEMENT] Steer host RPS away from SPDK reactor cores 13502 - @bachmanity1 @roger-ryao
- [BACKPORT][v1.12.1][IMPROVEMENT] always setup NetworkPolicy for the internal communication 13439 - @COLDTURNIP @roger-ryao
- [BACKPORT][v1.12.1][IMPROVEMENT] updateBackupCompressionMethod may write the Volume even when the method is unchanged 13480 - @yangchiu
- [BACKPORT][v1.12.1][IMPROVEMENT] Add metrics to collect information about V2 data engine usage 13262 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Improving error transparency for volume attachment failure 13431 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Allow configuring SPDK iobuf large pool size 13415 - @chriscchien @bachmanity1
- [BACKPORT][v1.12.1][IMPROVEMENT] V2 volume write I/O stalls(~10s) when a replica is removed during migration 13310 - @hookak @chriscchien
- [BACKPORT][v1.12.1][IMPROVEMENT] Support mTLS encrypted communication for remaining gRPC services in instance manager 13299 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.12.1][IMPROVEMENT] Add metrics to collect information about LONGHORN_DISTRO 13253 - @derekbit @chriscchien
Bug
- [BACKPORT][v1.12.1][BUG] Incorrect Web Link in GUI 13539 - @yangchiu @sushant-suse
- [BACKPORT][v1.12.1][BUG] Fail to restore a volume from a full backup if a previous backup is corrupted 13538 - @mantissahz
- [BACKPORT][v1.12.1][BUG] Encrypted V2 volume size is 16MB short of the claimed size 13175 - @mantissahz
- [BACKPORT][v1.12.1][BUG] V2 Data Engine: UBLK fails with EINVAL on Linux kernel 6.17.0 13274 - @chriscchien @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Kernel Workqueue Lockup and Unstable RKE2 Service After Enabling LH V2 in Harvester 13495 - @derekbit @chriscchien
- [BACKPORT][v1.12.1][BUG] V2 expansion can report success while the engine remains at the old size 13380 - @davidcheng0922 @chriscchien
- [BACKPORT][v1.12.1][BUG] FilesystemReadOnly never detected on kernel >= 6.12 — ext4 reports emergency_ro, not ro; read-only auto-remount silently inoperative 13482 - @yangchiu
- [BACKPORT][v1.12.1][BUG] csi.ReplicaCount Helm values silently no-op on existing csi- deployments (only applied at first creation) 13465 - @roger-ryao
- [BACKPORT][v1.12.1][BUG] v2 volume may crash again after the auto reattachment 13337 - @shuo-wu @roger-ryao
- [BACKPORT][v1.12.1][BUG] V2 Volume Cannot Be Attached When the Storage Network Is Enabled 13490 - @c3y1huang
- [BACKPORT][v1.12.1][BUG] v2 volume repeated replica reuse failure 13336 - @shuo-wu @chriscchien
- [BACKPORT][v1.12.1][BUG] V2 Encrypted Volume Restore Fails 13365 - @mantissahz @roger-ryao
- [BACKPORT][v1.12.1][BUG] Longhorn 1.12.0: AWS chunked encoding not supported with OCI S3 buckets 13478 - @mantissahz
- [BACKPORT][v1.12.1][BUG] Failed to add v2 block disk with virtio-scsi BDF path 13475 - @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] V2 backup/snapshot can leave NVMe/TCP frontend or dm device stale, causing pod EIO on attached volumes 13332 - @davidcheng0922 @chriscchien
- [BACKPORT][v1.12.1][BUG] (chart) ArgoCD OutOfSync when using Gateway API 13446 - @yangchiu
- [BACKPORT][v1.12.1][BUG] Migration Engine Can Be Unexpectedly Deleted If the Target Node Is Still in Readiness Transition 13367 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.12.1][BUG] Recurring trim job fails with deadlock 13425 - @c3y1huang @roger-ryao
- [BACKPORT][v1.12.1][BUG] volume expansion stuck 13368 - @shuo-wu @chriscchien
- [BACKPORT][v1.12.1][BUG] pvc resize fails after iscsid restart 13412 - @yangchiu @shuo-wu
- [BACKPORT][v1.12.1][BUG] expanding the volume fails 13384 - @chriscchien
- [BACKPORT][v1.12.1][BUG] Test case
test_rwx_delete_share_manager_podfails because it's unable to find the exported volume in share manager pod after it's deleted and restarted 13226 - @davidcheng0922 @roger-ryao - [BACKPORT][v1.12.1][BUG] System Backup RecurringJob retention prunes newest CR — sorts by Status.CreatedAt (zero for Error/racing CRs) 13209 - @roger-ryao
- [BACKPORT][v1.12.1][BUG] CSI components may have 0 running replica during upgrade 13348 - @yangchiu @carterli0407-cell
- [BACKPORT][v1.12.1][BUG] Node update forces a complete rebuild 13357 - @mantissahz
- [BACKPORT][v1.12.1][BUG] Creating backup for a v2 volume may fail 13191 - @mantissahz
- [BACKPORT][v1.12.1][BUG] when uploading backup to S3 storage (NetApp appliance) it fails 13297 - @mantissahz
- [BACKPORT][v1.12.1][BUG] spdk interrupt mode value is missing in chart/values.yaml 13269 - @yangchiu
- [BACKPORT][v1.12.1][BUG] Test case
test_best_effort_data_localityfails because there is no replica for the created volume 13225 - @carterli0407-cell
Misc
- [BACKPORT][v1.12.1][BUG] v2 volume stuck
attachingwith Storage Network enabled because theEngineFrontendtarget uses the engine pod IP instead ofStorageIP13353 - @yangchiu @c3y1huang
Contributors
- @COLDTURNIP
- @Copilot
- @bachmanity1
- @c3y1huang
- @carterli0407-cell
- @chriscchien
- @davidcheng0922
- @derekbit
- @github-actions[bot]
- @hookak
- @innobead
- @mantissahz
- @roger-ryao
- @shuo-wu
- @sushant-suse
- @yangchiu
Longhorn v1.11.3
Longhorn v1.11.3 Release Notes
Longhorn 1.11.3 introduces several improvements and bug fixes that are intended to improve system quality, resilience, stability and security.
We welcome feedback and contributions to help continuously improve Longhorn.
For terminology and context on Longhorn releases, see Releases.
Important Fixes
This release includes several critical stability fixes.
V1 volumes not operable fix after iscsid restart
Resolved the issue caused by the iscsid restart where volume operations could become stuck or fail, including PVC resize failures.
For more details, see #13411, #13413, and #13383.
Replica rebuild stability fix
Fixed a nil pointer dereference panic in longhorn-instance-manager during replica rebuild, improving rebuild stability under failure conditions.
For more details, see #13129.
Migration engine readiness transition fix
Resolved an issue where the migration engine could be unexpectedly deleted while the target node was still transitioning to ready, which could interrupt migration workflows.
For more details, see #13133.
Recurring trim job deadlock fix
Resolved a deadlock that could cause recurring trim jobs to fail.
For more details, see #13424.
Installation
Important
Due to the upgrade of the CSI external provisioner to v6.3.0, ensure that your cluster is running Kubernetes v1.34 or later before installing Longhorn v1.11.3.
You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.
Upgrade
Important
Due to the upgrade of the CSI external provisioner to v6.3.0, ensure that your cluster is running Kubernetes v1.34 or later before upgrading from Longhorn v1.10.x or v1.11.0 to v1.11.3.
Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.
Post-Release Known Issues
For information about issues identified after this release, see Release-Known-Issues.
Resolved Issues in this release
Improvement
- [BACKPORT][v1.11.3][IMPROVEMENT] longhorn-manager pods race on webhook TLS Secret at scale 13116 - @yangchiu @hookak
- [BACKPORT][v1.11.3][IMPROVEMENT] Add metrics to collect information about LONGHORN_DISTRO 13283 - @derekbit @chriscchien
Bug
- [BACKPORT][v1.11.3][BUG] volume expansion stuck 13411 - @shuo-wu @roger-ryao
- [BACKPORT][v1.11.3][BUG] pvc resize fails after iscsid restart 13413 - @yangchiu @shuo-wu
- [BACKPORT][v1.11.3][BUG] expanding the volume fails 13383 - @chriscchien
- [BACKPORT][v1.11.3][BUG] Migration Engine Can Be Unexpectedly Deleted If the Target Node Is Still in Readiness Transition 13133 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.11.3][BUG] System Backup RecurringJob retention prunes newest CR — sorts by Status.CreatedAt (zero for Error/racing CRs) 13211 - @roger-ryao
- [BACKPORT][v1.11.3][BUG] when uploading backup to S3 storage (NetApp appliance) it fails 13296 - @mantissahz
- [BACKPORT][v1.11.3][BUG] spdk interrupt mode value is missing in chart/values.yaml 13270 - @yangchiu
- [BACKPORT][v1.11.3][BUG] nil pointer dereference panic in instance-manager during replica rebuild. 13129 - @derekbit @shuo-wu @roger-ryao
- [BACKPORT][v1.11.3][BUG] Test Encrypted Volume Upgrade: Old-engine RWO volume shows 1008 MiB after expansion to 2 GiB instead of expected 2032 MiB 13200 - @derekbit @mantissahz @roger-ryao
- [BACKPORT][v1.11.3][BUG] HTTP response body leaks in support bundle status polling and webhook readiness checks 13124 - @derekbit @roger-ryao
- [BACKPORT][v1.11.3][BUG] global.cattle.systemDefaultRegistry is not applied as the image registry prefix in 108.2.1+up1.10.2 13099 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.11.3][BUG] [longhorn-engine/dataserver] Handling EOF returned by io.ReadFull robustly 13080 - @yangchiu
- [BACKPORT][v1.11.3][BUG] PrometheusTimeseriesCardinality for metric longhorn_rest_client_rate_limiter_latency_seconds_bucket 13088 - @derekbit
- [BACKPORT][v1.11.3][BUG] Recurring trim job fails with deadlock 13424 - @c3y1huang @roger-ryao
Stability
- [BACKPORT][v1.11.3][BUG] longhorn-manager panic in BackupController.setInprogressDeletionMap during backup deletion 13247 - @roger-ryao
Misc
- [BACKPORT][v1.11.3][TASK] Add distro information to upgrade responder requests 13278 - @davidcheng0922 @roger-ryao
Contributors
Longhorn v1.11.3-rc2
DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.
Resolved Issues in this release
Improvement
- [BACKPORT][v1.11.3][IMPROVEMENT] longhorn-manager pods race on webhook TLS Secret at scale 13116 - @yangchiu @hookak
- [BACKPORT][v1.11.3][IMPROVEMENT] Add metrics to collect information about LONGHORN_DISTRO 13283 - @derekbit @chriscchien
Bug
- [BACKPORT][v1.11.3][BUG] volume expansion stuck 13411 - @shuo-wu @roger-ryao
- [BACKPORT][v1.11.3][BUG] pvc resize fails after iscsid restart 13413 - @yangchiu @shuo-wu
- [BACKPORT][v1.11.3][BUG] expanding the volume fails 13383 - @chriscchien
- [BACKPORT][v1.11.3][BUG] Migration Engine Can Be Unexpectedly Deleted If the Target Node Is Still in Readiness Transition 13133 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.11.3][BUG] System Backup RecurringJob retention prunes newest CR — sorts by Status.CreatedAt (zero for Error/racing CRs) 13211 - @roger-ryao
- [BACKPORT][v1.11.3][BUG] when uploading backup to S3 storage (NetApp appliance) it fails 13296 - @mantissahz
- [BACKPORT][v1.11.3][BUG] spdk interrupt mode value is missing in chart/values.yaml 13270 - @yangchiu
- [BACKPORT][v1.11.3][BUG] nil pointer dereference panic in instance-manager during replica rebuild. 13129 - @derekbit @shuo-wu @roger-ryao
- [BACKPORT][v1.11.3][BUG] Test Encrypted Volume Upgrade: Old-engine RWO volume shows 1008 MiB after expansion to 2 GiB instead of expected 2032 MiB 13200 - @derekbit @mantissahz @roger-ryao
- [BACKPORT][v1.11.3][BUG] HTTP response body leaks in support bundle status polling and webhook readiness checks 13124 - @derekbit @roger-ryao
- [BACKPORT][v1.11.3][BUG] global.cattle.systemDefaultRegistry is not applied as the image registry prefix in 108.2.1+up1.10.2 13099 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.11.3][BUG] [longhorn-engine/dataserver] Handling EOF returned by io.ReadFull robustly 13080 - @yangchiu
- [BACKPORT][v1.11.3][BUG] PrometheusTimeseriesCardinality for metric longhorn_rest_client_rate_limiter_latency_seconds_bucket 13088 - @derekbit
Stability
- [BACKPORT][v1.11.3][BUG] longhorn-manager panic in BackupController.setInprogressDeletionMap during backup deletion 13247 - @roger-ryao
Misc
- [BACKPORT][v1.11.3][TASK] Add distro information to upgrade responder requests 13278 - @davidcheng0922 @roger-ryao
Contributors
- @COLDTURNIP
- @chriscchien
- @davidcheng0922
- @derekbit
- @github-actions[bot]
- @hookak
- @innobead
- @mantissahz
- @roger-ryao
- @shuo-wu
- @yangchiu
Longhorn v1.11.3-rc1
DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.
Resolved Issues in this release
Improvement
- [BACKPORT][v1.11.3][IMPROVEMENT] longhorn-manager pods race on webhook TLS Secret at scale 13116 - @yangchiu @hookak
- [BACKPORT][v1.11.3][IMPROVEMENT] Add metrics to collect information about LONGHORN_DISTRO 13283 - @derekbit @chriscchien
Bug
- [BACKPORT][v1.11.3][BUG] expanding the volume fails 13383 - @chriscchien
- [BACKPORT][v1.11.3][BUG] Migration Engine Can Be Unexpectedly Deleted If the Target Node Is Still in Readiness Transition 13133 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.11.3][BUG] System Backup RecurringJob retention prunes newest CR — sorts by Status.CreatedAt (zero for Error/racing CRs) 13211 - @roger-ryao
- [BACKPORT][v1.11.3][BUG] when uploading backup to S3 storage (NetApp appliance) it fails 13296 - @mantissahz
- [BACKPORT][v1.11.3][BUG] spdk interrupt mode value is missing in chart/values.yaml 13270 - @yangchiu
- [BACKPORT][v1.11.3][BUG] nil pointer dereference panic in instance-manager during replica rebuild. 13129 - @derekbit @shuo-wu @roger-ryao
- [BACKPORT][v1.11.3][BUG] Test Encrypted Volume Upgrade: Old-engine RWO volume shows 1008 MiB after expansion to 2 GiB instead of expected 2032 MiB 13200 - @derekbit @mantissahz @roger-ryao
- [BACKPORT][v1.11.3][BUG] HTTP response body leaks in support bundle status polling and webhook readiness checks 13124 - @derekbit @roger-ryao
- [BACKPORT][v1.11.3][BUG] global.cattle.systemDefaultRegistry is not applied as the image registry prefix in 108.2.1+up1.10.2 13099 - @COLDTURNIP @yangchiu
- [BACKPORT][v1.11.3][BUG] [longhorn-engine/dataserver] Handling EOF returned by io.ReadFull robustly 13080 - @yangchiu
- [BACKPORT][v1.11.3][BUG] PrometheusTimeseriesCardinality for metric longhorn_rest_client_rate_limiter_latency_seconds_bucket 13088 - @derekbit
Stability
- [BACKPORT][v1.11.3][BUG] longhorn-manager panic in BackupController.setInprogressDeletionMap during backup deletion 13247 - @roger-ryao
Misc
- [BACKPORT][v1.11.3][TASK] Add distro information to upgrade responder requests 13278 - @davidcheng0922 @roger-ryao
Contributors
- @COLDTURNIP
- @chriscchien
- @davidcheng0922
- @derekbit
- @github-actions[bot]
- @hookak
- @innobead
- @mantissahz
- @roger-ryao
- @shuo-wu
- @yangchiu
Longhorn v1.12.0
Longhorn v1.12.0 Release Notes
The Longhorn team is excited to announce the release of Longhorn v1.12.0. This feature release marks a major milestone for Longhorn: the V2 Data Engine is now officially Generally Available (GA).
With the V2 Data Engine reaching GA, Longhorn v1.12.0 strengthens the production story for modern workloads with topology-aware provisioning, dual-stack and V2 IPv6 support, improved observability and operational tooling, and clearer guidance around V1 and V2 behavior and feature parity.
For terminology and background on Longhorn releases, see Releases.
Removal
V2 Backing Image Removal
V2 Backing Images are removed in Longhorn v1.12.0. Suggest using the Containerized Data Importer (CDI) to import VM disk images into V2 volumes to achieve the same purpose.
If you have V2 volumes that were created from backing images, you must migrate them before upgrading to v1.12.0:
- Backup and recreate (recommended): Create a backup of the V2 volume, delete the original volume, then restore from backup. The restored volume will not have a backing image dependency.
- Delete the volume: If the data is not needed, delete the V2 volume directly.
V2 volumes with backing image dependencies cannot be upgraded in-place. Attempting to upgrade without migration may result in volume attachment failures.
Primary Highlights
V2 Data Engine
Generally Available
We are pleased to announce that the V2 Data Engine has officially graduated to General Availability in Longhorn v1.12.0.
This milestone reflects major progress in stability, operational safety, networking support, and feature maturity. Compared with earlier releases, V2 volumes are better positioned for production use, combining GA readiness with modern networking support, more precise scheduling behavior, and clearer visibility into where V2 already matches V1 behavior and where differences still matter.
Important
V2 Live Upgrade:
V2 volumes do not support live upgrades between Longhorn v1.12 patch releases and must be detached before upgrading. Support is planned when upgrading from a Longhorn v1.12 release to a Longhorn v1.13 release.
V2 Volume Attach Latency at Scale:
In environments with a growing number of attached V2 volumes, increased attach latency has been observed for subsequent volumes. Initial analysis suggests this may be related to NVMe-TCP connection handling at scale, though the precise layer, SPDK user-space or Linux kernel, has not yet been identified. Further investigation is in progress. For follow-up status, see Issue #13241.
ARM64 NVMe-backed Block-Type Node Disk Limitation:
On ARM64 systems, V2 volumes may experience stuck I/O when SPDK is configured with two or more CPU cores and node disks use the NVMe driver. The root cause may lie in either the Linux kernel or SPDK itself, and further investigation is required. As a workaround, use AIO-backed node disks instead of NVMe-backed node disks on ARM64 systems. For follow-up status, see Issue #13243.
For a summary of the current V1 and V2 volume behavior differences and feature parity, see V1 and V2 Volume Feature Support.
Looking ahead, the roadmap remains active: fast volume cloning for V2 data engine (#12552) and Sharding Storage (Experimental Feature) (#1061) are planned for Longhorn v1.12.1.
Smarter Provisioning and Modern Networking
Topology-Aware PV Node Affinity Control
Longhorn v1.12.0 adds the csi-allowed-topology-keys setting and strictTopology StorageClass parameter for more precise control of PV nodeAffinity. These options allow users to limit which topology keys are propagated and, with WaitForFirstConsumer, pin the PV to the selected node topology when needed.
IPv6 Support for V2 Volumes
V2 volumes now support single-stack IPv6 Kubernetes clusters.
Dual-Stack Cluster Support
Longhorn now supports dual-stack Kubernetes clusters when all nodes are configured with their IP families in the same order, either all IPv4-first or all IPv6-first. This applies to both the V1 and V2 data engines.
Warning: Dual-stack clusters with mixed IP family ordering across nodes are not supported and may result in connectivity failures between replicas and the engine.
Better Operations and Observability
Default CPU Allocation
Longhorn v1.12.0 changes the default data-engine-cpu-mask from 0x1, one CPU core, to 0x3, two CPU cores. V2 Data Engine uses a busy-polling reactor model where the master reactor handles both I/O polling and management RPCs. When only a single core is assigned, heavy I/O workloads can delay or starve RPC processing, resulting in increased latency, timeout events, and operational instability.
Assigning two or more cores allows I/O and management tasks to run on separate reactors, improving responsiveness and operational stability.
On-Demand Snapshot Checksum Calculation
Longhorn v1.12.0 adds longhornctl support for triggering on-demand snapshot checksum calculation. The command can target a specific volume, all volumes on a specific node, or all volumes in the cluster, and the checksum operation runs asynchronously in the background.
Toggle Kubernetes Metrics Server Integration
Longhorn v1.12.0 adds the Kubernetes Metrics Server Metrics Enabled setting to disable metrics-server-dependent metrics when the Kubernetes Metrics Server API is unavailable. This reduces repeated scrape warnings and unnecessary API calls while preserving other Longhorn metrics.
Longhorn Manager Memory Optimization
Longhorn v1.12.0 optimizes longhorn-manager informer caching to reduce memory usage, especially in large clusters with high pod counts. This lowers cluster-wide memory overhead caused by repeated caching of non-Longhorn pod data on every manager instance.
Configurable Engine Image Pod Liveness Probe
Longhorn v1.12.0 adds settings to configure the engine-image DaemonSet liveness probe period, timeout, and failure threshold. These settings help reduce unnecessary engine-image pod restarts on resource-constrained clusters, especially during upgrades or transient CPU spikes.
Critical Stability Fixes
Instance Manager Stability During Replica Rebuild Storms
Longhorn v1.12.0 fixes an instance-manager panic that could occur during replica rebuild storms. In affected environments, the panic could terminate all iSCSI targets served by the instance-manager and trigger cascading volume detachments across multiple PVCs.
Replica Rebuild Progress Reporting
Longhorn v1.12.0 fixes a replica rebuild progress reporting bug that could display values greater than 100% after file-sync retries on unstable networks. Progress accounting is now reset correctly for retried files, so rebuild progress remains within the valid 0% to 100% range.
Replica Auto-Balance Scheduling Loop
Longhorn v1.12.0 fixes a regression in replica auto-balance that could trigger a repeated replica create-and-delete loop when Replica Auto Balance was set to best-effort. In affected clusters, Longhorn could keep scheduling an extra replica instead of stabilizing at the configured replica count.
Replica CR Leak During Failed Local Scheduling
Longhorn v1.12.0 fixes a replica scheduling issue where large numbers of stopped Replica CRs could accumulate when dataLocality was set to best-effort and the node did not have enough eligible local disk space for another replica. In affected clusters, recurring reconciliation could keep creating placeholder Replica CRs instead of reusing a single failed-schedule placeholder.
CSI Storage Capacity Tracking
Longhorn v1.12.0 fixes a CSIStorageCapacity scheduling issue that could cause compute nodes without Longhorn disks to report zero capacity and be rejected by WaitForFirstConsumer scheduling. In affected clusters with separated compute and storage nodes, new PVCs could remain pending even though eligible storage was available on storage nodes.
Encrypted Volume Size Correction
Longhorn v1.12.0 pre-allocates the 16 MiB LUKS2 header in the replica backend file for encrypted volumes, so the dm-...
Longhorn v1.12.0-rc4
DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.
Resolved Issues in this release
Highlight
- [FEATURE] Decouple V2 Data Engine Initiator and Target Placement 7124 - @derekbit @shuo-wu @chriscchien
- [FEATURE] IPv6 for V2 Data Engine 10928 - @COLDTURNIP @chriscchien
- [FEATURE] Support IPv4/IPv6 Dual-Stack with IPv6 Family First or IPv4 Family First 11531 - @COLDTURNIP @c3y1huang @chriscchien
- [FEATURE] Support v2 Data Engine (GA) 6229 - @derekbit
Feature
- [FEATURE] Support on-demand snapshot checksum calculation 11442 - @yangchiu @davidcheng0922
- [FEATURE] Add
--tolerationsflag tolonghornctlfor scheduling DaemonSet pods on tainted nodes 12993 - @chriscchien @bachmanity1
Improvement
- [IMPROVEMENT] Remove v2 backing image monitoring 13181 - @COLDTURNIP @derekbit @chriscchien
- [IMPROVEMENT] Wait for spdk_tgt process to terminate during pre-stop cleanup 13179 - @derekbit @chriscchien
- [IMPROVEMENT] Restart Instance Manager pod when hugepage settings change and no instances are running 13170 - @derekbit @chriscchien
- [IMPROVEMENT] Support CPU list format for V2 Data Engine CPU Mask setting with automatic conversion to hex mask 13166 - @derekbit @chriscchien
- [IMPROVEMENT] Update Longhorn
distroin chart tolonghorn13160 - @derekbit @chriscchien - [IMPROVEMENT] Misleading storage values 12633 - @elTwingo @davidcheng0922 @houhoucoop @roger-ryao
- [IMPROVEMENT] Implement Network Reconnection for Enhancing Replica Rebuilding Resilience 9626 - @yangchiu @mschneider82
- [IMPROVEMENT] Add support of new StorageClass parameters to helm chart 9324 - @yangchiu @TheFutonEng
- [IMPROVEMENT] Make Kubernetes Metrics Server (metrics.k8s.io) integration toggleable 13011 - @yangchiu @mantissahz @hookak
- [IMPROVEMENT] Reduce longhorn-manager memory usage by optimizing cluster-wide informer caching 12771 - @hookak @roger-ryao
- [IMPROVEMENT] Topology-aware PV nodeAffinity control: allowedTopologies keys + strictTopology 12684 - @hookak @roger-ryao
- [IMPROVEMENT] Set storage class annotations using helm values 13137 - @yangchiu @Profiidev
- [IMPROVEMENT] longhorn-manager pods race on webhook TLS Secret at scale 13012 - @yangchiu @hookak
- [IMPROVEMENT] Improve Longhorn auto-salvage observability 13018 - @yangchiu @derekbit
- [IMPROVEMENT] Removing Scheduled condition check during volume expansion 12606 - @yangchiu @davidcheng0922
- [IMPROVEMENT]
TooManySnapshotsvolume condition uses a hard-coded threshold despite configurable snapshot max count 12396 - @COLDTURNIP @yangchiu - [IMPROVEMENT] Move v2 volume backup restore from replica to engine 9277 - @davidcheng0922 @roger-ryao
- [IMPROVEMENT] Is there any way to have longhorn without python 12679 - @roger-ryao
- [IMPROVEMENT] sparse-tools APIs must not introduce breaking changes to existing APIs. 12967 - @yangchiu @derekbit
- [UI][IMPROVEMENT]
TooManySnapshotsvolume condition uses a hard-coded threshold despite configurable snapshot max count 12922 - @chriscchien @houhoucoop - [IMPROVEMENT] Add
Backup Targetto volume listcustom columnoptions 12619 - @yangchiu @houhoucoop - [IMPROVEMENT] Allow disabling creation of the default longhorn StorageClass via Helm 12906 - @hookak @roger-ryao
- [IMPROVEMENT][TEST] Add unit tests for util parsing and string conversion helpers 12898 - @archy-rock3t-cloud @chriscchien
- [IMPROVEMENT] Metrics for backups 11387 - @yangchiu @mantissahz @Copilot
- [IMPROVEMENT] chart: allow specifying spec.sampleLimit on ServiceMonitor 12671 - @grelland @yangchiu
- [IMPROVEMENT] Add metrics for non-Encrypted and encrypted volumes 12462 - @derekbit @mantissahz @chriscchien @Copilot
- [IMPROVEMENT] Clarify helm version in generate-longhorn-yaml error message 12630 - @luojiyin1987 @chriscchien
- [IMPROVEMENT][UI] Link version number to git releases 11132 - @chriscchien @houhoucoop
- [IMPROVEMENT] Record the current share manager image in the Share Manager CR status 11203 - @derekbit @roger-ryao @Copilot
- [IMPROVEMENT] Snapshot tree color explanation 12247 - @houhoucoop
- [IMPROVEMENT] Refuse to attach strict-local volume to the wrong node 8546 - @yangchiu @derekbit @mantissahz @Copilot
- [IMPROVEMENT] Ensure V2 Engine ReplicaAdd respects the fast-replica-rebuild-enabled setting 12540 - @davidcheng0922 @roger-ryao
- [IMPROVEMENT] Relax
endpoint-network-for-rwx-volumevalidation for migratable block-mode volumes 12644 - @c3y1huang @chriscchien - [IMPROVEMENT] detailed log for the reason of node controller deleting backing image copies 12584 - @COLDTURNIP @yangchiu
- [IMPROVEMENT] RBAC permissions for csi-resizer 12681 - @yangchiu @konstantin-kelemen
- [IMPROVEMENT] Adding a message to hint users to clean up non-existing disks in Backing Image CR 10617 - @chriscchien @Copilot
- [IMPROVEMENT] Keep workload pod in the original zone and region 12517 - @bachmanity1
- [IMPROVEMENT] Consider node storage capacity when scheduling pods with existing PVs 12398 - @bachmanity1
- [IMPROVEMENT] Volume may enter faulty state without clear reason when backing image size mismatches 11673 - @COLDTURNIP @derekbit @roger-ryao @Copilot
Bug
- [BUG] nil pointer dereference panic in instance-manager during replica rebuild. 13087 - @derekbit @shuo-wu @roger-ryao
- [BUG] v2 RWX workload IO timed out after Longhorn components are deleted and restarted 13217 - @yangchiu @derekbit
- [BUG] v2 volume gets stuck in
Degradedstate after instance manager is deleted and restarted 13215 - @derekbit - [BUG] Test Encrypted Volume Upgrade: Old-engine RWO volume shows 1008 MiB after expansion to 2 GiB instead of expected 2032 MiB 13194 - @derekbit @mantissahz @roger-ryao
- [BUG] v2 volume deletion clears
Spec.NodeIDbefore delete, potentially orphaning replicas whenStatus.InstanceManagerNameis empty 13198 - @derekbit @chriscchien - [BUG] Backup target still shows
Availableafter being reset to empty 13195 - @yangchiu @derekbit - [BUG] v2 instance-manager pod stuck in create/delete loop when engine frontend recovery blocks gRPC startup 13185 - @derekbit @chriscchien
- [BUG] global.cattle.systemDefaultRegistry is not applied as the image registry prefix in 108.2.1+up1.10.2 13071 - @COLDTURNIP @yangchiu
- [BUG] Potential resource leak in longhorn-instance-manager 13143 - @derekbit @chriscchien
- [BUG] Encrypt volume provided size is 16MB shorter than the claimed size 9205 - @mantissahz @roger-ryao
- [BUG] CSIStorageCapacity reports 0 for compute nodes without Longhorn disks, breaking WaitForFirstConsumer scheduling 12807 - @bachmanity1 @roger-ryao
- [BUG] Google Cloud Storage (GCS) backup target always fails with SignatureDoesNotMatch due to AWS SDK Go v2 CRC32 checksum incompatibility 12676 - @mantissahz @chriscchien
- [BUG] Longhorn Fails to enable volume security on FIPS enabled systems 12721 - @davidcheng0922 @c...
Longhorn v1.12.0-rc3
DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.
Resolved Issues in this release
Highlight
- [FEATURE] Decouple V2 Data Engine Initiator and Target Placement 7124 - @derekbit @shuo-wu @chriscchien
- [FEATURE] IPv6 for V2 Data Engine 10928 - @COLDTURNIP @chriscchien
- [FEATURE] Support IPv4/IPv6 Dual-Stack with IPv6 Family First or IPv4 Family First 11531 - @COLDTURNIP @c3y1huang @chriscchien
- [FEATURE] Support v2 Data Engine (GA) 6229 - @derekbit
Feature
- [FEATURE] Support on-demand snapshot checksum calculation 11442 - @yangchiu @davidcheng0922
- [FEATURE] Add
--tolerationsflag tolonghornctlfor scheduling DaemonSet pods on tainted nodes 12993 - @chriscchien @bachmanity1
Improvement
- [IMPROVEMENT] Remove v2 backing image monitoring 13181 - @COLDTURNIP @derekbit @chriscchien
- [IMPROVEMENT] Wait for spdk_tgt process to terminate during pre-stop cleanup 13179 - @derekbit @chriscchien
- [IMPROVEMENT] Restart Instance Manager pod when hugepage settings change and no instances are running 13170 - @derekbit @chriscchien
- [IMPROVEMENT] Support CPU list format for V2 Data Engine CPU Mask setting with automatic conversion to hex mask 13166 - @derekbit @chriscchien
- [IMPROVEMENT] Update Longhorn
distroin chart tolonghorn13160 - @derekbit @chriscchien - [IMPROVEMENT] Misleading storage values 12633 - @elTwingo @davidcheng0922 @houhoucoop @roger-ryao
- [IMPROVEMENT] Implement Network Reconnection for Enhancing Replica Rebuilding Resilience 9626 - @yangchiu @mschneider82
- [IMPROVEMENT] Add support of new StorageClass parameters to helm chart 9324 - @yangchiu @TheFutonEng
- [IMPROVEMENT] Make Kubernetes Metrics Server (metrics.k8s.io) integration toggleable 13011 - @yangchiu @mantissahz @hookak
- [IMPROVEMENT] Reduce longhorn-manager memory usage by optimizing cluster-wide informer caching 12771 - @hookak @roger-ryao
- [IMPROVEMENT] Topology-aware PV nodeAffinity control: allowedTopologies keys + strictTopology 12684 - @hookak @roger-ryao
- [IMPROVEMENT] Set storage class annotations using helm values 13137 - @yangchiu @Profiidev
- [IMPROVEMENT] longhorn-manager pods race on webhook TLS Secret at scale 13012 - @yangchiu @hookak
- [IMPROVEMENT] Improve Longhorn auto-salvage observability 13018 - @yangchiu @derekbit
- [IMPROVEMENT] Removing Scheduled condition check during volume expansion 12606 - @yangchiu @davidcheng0922
- [IMPROVEMENT]
TooManySnapshotsvolume condition uses a hard-coded threshold despite configurable snapshot max count 12396 - @COLDTURNIP @yangchiu - [IMPROVEMENT] Move v2 volume backup restore from replica to engine 9277 - @davidcheng0922 @roger-ryao
- [IMPROVEMENT] Is there any way to have longhorn without python 12679 - @roger-ryao
- [IMPROVEMENT] sparse-tools APIs must not introduce breaking changes to existing APIs. 12967 - @yangchiu @derekbit
- [UI][IMPROVEMENT]
TooManySnapshotsvolume condition uses a hard-coded threshold despite configurable snapshot max count 12922 - @chriscchien @houhoucoop - [IMPROVEMENT] Add
Backup Targetto volume listcustom columnoptions 12619 - @yangchiu @houhoucoop - [IMPROVEMENT] Allow disabling creation of the default longhorn StorageClass via Helm 12906 - @hookak @roger-ryao
- [IMPROVEMENT][TEST] Add unit tests for util parsing and string conversion helpers 12898 - @archy-rock3t-cloud @chriscchien
- [IMPROVEMENT] Metrics for backups 11387 - @yangchiu @mantissahz @Copilot
- [IMPROVEMENT] chart: allow specifying spec.sampleLimit on ServiceMonitor 12671 - @grelland @yangchiu
- [IMPROVEMENT] Add metrics for non-Encrypted and encrypted volumes 12462 - @derekbit @mantissahz @chriscchien @Copilot
- [IMPROVEMENT] Clarify helm version in generate-longhorn-yaml error message 12630 - @luojiyin1987 @chriscchien
- [IMPROVEMENT][UI] Link version number to git releases 11132 - @chriscchien @houhoucoop
- [IMPROVEMENT] Record the current share manager image in the Share Manager CR status 11203 - @derekbit @roger-ryao @Copilot
- [IMPROVEMENT] Snapshot tree color explanation 12247 - @houhoucoop
- [IMPROVEMENT] Refuse to attach strict-local volume to the wrong node 8546 - @yangchiu @derekbit @mantissahz @Copilot
- [IMPROVEMENT] Ensure V2 Engine ReplicaAdd respects the fast-replica-rebuild-enabled setting 12540 - @davidcheng0922 @roger-ryao
- [IMPROVEMENT] Relax
endpoint-network-for-rwx-volumevalidation for migratable block-mode volumes 12644 - @c3y1huang @chriscchien - [IMPROVEMENT] detailed log for the reason of node controller deleting backing image copies 12584 - @COLDTURNIP @yangchiu
- [IMPROVEMENT] RBAC permissions for csi-resizer 12681 - @yangchiu @konstantin-kelemen
- [IMPROVEMENT] Adding a message to hint users to clean up non-existing disks in Backing Image CR 10617 - @chriscchien @Copilot
- [IMPROVEMENT] Keep workload pod in the original zone and region 12517 - @bachmanity1
- [IMPROVEMENT] Consider node storage capacity when scheduling pods with existing PVs 12398 - @bachmanity1
- [IMPROVEMENT] Volume may enter faulty state without clear reason when backing image size mismatches 11673 - @COLDTURNIP @derekbit @roger-ryao @Copilot
Bug
- [BUG] Potential resource leak in longhorn-instance-manager 13143 - @derekbit @chriscchien
- [BUG] nil pointer dereference panic in instance-manager during replica rebuild. 13087 - @derekbit @shuo-wu @roger-ryao
- [BUG] Encrypt volume provided size is 16MB shorter than the claimed size 9205 - @mantissahz @roger-ryao
- [BUG] CSIStorageCapacity reports 0 for compute nodes without Longhorn disks, breaking WaitForFirstConsumer scheduling 12807 - @bachmanity1 @roger-ryao
- [BUG] Google Cloud Storage (GCS) backup target always fails with SignatureDoesNotMatch due to AWS SDK Go v2 CRC32 checksum incompatibility 12676 - @mantissahz @chriscchien
- [BUG] Longhorn Fails to enable volume security on FIPS enabled systems 12721 - @davidcheng0922 @chriscchien
- [BUG] Replica Auto-Balance Causes Infinite Replica Scheduling Loop 12926 - @yangchiu @shuo-wu
- [BUG] Replica rebuild progress can go over 100% 12949 - @yangchiu @mschneider82 @davidcheng0922
- [BUG] v2 backup/restore open failure paths can leak NVMe initiators and exposed bdevs 13114 - @derekbit @roger-ryao
- [BUG] Connection leak in longhorn-spdk-engine 13101 - @derekbit @roger-ryao @Copilot
- [BUG] HTTP response body leaks in support bundle status polling and webhook readiness checks 13115 - @derekbit @roger-ryao
- [BUG] Encrypted volume stuck in Attaching/Detaching loop after node reboot and instance manager deletion 11510 - @yangchiu @mantissahz
- [BUG] Test case
test_cleanup_system_generated_snapshotsfails on v2 volumes 13123 - @yangchiu @davidcheng0922 - [BUG] Test case
test_drain_with_block_for_eviction_if_contains_last_replica_successfailed on v1 volumes [13103](#13103...