[LTS 9.4] nvme-tcp: fix potential memory corruption in nvme_tcp_recv_pdu()

[pvts-mat]

[LTS 9.4]
CVE-2025-21927
VULN-56029

Problem

https://www.cve.org/CVERecord?id=CVE-2025-21927

In the Linux kernel, the following vulnerability has been resolved: nvme-tcp: fix potential memory corruption in nvme_tcp_recv_pdu() nvme_tcp_recv_pdu() doesn't check the validity of the header length. When header digests are enabled, a target might send a packet with an invalid header length (e.g. 255), causing nvme_tcp_verify_hdgst() to access memory outside the allocated area and cause memory corruptions by overwriting it with the calculated digest. Fix this by rejecting packets with an unexpected header length.

Analysis and Solution (same as for ciqlts8_8)

Context

NVME (Non-Volatile Memory Express) is a communication protocol designed for accessing high-speed storage media, particularly solid-state drives (SSDs). NVMe over Fabrics (NVMe-oF) is an extension of the NVMe protocol that allows NVMe commands to be sent over a network fabric, enabling remote access to NVMe storage devices.

The "target" mentioned in CVE description is the host providing access to the local NVME device (the server). The host importing the remote NVME device is called simply a "host", or "initiator" (the client). The module implementing NVMe-oF on target's side is nvmet-tcp, on the initiator's side it's nvme-tcp - the subject of this patch.

Applicability

All the key options related to NVMe-oF, specifically CONFIG_NVME_TCP enabling the nvme-tcp module, are enabled in ciqlts9_4. Per .config file created from configs/kernel-x86_64-rhel.config:

#
# NVME Support
#
CONFIG_NVME_COMMON=m
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_VERBOSE_ERRORS=y
# CONFIG_NVME_HWMON is not set
CONFIG_NVME_FABRICS=m
CONFIG_NVME_RDMA=m
CONFIG_NVME_FC=m
CONFIG_NVME_TCP=m
CONFIG_NVME_AUTH=y
CONFIG_NVME_TARGET=m
# CONFIG_NVME_TARGET_PASSTHRU is not set
CONFIG_NVME_TARGET_LOOP=m
CONFIG_NVME_TARGET_RDMA=m
CONFIG_NVME_TARGET_FC=m
CONFIG_NVME_TARGET_FCLOOP=m
CONFIG_NVME_TARGET_TCP=m
CONFIG_NVME_TARGET_AUTH=y
# end of NVME Support

Solution

The solution in the mainline kernel is provided in the ad95bab commit. It was not backported to any stable kernel older than 6.12.

Naive cherry-picking results in conflicts with git's attempt to introduce additional functions (nvme_tcp_tls_configured, nvme_tcp_queue_tls) and code branches (nvme_tcp_c2h_term packet type check in the nvme_tcp_recv_pdu function) introduced in more recent versions of the module but not related to the bug fix.

A small change was made to the nvme_tcp_recv_pdu_supported function introduced in the official fix ad95bab for the sake of nvme_tcp_recv_pdu's behavior consistency between the scenarios of receiving a packet with a proper and an improper header - the removal of the nvme_tcp_c2h_term case.

Consider the behavior cases in case a packet with a proper header was received:

	Packet type:	X ∈ {c2h_term}	X ∈ {c2h_data, rsp, r2t}	X ∉ {c2h_term, c2h_data, rsp, r2t}
a	Mainline, after patch (ad95bab)	nvme_tcp_handle_X	nvme_tcp_handle_X	"unsupported pdu type …", -EINVAL
b	ciqlts9_4, after patch, c2h_term included	"unsupported pdu type …", -EINVAL	nvme_tcp_handle_X	"unsupported pdu type …", -EINVAL
c	ciqlts9_4, after patch, c2h_term excluded	"unsupported pdu type …", -EINVAL	nvme_tcp_handle_X	"unsupported pdu type …", -EINVAL

Then in case a packet with an improper header was received:

	Packet type:	X ∈ {c2h_term}	X ∈ {c2h_data, rsp, r2t}	X ∉ {c2h_term, c2h_data, rsp, r2t}
x	Mainline, after patch (ad95bab)	"pdu type %d has unexpected header length", -EPROTO	"pdu type %d has unexpected header length", -EPROTO	"unsupported pdu type …", -EINVAL
y	ciqlts9_4, after patch, c2h_term included	"pdu type %d has unexpected header length", -EPROTO	"pdu type %d has unexpected header length", -EPROTO	"unsupported pdu type …", -EINVAL
z	ciqlts9_4, after patch, c2h_term excluded	"unsupported pdu type …", -EINVAL	"pdu type %d has unexpected header length", -EPROTO	"unsupported pdu type …", -EINVAL

Solution a is to x not as b is to y, but as c is to z, thus the c, z pair was chosen.

kABI check: passed

DEBUG=1 CVE=CVE-2025-21927 ./ninja.sh _kabi_checked__x86_64--test--ciqlts9_4-CVE-2025-21927

ninja: Entering directory `/data/build/rocky-patching'
[0/1] Check ABI of kernel [ciqlts9_4-CVE-2025-21927]
++ uname -m
+ python3 /data/src/ctrliq-github/kernel-dist-git-el-9.4/SOURCES/check-kabi -k /data/src/ctrliq-github/kernel-dist-git-el-9.4/SOURCES/Module.kabi_x86_64 -s vms/x86_64--build--ciqlts9_4/build_files/kernel-src-tree-ciqlts9_4-CVE-2025-21927/Module.symvers
kABI check passed
+ touch state/kernels/ciqlts9_4-CVE-2025-21927/x86_64/kabi_checked

(The el-9.4 branch was missing from https://github.com/ciq-rocky-lts/kernel, it was added by hand on the imports/r9/kernel-5.14.0-427.42.1.el9_4 tag from the added https://git.rockylinux.org/staging/rpms/kernel.git repo)

Boot test: passed

boot_test.log

Kselftests: passed relative

Methodology

The selftests were source-compiled from the recent ciqlts9_4 branch (commit f794e72). The bpf suite was run from the kernel-selftests-internal package.

The tests were run using an explicit list which omitted certain tests known to give inconsistent results. Details in the src/run-kselftests.sh script of the rocky-patching project.

Coverage

bpf (patial, 11/39: get_cgroup_id_user, test_cgroup_storage, test_dev_cgroup, test_lpm_map, test_lru_map, test_maps, test_sock, test_sysctl, test_tag, test_tcpnotify_user, test_verifier), breakpoints, capabilities, clone3, cpu-hotplug, cpufreq, drivers/dma-buf, drivers/net/bonding, drivers/net/team, efivarfs, exec, filesystems/binderfs, filesystems/epoll, firmware, fpu, ftrace, futex, gpio, intel_pstate, iommu, ipc, ir, kcmp, kexec, kvm, landlock, lib, livepatch, lkdtm (except stack-entropy.sh), membarrier, memfd, memory-hotplug, mincore, mount, mqueue, nci, net/forwarding (except sch_tbf_root.sh, sch_red.sh, sch_tbf_prio.sh, sch_ets.sh, ipip_hier_gre_keys.sh, tc_police.sh, sch_tbf_ets.sh, dual_vxlan_bridge.sh, tc_actions.sh), net/hsr, net/mptcp (except simult_flows.sh), net (except reuseport_addr_any.sh, gro.sh, udpgso_bench.sh, xfrm_policy.sh, txtimestamp.sh, ip_defrag.sh, reuseaddr_conflict), netfilter (except nft_trans_stress.sh), nsfs, pid_namespace, pidfd, proc (except proc-pid-vm), pstore, ptrace, rlimits, rseq, seccomp, sgx, sigaltstack, size, splice, static_keys, sync, syscall_user_dispatch, sysctl, tc-testing, tdx, timens, timers, tmpfs, tpm2, tty, user, vDSO, x86, zram.

Reference

kselftests–ciqlts9_4–run1.log
kselftests–ciqlts9_4–run2.log
kselftests–ciqlts9_4–run3.log
kselftests–ciqlts9_4–run4.log

Runs 3 and 4 are complementary - the test 3 was interrupted and continued in 4.

Patch

kselftests–ciqlts9_4-CVE-2025-21927–run1.log

Comparison

./ktests.xsh diff -d kselftests*.log

Column    File
--------  ----------------------------------------------
Status0   kselftests--ciqlts9_4--run1.log
Status1   kselftests--ciqlts9_4--run2.log
Status2   kselftests--ciqlts9_4--run3.log
Status3   kselftests--ciqlts9_4--run4.log
Status4   kselftests--ciqlts9_4-CVE-2025-21927--run1.log

TestCase                                     Status0  Status1  Status2  Status3  Status4  Summary
net/forwarding:mirror_gre_bridge_1d_vlan.sh  pass     pass              fail     pass     diff
net/forwarding:router_bridge_1d_lag.sh       fail     fail              fail     pass     diff
net/forwarding:router_bridge_lag.sh          pass     fail              fail     fail     diff
net/mptcp:userspace_pm.sh                    fail     pass              pass     pass     diff

All differences are contained within the reference tests set. The net/forwarding:router_bridge_1d_lag.sh actually passes on the patched kernel unlike in the reference tests.

Differences highlights

net/forwarding:mirror_gre_bridge_1d_vlan.sh

A single subtest failed in a single test run. Marking as "flappy" for future tests before an investigation is done on how to obtain results consistency.

./ktests.xsh show_groups --test net/forwarding:mirror_gre_bridge_1d_vlan.sh -s kselftests*.log

kselftests--ciqlts9_4--run1.log:
kselftests--ciqlts9_4--run2.log:
kselftests--ciqlts9_4-CVE-2025-21927--run1.log:
net/forwarding:mirror_gre_bridge_1d_vlan.sh:
# TEST: ingress mirror to gretap (skip_hw)                            [ OK ]
# TEST: egress mirror to gretap (skip_hw)                             [ OK ]
# TEST: ingress mirror to ip6gretap (skip_hw)                         [ OK ]
# TEST: egress mirror to ip6gretap (skip_hw)                          [ OK ]
# TEST: mirror to gretap: STP state (skip_hw)                         [ OK ]
# TEST: mirror to ip6gretap: STP state (skip_hw)                      [ OK ]
# WARN: Could not test offloaded functionality
ok 1 selftests: net/forwarding: mirror_gre_bridge_1d_vlan.sh

kselftests--ciqlts9_4--run3.log:
net/forwarding:mirror_gre_bridge_1d_vlan.sh:
None

kselftests--ciqlts9_4--run4.log:
net/forwarding:mirror_gre_bridge_1d_vlan.sh:
# TEST: ingress mirror to gretap (skip_hw)                            [ OK ]
# TEST: egress mirror to gretap (skip_hw)                             [ OK ]
# TEST: ingress mirror to ip6gretap (skip_hw)                         [FAIL]
# 	Expected to capture 10 packets, got 13.
# TEST: egress mirror to ip6gretap (skip_hw)                          [ OK ]
# TEST: mirror to gretap: STP state (skip_hw)                         [ OK ]
# TEST: mirror to ip6gretap: STP state (skip_hw)                      [ OK ]
# WARN: Could not test offloaded functionality
not ok 1 selftests: net/forwarding: mirror_gre_bridge_1d_vlan.sh # exit=1

net/forwarding:router_bridge_1d_lag.sh, net/forwarding:router_bridge_lag.sh, net/mptcp:userspace_pm.sh

The tests show different patterns of failures. Marking as "flappy" for future runs before an investigation is done on how to obtain consistent results. Example for net/forwarding:router_bridge_lag.sh:

./ktests.xsh show_groups --test net/forwarding:router_bridge_lag.sh  -s kselftests--ciqlts9_4--run1.log kselftests--ciqlts9_4--run2.log kselftests--ciqlts9_4--run4.log kselftests--ciqlts9_4-CVE-2025-21927--run1.log

kselftests--ciqlts9_4--run1.log:
net/forwarding:router_bridge_lag.sh:
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth7
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Enslave veth7 to lag2
# INFO: Deslave veth1
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth7
# INFO: Enslave veth1 to lag2
# INFO: Enslave veth7 to lag2
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Remaster bridge slave
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth2
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Enslave veth2 to lag3
# INFO: Deslave veth4
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth2
# INFO: Enslave veth4 to lag3
# INFO: Enslave veth2 to lag3
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Move lag3 to the bridge, then out again
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
ok 1 selftests: net/forwarding: router_bridge_lag.sh

kselftests--ciqlts9_4--run2.log:
net/forwarding:router_bridge_lag.sh:
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth7
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Enslave veth7 to lag2
# INFO: Deslave veth1
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth7
# INFO: Enslave veth1 to lag2
# INFO: Enslave veth7 to lag2
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [ OK ]
# INFO: Remaster bridge slave
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Deslave veth2
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Enslave veth2 to lag3
# INFO: Deslave veth4
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Deslave veth2
# INFO: Enslave veth4 to lag3
# INFO: Enslave veth2 to lag3
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Move lag3 to the bridge, then out again
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
not ok 1 selftests: net/forwarding: router_bridge_lag.sh # exit=1

kselftests--ciqlts9_4--run4.log:
net/forwarding:router_bridge_lag.sh:
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth7
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Enslave veth7 to lag2
# INFO: Deslave veth1
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth7
# INFO: Enslave veth1 to lag2
# INFO: Enslave veth7 to lag2
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Remaster bridge slave
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth2
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Enslave veth2 to lag3
# INFO: Deslave veth4
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth2
# INFO: Enslave veth4 to lag3
# INFO: Enslave veth2 to lag3
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Move lag3 to the bridge, then out again
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
not ok 1 selftests: net/forwarding: router_bridge_lag.sh # exit=1

kselftests--ciqlts9_4-CVE-2025-21927--run1.log:
net/forwarding:router_bridge_lag.sh:
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# This program is not intended to be run as root.
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth7
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Enslave veth7 to lag2
# INFO: Deslave veth1
# TEST: ping                                                          [ OK ]
# TEST: ping6                                                         [ OK ]
# INFO: Deslave veth7
# INFO: Enslave veth1 to lag2
# INFO: Enslave veth7 to lag2
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Remaster bridge slave
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Deslave veth2
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Enslave veth2 to lag3
# INFO: Deslave veth4
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Deslave veth2
# INFO: Enslave veth4 to lag3
# INFO: Enslave veth2 to lag3
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
# INFO: Move lag3 to the bridge, then out again
# TEST: ping                                                          [FAIL]
# TEST: ping6                                                         [FAIL]
not ok 1 selftests: net/forwarding: router_bridge_lag.sh # exit=1

Specific tests: suspended

See the situation for #234.

[thefossguy-ciq]

🚤