Describe the problem
roachprod pins the Azure Ubuntu image to 22.04.202312060 (Dec 2023, kernel 6.2.0-1018-azure) in azure.go#L1080. On v6-series VMs (e.g. Standard_D4s_v6), which have MANA NICs only (no Mellanox), this kernel's mana driver fails to probe:
mana 7870:00:00.0: HWC: Failed hw_channel req: 0xc00000bb
mana 7870:00:00.0: gdma probe failed: err = -71
All traffic then takes the unaccelerated synthetic hv_netvsc path. Under load (e.g. IMPORT during test setup), inter-node RPC latency spikes to seconds (latency jump (prev avg 0.45ms, current 4472.26ms)), gRPC connections flap, and DistSQL flows die. The flake is nondeterministic (host-placement-dependent) but the degraded networking is deterministic: every v6 VM runs unaccelerated. Dv5/Ebsv5-series are also eligible to land on MANA hosts now, so exposure will grow.
Verified fix
Booting a Standard_D4s_v6 in eastus with the current jammy gen2 image 22.04.202607140 (kernel 6.8.0-1062-azure), MANA probes cleanly and traffic flows through the VF:
mana 7870:00:00.0 eth1: joined to eth0
hv_netvsc ... eth0: Data path switched to VF: enP30832s1
vf_rx_packets: 9439 / vf_tx_dropped: 0
Next steps
Jira issue: CRDB-66505
Describe the problem
roachprod pins the Azure Ubuntu image to
22.04.202312060(Dec 2023, kernel6.2.0-1018-azure) in azure.go#L1080. On v6-series VMs (e.g.Standard_D4s_v6), which have MANA NICs only (no Mellanox), this kernel'smanadriver fails to probe:All traffic then takes the unaccelerated synthetic
hv_netvscpath. Under load (e.g. IMPORT during test setup), inter-node RPC latency spikes to seconds (latency jump (prev avg 0.45ms, current 4472.26ms)), gRPC connections flap, and DistSQL flows die. The flake is nondeterministic (host-placement-dependent) but the degraded networking is deterministic: every v6 VM runs unaccelerated. Dv5/Ebsv5-series are also eligible to land on MANA hosts now, so exposure will grow.Verified fix
Booting a
Standard_D4s_v6in eastus with the current jammy gen2 image22.04.202607140(kernel6.8.0-1062-azure), MANA probes cleanly and traffic flows through the VF:Next steps
imageSKU(22_04-lts,22_04-lts-gen2,22_04-lts-arm64)Jira issue: CRDB-66505