Skip to content

roachprod: bump pinned Azure image with broken MANA on v6-series VMs #173178

Description

@andyyang890

Describe the problem

roachprod pins the Azure Ubuntu image to 22.04.202312060 (Dec 2023, kernel 6.2.0-1018-azure) in azure.go#L1080. On v6-series VMs (e.g. Standard_D4s_v6), which have MANA NICs only (no Mellanox), this kernel's mana driver fails to probe:

mana 7870:00:00.0: HWC: Failed hw_channel req: 0xc00000bb
mana 7870:00:00.0: gdma probe failed: err = -71

All traffic then takes the unaccelerated synthetic hv_netvsc path. Under load (e.g. IMPORT during test setup), inter-node RPC latency spikes to seconds (latency jump (prev avg 0.45ms, current 4472.26ms)), gRPC connections flap, and DistSQL flows die. The flake is nondeterministic (host-placement-dependent) but the degraded networking is deterministic: every v6 VM runs unaccelerated. Dv5/Ebsv5-series are also eligible to land on MANA hosts now, so exposure will grow.

Verified fix

Booting a Standard_D4s_v6 in eastus with the current jammy gen2 image 22.04.202607140 (kernel 6.8.0-1062-azure), MANA probes cleanly and traffic flows through the VF:

mana 7870:00:00.0 eth1: joined to eth0
hv_netvsc ... eth0: Data path switched to VF: enP30832s1
vf_rx_packets: 9439 / vf_tx_dropped: 0

Next steps

  • Confirm a version string available for all three SKUs used by imageSKU (22_04-lts, 22_04-lts-gen2, 22_04-lts-arm64)
  • Bump the pin; watch Azure nightlies (fleet-wide kernel jump 6.2 -> 6.8)

Jira issue: CRDB-66505

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-roachprodC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-agentFiled by an AI agent; usually the result of a human/agent investigation sessionT-testengTestEng Team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions