test failed in CI: test_join_by_ip_existing_group #9711
Description
This test failed on a CI run on #9709:
https://github.com/oxidecomputer/omicron/pull/9709/checks?check_run_id=61188463967
Log showing the specific test failure:
Excerpt from the log showing the failure:
8885 2026-01-22T20:07:12.665Z stdout ───
8886 2026-01-22T20:07:12.665Z
8887 2026-01-22T20:07:12.665Z running 1 test
8888 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8889 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: System }, version: Some(RouterVersion { router_id: 91d74552-6045-45ec-8999-46245913e5d8, version: 5 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(9056d208-bef7-4327-923d-71ee42e5fa47)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(9056d208-bef7-4327-923d-71ee42e5fa47)) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd1e:4e75:3b8f::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd1e:4e75:3b8f::, width: 64 })) }} }
8890 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8891 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: System }, version: Some(RouterVersion { router_id: 91d74552-6045-45ec-8999-46245913e5d8, version: 6 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(9056d208-bef7-4327-923d-71ee42e5fa47)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd1e:4e75:3b8f::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd1e:4e75:3b8f::, width: 64 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(9056d208-bef7-4327-923d-71ee42e5fa47)) }} }
8892 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8893 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8894 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8895 2026-01-22T20:07:12.665Z test integration_tests::multicast::api::test_join_by_ip_existing_group has been running for over 60 seconds
8896 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8897 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8898 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8899 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8900 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8901 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8902 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8903 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8904 2026-01-22T20:07:12.665Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8905 2026-01-22T20:07:12.666Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8906 2026-01-22T20:07:12.666Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8907 2026-01-22T20:07:12.666Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8908 2026-01-22T20:07:12.666Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8909 2026-01-22T20:07:12.666Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8910 2026-01-22T20:07:12.666Z sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(9092426), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8911 2026-01-22T20:07:12.666Z test integration_tests::multicast::api::test_join_by_ip_existing_group ... FAILED
8912 2026-01-22T20:07:12.666Z
8913 2026-01-22T20:07:12.666Z failures:
8914 2026-01-22T20:07:12.666Z
8915 2026-01-22T20:07:12.666Z failures:
8916 2026-01-22T20:07:12.666Z integration_tests::multicast::api::test_join_by_ip_existing_group
8917 2026-01-22T20:07:12.666Z
8918 2026-01-22T20:07:12.666Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 618 filtered out; finished in 357.21s
8919 2026-01-22T20:07:12.666Z
8920 2026-01-22T20:07:12.666Z stderr ───
8921 2026-01-22T20:07:12.666Z log file: /var/tmp/omicron_tmp/test_all-3df2ed71801be079-test_join_by_ip_existing_group.225601.0.log
8922 2026-01-22T20:07:12.666Z note: configured to log to "/var/tmp/omicron_tmp/test_all-3df2ed71801be079-test_join_by_ip_existing_group.225601.0.log"
8923 2026-01-22T20:07:12.666Z DB URL: postgresql://root@[::1]:35429/omicron?sslmode=disable
8924 2026-01-22T20:07:12.666Z DB address: [::1]:35429
8925 2026-01-22T20:07:12.666Z log file: /var/tmp/omicron_tmp/test_all-3df2ed71801be079-test_join_by_ip_existing_group.225601.2.log
8926 2026-01-22T20:07:12.666Z note: configured to log to "/var/tmp/omicron_tmp/test_all-3df2ed71801be079-test_join_by_ip_existing_group.225601.2.log"
8927 2026-01-22T20:07:12.666Z Instance existing-inst-1 in Starting state - will wait for Running then stop
8928 2026-01-22T20:07:12.666Z Instance existing-inst-2 in Starting state - will wait for Running then stop
8929 2026-01-22T20:07:12.666Z Waiting for 2 instances to finish starting...
8930 2026-01-22T20:07:12.666Z Instance existing-inst-1 reached Running state
8931 2026-01-22T20:07:12.666Z Instance existing-inst-2 reached Running state
8932 2026-01-22T20:07:12.666Z
8933 2026-01-22T20:07:12.666Z thread 'integration_tests::multicast::api::test_join_by_ip_existing_group' (225602) panicked at nexus/tests/integration_tests/instances.rs:8521:19:
8934 2026-01-22T20:07:12.666Z instance 3301b286-46c8-453e-a539-4f5eb4b8d9ec did not transition to Stopped after 320s: timed out after 320.708415726s