[RSS] Move switch_port table population to a Nexus bg task

[jgallagher]

Please start reviewing with the module-level comments in nexus/src/app/background/tasks/populate_switch_ports.rs; they attempt to explain the history and motivation of this change. It's probably the most important thing to review - if any of that reasoning is wrong, the change I'm making in this PR probably is too!

This is (preemptively) removing one of the internal-to-sled-agent uses of the early_networking module, in the hopes that we can eventually remove it entirely. Also closes #3069.

[internet-diglett]

Taking a look at this now

[internet-diglett]

LGTM!

[internet-diglett]

nice test!

[internet-diglett]

Looks like we might be also closing #3602 with this change? 🤔

[jgallagher]

I think that's mostly true, yeah. We did get rid of ExternalPortDiscovery::Static. I didn't want to add a delay to every test to wait for us to discover the ports, though, so we still inject qsfp0 for both switches on test startup:

omicron/nexus/src/lib.rs

Lines 398 to 418 in 5c70790

	// In the real rack init handoff, the `rack_network_config` will contain
	// at least one configured port, and handoff will only complete
	// successfully if the `populate_switch_ports` background task has
	// completed successfully to populate the corresponding qsfp* values in
	// the `switch_port` table. We could block here until that task
	// activates successfully, contacting whatever `dpd` instance has been
	// stood up for this test, but in the interest of streamlining this
	// handoff (which happens for _every_ Nexus test, including those
	// completely uninterested in switch port interaction), we'll insert a
	// single qsfp0 for each switch to the db directly.
	for which_switch in SwitchSlot::iter() {
	datastore
	.switch_port_create(
	&opctx,
	config.deployment.rack_id,
	which_switch,
	nexus_db_model::Name("qsfp0".parse().unwrap()),
	)
	.await
	.expect("inserted qsfp0");
	}

Do you think it's fine to close the issue despite that? Or should we change the test startup path here to wait until the bg task has talked to dendrite the way real RSS does (or at least see how badly that slows down the tests)?

[internet-diglett]

I think it's okay to close it since we've taken it out of the production code path

[jgallagher]

Made one minor change to not stop on first error in 77605f3