[sled-agent] Launch from bootstrap agent, derive bootstrap addresses, allocate sled /64s

[smklein]

Bootstrap Agent

• Stop relying on the "Unspecified" address for bootstrap addrs - instead, follow the advice of RFD 63, and allocate addresses explicitly based on the physical link MAC address. Admittedly, RFD 63 suggests using the SP MAC, but the physical link MAC is the one we have today.
  • Re-work some of the link-local multicast code, ensuring that the sender address properly advertises the bootstrap address, rather than an unspecified interface. Provide an integration test (illumos-only) to validate that this works how we want it to work - at least, until integration with Maghemite progresses far enough that multicast can be removed.
• Avoid launching the Sled Agent "by default" - instead, keep the sled agent uninitialized until explicitly requested, alongside a whole /64 for which it should be responsible. Store configuration data to local storage within OMICRON_CONFIG_PATH to automatically launch these agents on reboot, after RSS setup.
• Provide a PeerMonitorObserver structure that may be used to await the notification of new peers within the bootstrap agent.

Sled Agent

• Acquire initial address from subnet, avoid blocking on Nexus notification during setup.

RSS

• Await the appearance of enough bootstrap agent peers before sending requests. This emulates an operator "seeing enough sleds come up" before deciding to initialize the rack.
• Allocates subnets based on an arbitrary initial order of peers (presumably Nexus will take over this allocation later on).

Configs (mostly in smf/)

• Adjust all addresses to be within the subnet of the first sled. Also, condense the range of IPv6 addresses we're using for hardcoded stuff.
• Remove the explicit addresses for the "bootstrap_agent" and "sled_agent" - these should be inferred at runtime.

TODO

• Deal with idempotency conditions - what if RSS fails partway through operation?
• Add more tests?
• Take another swing at the PeerMonitorObserver structure, maybe it can be simplified
• Grep for old addresses (See: README) and update them

TODOs for later PRs (but which should be unblocked now)

• Remove more addresses from config files, as they may be inferred at runtime from within the sled subnet [Sled Agent] [Bootstrap Agent] Distinguish between "package-specific" and "sled-specific" configurations #822
• Transfer more responsibility upward to Nexus (e.g., Crucible, Oximeter, Clickhouse provisioning, additional subnet allocations) [rss][nexus][sled-agent] Responsibility for deploying services should (where possible) migrate into Nexus #732
• Sled Agent: Assign UUID at runtime. "One less thing" from the config.

Fixes #821

[jmpesp]

Looks good to me, some questions:

[jmpesp]

the way this is worded seems like it's not done yet. did create_plan perform a flush to durable storage?

[smklein]

I'll move the comment up, indicating that it is supposed to refer to the moment create_plan completes.

• create_plan performs the write to durable storage.
• It's not calling flush explicitly, but I believe it's being implicitly called when the tokio::fs::write - which calls std::fs::write - which itself calls File::create - drops the created File object.

[jmpesp]

is this cleaned up after the test?

[smklein]

It will now.

[bnaecker]

This looks really good. I have a few clarifying comments and questions, but overall a +1 from me!

[bnaecker]

Looks like you've got a test flake on the check-omicron-deployment step. I had the same last week, seems an OOM situation on macOS, bummer.

Otherwise, LGTM!