Blueprint structure allows a variety of "illegal" combinations

[jgallagher]

Blueprint currently has four different maps, all keyed by sled ID:

omicron/nexus/types/src/deployment.rs

Lines 144 to 163 in 7cf372d

	/// A map of sled id -> desired state of the sled.
	///
	/// A sled is considered part of the control plane cluster iff it has an
	/// entry in this map.
	pub sled_state: BTreeMap<SledUuid, SledState>,
	/// A map of sled id -> zones deployed on each sled, along with the
	/// [`BlueprintZoneDisposition`] for each zone.
	///
	/// Unlike `sled_state`, this map may contain entries for sleds that are no
	/// longer a part of the control plane cluster (e.g., sleds that have been
	/// decommissioned, but still have expunged zones where cleanup has not yet
	/// completed).
	pub blueprint_zones: BTreeMap<SledUuid, BlueprintZonesConfig>,
	/// A map of sled id -> disks in use on each sled.
	pub blueprint_disks: BTreeMap<SledUuid, BlueprintPhysicalDisksConfig>,
	/// A map of sled id -> datasets in use on each sled
	pub blueprint_datasets: BTreeMap<SledUuid, BlueprintDatasetsConfig>,

In general we would expect all those maps to have the same keys, but in practice that isn't true. A couple examples:

• sled_state drops decommissioned sleds, but they may still exist in the other maps (but should only contain expunged values). We have to work around this when diff'ing blueprints.
• Clients that use BlueprintBuilder, not the planner, that fail to call sled_ensure_disks / sled_ensure_datasets at appropriate times will emit blueprints that contain zones that reference datasets / disks that one would reasonably expect to exist in their relative maps, but those maps may be out of date or empty. Empty maps happen to work today because for backwards compatibility reasons we allow them to be empty, but doing so is a little messy and will eventually be removed once production systems can all be expected to have populated maps. That will break any of these clients (tests and reconfigurator-cli, today).

I think (?) we should probably only have one map here, keyed by sled ID, with values that encompass all the blueprint details for a single sled (today: state + zones + disks + datasets, growing in the future to support update versioning as needed). This may require some fundamental rework of BlueprintBuilder and possibly the planner, as today they manage these maps mostly independently (which is problematic!).

[jgallagher]

This shouldn't have been closed; github picked up on "fix #7078" in a commit message, but in context it's clearly not that the commit fixes this.