rack2: Dataset ledger and blueprint are missing zone filesystem roots

[jgallagher]

Looking at the ledger for datasets on a sled of dublin, we see (for each disk) a debug, zone, crucible dataset, plus a set of zone/oxz_* for all the zone filesystem roots:

BRM42220026 #  cat /pool/int/2196ba5d-e211-4495-9c0b-e0948ba0afbd/config/omicron-datasets.json  | jq '.datasets' | grep kind  | sort
      "kind": "cockroachdb"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "external_dns"
      "kind": "internal_dns"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone/oxz_cockroachdb_d5dfdf09-a943-4e8c-a7f0-18e1e1644dae"
      "kind": "zone/oxz_crucible_27e2564b-4d14-4ba7-a198-7fd39c0345b4"
      "kind": "zone/oxz_crucible_36952839-104d-418c-a6cb-cc509b12531e"
      "kind": "zone/oxz_crucible_4586ae39-ec1c-4d1e-a36c-4c1e6621ec3c"
      "kind": "zone/oxz_crucible_583f9a41-2587-436e-8b1d-c31892ec84ff"
      "kind": "zone/oxz_crucible_722a3bd6-09d1-49af-8001-aabd2d8fd2c5"
      "kind": "zone/oxz_crucible_75936029-f293-40fb-96d5-f434332953cc"
      "kind": "zone/oxz_crucible_e5a997e7-1551-4d9f-95b1-ede53409d0a0"
      "kind": "zone/oxz_crucible_e8b9f2da-79c4-4ded-a7e4-631f55429c2a"
      "kind": "zone/oxz_crucible_f3c60b6c-76cf-4535-b07b-9fd97abca79c"
      "kind": "zone/oxz_crucible_pantry_462c1eae-4f59-4314-8308-fea70aa6ba5d"
      "kind": "zone/oxz_external_dns_7bc405cf-c950-48ef-975e-d80756d67292"
      "kind": "zone/oxz_internal_dns_2fdd5688-482b-40d7-b16b-b2c01715e6fc"
      "kind": "zone/oxz_ntp_68ec5db8-1f62-4640-ba51-44c09cba28de"
      "kind": "zone/oxz_oximeter_a97b1d79-18a2-4589-b7c3-a9bb3929e254"

However, on a sled of dogfood, none of the zone/oxz_* filesystem roots are present; only the durable datasets:

BRM42220014 # cat /pool/int/7c377f39-95bf-4074-8dda-7a7a102b9d2c/config/omicron-datasets.json  | jq '.datasets' | grep kind | sort
      "kind": "cockroachdb"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "crucible"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "debug"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"
      "kind": "zone"

This appears consistent with the blueprint: the current target blueprint has filesystem_pool: null for all zones. However, it is surprising; we've been doing work under the assumption that this value is set for all zones, and disk expungement may not work correctly (expunging a disk should expunge any zones that depended on that disk, but that assumes reconfigurator knows what zpools the zones are using!).

[jgallagher]

Some notes on starting to look into fixing this. An example for zpool from one sled of the dogfood rack; the datasets reported in inventory collections are:

id                                   name
------------------------------------ --------------------------------------------
                                null oxp_7e1293ad-b903-4054-aeae-2182d5e4a785
fcdda266-fc6a-4518-89db-aec007a4b682 oxp_7e1293ad-b903-4054-aeae-2182d5e4a785/crucible
                                null oxp_7e1293ad-b903-4054-aeae-2182d5e4a785/crypt
db53fc46-ad60-4721-ae7f-df42ab84c3ff oxp_7e1293ad-b903-4054-aeae-2182d5e4a785/crypt/debug
4857d53c-9279-4d51-9433-b027017b6f18 oxp_7e1293ad-b903-4054-aeae-2182d5e4a785/crypt/zone
                                null oxp_7e1293ad-b903-4054-aeae-2182d5e4a785/crypt/zone/oxz_crucible_fcdda266-fc6a-4518-89db-aec007a4b682
                                null oxp_7e1293ad-b903-4054-aeae-2182d5e4a785/crypt/zone/oxz_nexus_20b100d0-84c3-4119-aa9b-0c632b0b6a3a
                                null oxp_7e1293ad-b903-4054-aeae-2182d5e4a785/crypt/zone/oxz_ntp_c3ec3d1a-3172-4d36-bfd3-f54a04d5ba55

and the entries for this zpool in blueprint_datasets are:

id                                   pool                                     kind
------------------------------------ ---------------------------------------- ------------
4857d53c-9279-4d51-9433-b027017b6f18 oxp_7e1293ad-b903-4054-aeae-2182d5e4a785 zone
db53fc46-ad60-4721-ae7f-df42ab84c3ff oxp_7e1293ad-b903-4054-aeae-2182d5e4a785 debug
fcdda266-fc6a-4518-89db-aec007a4b682 oxp_7e1293ad-b903-4054-aeae-2182d5e4a785 crucible

This is all consistent with the original issue:

• The nexus, ntp, and crucible blueprint zone configs all have filesystem_pool: None
• Therefore, none of those zones have a corresponding TransientZoneRoot { .. } dataset in blueprint_datasets
• The three datasets that do exist in the blueprint are faithfully reported by sled-agent

Backfilling filesystem_pool and creating the TransientZoneRoot datasets in the blueprint appears to be straightforward: the inventory dataset names contain all the information we need to know which zpool each zone's actual transient root is on. However, "generate a new blueprint that fills in the filesystem_pool and creates the corresponding TransientZoneRoot dataset" has a couple problems in practice:

1. For zones that don't have a durable dataset, there's a TOCTOU problem. If sled-agent has restarted since the inventory collection we're looking at, it might have chosen a different zpool for those zones' transient roots. (For zones with a durable dataset, it picks the same zpool as the durable dataset; for zones without, it chooses a random zpool each time it starts the zone.)
2. Filling in filesystem_pool will change the OmicronZoneConfig for every zone. sled-agent will treat that as a command to shut down all the zones and then restart them. This seems... unfortunately disruptive. (Although, it does address the TOCTOU problem in 1, in that it forces sled-agent to shut down the zone and restart it using the zpool the blueprint has chosen.)

[smklein]

2. Filling in filesystem_pool will change the OmicronZoneConfig for every zone. sled-agent will treat that as a command to shut down all the zones and then restart them. This seems... unfortunately disruptive. (Although, it does address the TOCTOU problem in 1, in that it forces sled-agent to shut down the zone and restart it using the zpool the blueprint has chosen.)

One possible solution here - the logic to decide "which zones to add/remove" in the sled agent compares the entire zone config. So, as you say, changing from filesystem_pool: None -> filesystem_pool: Some(...) will be treated as a "totally different config", and will remove + recreate the zone.

We could make this case more cautious, and perhaps:

• If the config is different, even ignoring filesystem_pool, treat the config as "different", and destroy/recreate the zone.
• If the config is the same, aside from filesystem_pool, and the transient filesystem root matches the choice used by the zone, treat it as a no-op. Do not shut down / restart the zone.
• If the config is the same, aside from filesystem_pool, and the transient filesystem root does not match the choice used by the zone, we could remove/recreate the zone. This would also handle your TOCTTOU case of the sled agent restarting. Alternatively, we could throw an error back to the reconfigurator, and try to re-evaluate inventory, but this would likely require a new blueprint.