Adding a previously-expunged sled leaves it expunged

[jgallagher]

On an a4x2 testbed, I went through this sequence:

• Set up with three sleds (g0, g1, g3)
• Add sled g2
• Step through blueprints to give g2 NTP and crucible
• Use omdb db reconfigurator-save, reconfigurator-cli, and omdb nexus blueprints import to craft a blueprint that gives g2 a Nexus zone
• Confirm the Nexus zone comes online and is externally reachable
• Manually clean-slate the sled (disable sled-agent and mg-ddm, remove relevant files in /pool/int/*)
• a4x2 hyperstop g2 to shut off the sled
• omdb db nexus sleds expunge ... to expunge the sled
• a4x2 hyperstart g2 to bring the sled back online
• Reenable sled-agent and mg-ddm

At this point the sled shows up as uninitialized:

root@oxz_switch:~# omdb nexus sleds list-uninitialized
RACK_ID                              CUBBY SERIAL PART  REVISION
73d99599-d25c-41d6-a78b-246ceba80faa 2     g2     i86pc 0

And adding it appears to succeed:

root@oxz_switch:~# omdb -w nexus sleds add g2 i86pc
added sled g2 (i86pc)

However, after doing so, it still shows up in list-uninitialized:

root@oxz_switch:~# omdb nexus sleds list-uninitialized
RACK_ID                              CUBBY SERIAL PART  REVISION
73d99599-d25c-41d6-a78b-246ceba80faa 2     g2     i86pc 0

and we can even apparently add it again (again successfully):

root@oxz_switch:~# omdb -w nexus sleds add g2 i86pc
added sled g2 (i86pc)

but it remains in the list-uninitialized output.

We expect this to work, and the add should create a new sled_id for this same baseboard we've seen before. But looking in CRDB, we only have one row for g2 (the one created when we added it the first time), and it is still expunged:

root@[fd00:1122:3344:103::3]:32221/omicron> select id,time_created,time_modified,serial_number,sled_policy from sled;
                   id                  |         time_created          |         time_modified         | serial_number | sled_policy
---------------------------------------+-------------------------------+-------------------------------+---------------+--------------
  353582e7-6a8d-4de4-8aba-ea97ecd2f11b | 2024-04-25 17:24:32.016339+00 | 2024-04-25 19:06:50.980526+00 | g2            | expunged
  692fd79d-6f93-4787-9fa4-bbc47a3cc494 | 2024-04-25 17:18:30.451498+00 | 2024-04-25 17:18:30.451498+00 | g3            | in_service
  cacc195b-5c78-4e08-833c-5af089815eba | 2024-04-25 17:18:28.702021+00 | 2024-04-25 17:18:28.702021+00 | g0            | in_service
  fca3f869-15ce-49ee-a872-7a4129877efb | 2024-04-25 17:18:31.583653+00 | 2024-04-25 17:18:31.583653+00 | g1            | in_service
(4 rows)

Tracing through the add sled flow, I think the problem is in DataStore::allocate_sled_underlay_subnet_octets(). This is the method that creates a new sled ID (line 309); however, hw_baseboard_id is tied to the baseboard which hasn't changed. If we find a matching allocation on lines 315-318, we'll stop and return that prior allocation. In CRDB, we can see that we do have a prior allocation for this baseboard (as expected, since we've added this sled before!):

root@[fd00:1122:3344:103::3]:32221/omicron> select * from sled_underlay_subnet_allocation;
            hw_baseboard_id            |               rack_id                |               sled_id                | subnet_octet
---------------------------------------+--------------------------------------+--------------------------------------+---------------
  a41243b4-dc81-4db4-9223-4dd83dd00eea | 73d99599-d25c-41d6-a78b-246ceba80faa | 353582e7-6a8d-4de4-8aba-ea97ecd2f11b |           33
(1 row)

root@[fd00:1122:3344:103::3]:32221/omicron> select * from hw_baseboard_id where id = 'a41243b4-dc81-4db4-9223-4dd83dd00eea';
                   id                  | part_number | serial_number
---------------------------------------+-------------+----------------
  a41243b4-dc81-4db4-9223-4dd83dd00eea | i86pc       | g2
(1 row)

I think (?) if allocate_sled_underlay_subnet_octets() finds a prior allocation with a matching baseboard, it should check the sled table for that allocation's sled_id, and if it's expunged, ignore that allocation and continue looping (resulting in it either finding a non-expunged match, or exiting the loop and returning new_allocation).

[davepacheco]

Would implementing #5554 fix this?

[jgallagher]

Would implementing #5554 fix this?

Probably, with at least two caveats:

• sled expungement should clean up sled underlay allocations #5554 notes that we probably want to gate cleanup on decommissioning, which we don't do yet.
• If we added the sled back between expunging it and it being decommissioned (and therefore before cleanup), we'd still hit this.