Reject add sled requests for sleds that already exist

[jgallagher]

I ran into this as a part of #5625, where adding a previously-expunged sled appeared to succeed, but didn't actually add anything new.

Today if we try to add a sled that is already running, we get a 500, because Nexus fails when it tries to tell the sled-agent to start. But with this PR, we fail earlier: adding a sled that already has a subnet allocation fails before we even try to talk to the sled-agent, because it means someone has already added this sled:

root@oxz_switch:~# omdb -w nexus sleds add g2 i86pc
added sled g2 (i86pc): 90413e40-8139-43b4-9081-365dab6e5579
root@oxz_switch:~# omdb -w nexus sleds add g2 i86pc
Error: adding sled

Caused by:
    Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "9eb95a9f-3fe0-4f75-8846-13490b95500e", "content-length": "188", "date": "Tue, 30 Apr 2024 20:54:49 GMT"}; value: Error { error_code: Some("ObjectAlreadyExists"), message: "already exists: sled \"g2 / i86pc (90413e40-8139-43b4-9081-365dab6e5579)\"", request_id: "9eb95a9f-3fe0-4f75-8846-13490b95500e" }

This does change the external API slightly (204 -> 201 created, and we now return the ID), but I think (?) that's probably fine since we have no real consumers of that yet.

[andrewjstone]

Thanks for the fix!

[andrewjstone]

Why are these bootstore related changes in here?

[jgallagher]

Yeahhh, good question. I believe the sequence was:

1. I added the integration test, and it failed on the first sled add with an error that didn't make sense to me:

thread 'integration_tests::rack::test_sled_add' panicked at clients/sled-agent-client/src/lib.rs:451:58:
0:0:0:21::/64: doesn't match pattern "^([fF][dD])[0-9a-fA-F]{2}:(([0-9a-fA-F]{1,4}:){6}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,6}:)([0-9a-fA-F]{1,4})?\/([0-9]|[1-9][0-9]|1[0-1][0-9]|12[0-8])$"

2. 0:0:0:21::/64 is a valid IPv6 network, but it is not a valid Ipv6Net, which we require to be in the fd00::/8 range.
3. Why did we get a sled network that starts with 0:0:0:...? The rack_subnet in the database after the integration test does its fake RSS / setup was set to ::/56, even though we appear to set a valid rack_subnet when we call rack_initialize.
4. AFAIK, the rack_subnet we pass to rack_initialize is completely ignored. Instead, Nexus sets the rack subnet by reading the early network config from the bootstore.
5. The simulated sled agent always returns an empty(ish) early network config, including a rack_subnet of ::/56.

At this point, there seemed to be several options for getting this test to pass, including at least:

• Change rack_initialize to look at the rack subnet we give it (seems reasonable? but I don't know why it's reading from the bootstore, so also seems dicey?)
• Make nexus-test-utils set the rack subnet by hand
• Make the simulated sled agent support a more reasonable fake bootstore, so nexus-test-utils can set it here and rack_initialize can read it, so we set the rack_subnet similarly to how things happen in prod

and I went with option 3.

[andrewjstone]

Thank you for your sufferi^W^Wfix.

[andrewjstone]

Hmm, I'm thinking that we probably shouldn't read this from the bootstore. I don't know why we wouldn't use what's in rack_initialize. This seems like a bug waiting to happen, although I also don't see how that bug would arise exactly, given what's in the bootstore and rack_initialize should be the same.