Slow disk deletion and resurrection of previously deleted disks

[askfongjojo]

On rack2 that currently runs acdcbd94a4b734f386272e4a57e55348ab429bda, deleting 90+ disks took more than 15 minutes. This is substantially slower than what it used to be.

There are also a large nunber of previously deleted disks showing up in faulted state:

root@oxz_switch0:~# omdb db disks list | grep faulted
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (136.0.0)
WARN: listing disks: found 500 items (the limit).  There may be more items that were ignored.  Consider overriding with --fetch-limit.
03a8d85b-2b24-4cd0-acfe-4665e621239c 10 GiB   faulted                    deleted-03a8d85b-2b24-4cd0-acfe-4665e621239c                    -                                    
053ffff8-37f4-4243-981a-d6509255463f 10 GiB   faulted                    deleted-053ffff8-37f4-4243-981a-d6509255463f                    -                                    
08dbff70-58e8-4066-a269-79c3eb4f5d84 10 GiB   faulted                    deleted-08dbff70-58e8-4066-a269-79c3eb4f5d84                    -                                    
0a3939fa-0e38-4bfd-90af-56a3aee57286 10 GiB   faulted                    deleted-0a3939fa-0e38-4bfd-90af-56a3aee57286                    -                                    
0b4cddc4-cdf9-4645-9521-7b98bd5d60d5 10 GiB   faulted                    deleted-0b4cddc4-cdf9-4645-9521-7b98bd5d60d5                    -                                    
0d5bd532-bfe1-4472-86ff-7247cdfcadea 10 GiB   faulted                    deleted-0d5bd532-bfe1-4472-86ff-7247cdfcadea                    -                                    
1173bf29-c95e-46e6-ada3-b8d426d1ad59 10 GiB   faulted                    deleted-1173bf29-c95e-46e6-ada3-b8d426d1ad59                    -                                    
11b84367-86f3-490e-b650-afeadd4a0d8a 10 GiB   faulted                    deleted-11b84367-86f3-490e-b650-afeadd4a0d8a                    -                                    
125b61bb-6814-4e51-acbb-03fba2922a61 10 GiB   faulted                    deleted-125b61bb-6814-4e51-acbb-03fba2922a61                    -                                    
16f66d78-547e-4a35-a2d2-bdcf105d19ed 10 GiB   faulted                    deleted-16f66d78-547e-4a35-a2d2-bdcf105d19ed                    -                                    
2086c370-c628-45f5-8866-63d2fdb5851c 10 GiB   faulted                    deleted-2086c370-c628-45f5-8866-63d2fdb5851c                    -                                    
20da6097-e2c2-46c8-a268-92d18c61049f 10 GiB   faulted                    deleted-20da6097-e2c2-46c8-a268-92d18c61049f                    -                                    
29801c82-34df-46ae-b19a-faf068afe00f 10 GiB   faulted                    deleted-29801c82-34df-46ae-b19a-faf068afe00f                    -                                    
29f28827-0047-44f3-9489-7e142a15e0a3 10 GiB   faulted                    deleted-29f28827-0047-44f3-9489-7e142a15e0a3                    -                                    
2afe4eb1-c5eb-4001-85b5-44a2ed4bafdd 10 GiB   faulted                    deleted-2afe4eb1-c5eb-4001-85b5-44a2ed4bafdd                    -                                    
34730fa7-397f-41ab-af8c-8b4f1ed35a5c 10 GiB   faulted                    deleted-34730fa7-397f-41ab-af8c-8b4f1ed35a5c                    -                                    
39426cdb-1ffc-4f3c-ad9a-4c4bcec85950 10 GiB   faulted                    deleted-39426cdb-1ffc-4f3c-ad9a-4c4bcec85950                    -                                    
3e1bb4d6-6902-4b8b-ac06-b8025e567b69 10 GiB   faulted                    deleted-3e1bb4d6-6902-4b8b-ac06-b8025e567b69                    -                                    
3f0d03c7-362c-4ec5-a78f-ba9100b22533 10 GiB   faulted                    deleted-3f0d03c7-362c-4ec5-a78f-ba9100b22533                    -                                    
4340a280-2d7f-4553-9514-db31654928c7 10 GiB   faulted                    deleted-4340a280-2d7f-4553-9514-db31654928c7                    -                                    
446fee40-debc-4530-bbd3-c25f7af98ec0 10 GiB   faulted                    deleted-446fee40-debc-4530-bbd3-c25f7af98ec0                    -                                    
5bc81734-99eb-4a6f-9dc4-8c6d7d66dcad 10 GiB   faulted                    deleted-5bc81734-99eb-4a6f-9dc4-8c6d7d66dcad                    -                                    
64b7c17e-823c-4c68-9af9-c0f58ab37b37 10 GiB   faulted                    deleted-64b7c17e-823c-4c68-9af9-c0f58ab37b37                    -                                    
67fb5eaf-d9b7-4329-9a82-22cb4be50266 10 GiB   faulted                    deleted-67fb5eaf-d9b7-4329-9a82-22cb4be50266                    -                                    
69ba2dfc-b424-4933-9339-990f7eefffd9 10 GiB   faulted                    deleted-69ba2dfc-b424-4933-9339-990f7eefffd9                    -                                    
6a378f6b-15ee-4b18-98de-304793d79c4d 10 GiB   faulted                    deleted-6a378f6b-15ee-4b18-98de-304793d79c4d                    -                                    
6a6b68eb-cc59-4bf4-9b29-4ee8f48ed174 10 GiB   faulted                    deleted-6a6b68eb-cc59-4bf4-9b29-4ee8f48ed174                    -                                    
6e6e3453-10c0-4248-ac9b-354ed142849f 10 GiB   faulted                    deleted-6e6e3453-10c0-4248-ac9b-354ed142849f                    -                                    
70c2d84b-7df3-4f05-af25-f8ffc6bcb298 10 GiB   faulted                    deleted-70c2d84b-7df3-4f05-af25-f8ffc6bcb298                    -                                    
7600b290-c704-426b-b552-87d89decdac8 10 GiB   faulted                    deleted-7600b290-c704-426b-b552-87d89decdac8                    -                                    
7689005a-215a-4fd1-94a2-cceabf2c8308 10 GiB   faulted                    deleted-7689005a-215a-4fd1-94a2-cceabf2c8308                    -                                    
78b79d84-a5aa-431a-b981-4949d1e2b457 10 GiB   faulted                    deleted-78b79d84-a5aa-431a-b981-4949d1e2b457                    -                                    
79c6015f-0066-48ba-a973-0af08c9e525c 10 GiB   faulted                    deleted-79c6015f-0066-48ba-a973-0af08c9e525c                    -                                    
7b47ff9b-ed7d-4914-8f65-14101c59da8c 10 GiB   faulted                    deleted-7b47ff9b-ed7d-4914-8f65-14101c59da8c                    -                                    
800fa6eb-3989-4dec-b5aa-fa08adb077e1 10 GiB   faulted                    deleted-800fa6eb-3989-4dec-b5aa-fa08adb077e1                    -                                    
80b5b484-9f21-4c2f-ac76-8368a0471027 10 GiB   faulted                    deleted-80b5b484-9f21-4c2f-ac76-8368a0471027                    -                                    
8808c192-b0e4-4da6-9055-6229e339c6ba 10 GiB   faulted                    deleted-8808c192-b0e4-4da6-9055-6229e339c6ba                    -                                    
88e55704-865f-4dcb-82e1-e2d689e0cbd1 10 GiB   faulted                    deleted-88e55704-865f-4dcb-82e1-e2d689e0cbd1                    -                                    
8de85d24-fa7d-4b29-b5ad-7e571ba70686 10 GiB   faulted                    deleted-8de85d24-fa7d-4b29-b5ad-7e571ba70686                    -                                    
8e469a41-3f79-4d9f-9bd5-bb5da2361042 10 GiB   faulted                    deleted-8e469a41-3f79-4d9f-9bd5-bb5da2361042                    -                                    
94b99f0a-d21e-4b0e-a960-ced303136376 10 GiB   faulted                    deleted-94b99f0a-d21e-4b0e-a960-ced303136376                    -                                    
9e0e9cc6-d199-4166-ab4a-bbe178b2535c 10 GiB   faulted                    deleted-9e0e9cc6-d199-4166-ab4a-bbe178b2535c                    -                                    
9f90bb38-d942-496b-8f46-3475bdbf06ec 10 GiB   faulted                    deleted-9f90bb38-d942-496b-8f46-3475bdbf06ec                    -                                    
a126c530-3be1-4a85-a051-0e530613a01a 10 GiB   faulted                    deleted-a126c530-3be1-4a85-a051-0e530613a01a                    -                                    
a4ea7d2f-3ae4-45d4-962b-ddb4a1347aa9 10 GiB   faulted                    deleted-a4ea7d2f-3ae4-45d4-962b-ddb4a1347aa9                    -                                    
a7b7c817-5286-4740-b55e-220714886102 10 GiB   faulted                    deleted-a7b7c817-5286-4740-b55e-220714886102                    -                                    
abdbbfd3-375f-4367-a7c8-ff7950f4f8a5 10 GiB   faulted                    deleted-abdbbfd3-375f-4367-a7c8-ff7950f4f8a5                    -                                    
ada8529a-2f9e-416d-964d-b6d0b8cf569e 10 GiB   faulted                    deleted-ada8529a-2f9e-416d-964d-b6d0b8cf569e                    -                                    
ae5b6a64-4c1d-4357-a460-0fd4a5be1c47 10 GiB   faulted                    deleted-ae5b6a64-4c1d-4357-a460-0fd4a5be1c47                    -                                    
aef005e9-0b3c-4de6-9b3c-42c13cea3c13 10 GiB   faulted                    deleted-aef005e9-0b3c-4de6-9b3c-42c13cea3c13                    -                                    
af0adefd-277d-42bd-a74c-e4618356aa53 10 GiB   faulted                    deleted-af0adefd-277d-42bd-a74c-e4618356aa53                    -                                    
b31b46b1-3894-4010-823b-f8b2e09e51f0 10 GiB   faulted                    deleted-b31b46b1-3894-4010-823b-f8b2e09e51f0                    -                                    
b8aac489-acdd-46c7-8bdd-5d40a90476eb 10 GiB   faulted                    deleted-b8aac489-acdd-46c7-8bdd-5d40a90476eb                    -                                    
bb4df6b7-33ea-413d-b5b0-94f9e8b32f61 10 GiB   faulted                    deleted-bb4df6b7-33ea-413d-b5b0-94f9e8b32f61                    -                                    
bf6df41b-4d0b-40ae-a1be-62d659d0cbf3 10 GiB   faulted                    deleted-bf6df41b-4d0b-40ae-a1be-62d659d0cbf3                    -                                    
c453bb66-0be9-40dc-9d76-7cc1e711f1fb 10 GiB   faulted                    deleted-c453bb66-0be9-40dc-9d76-7cc1e711f1fb                    -                                    
c6c49604-916b-4321-85fe-57c166c90e74 10 GiB   faulted                    deleted-c6c49604-916b-4321-85fe-57c166c90e74                    -                                    
cd393eb6-589b-4b5d-820e-094ab16812d1 10 GiB   faulted                    deleted-cd393eb6-589b-4b5d-820e-094ab16812d1                    -                                    
d2ab411e-0b32-4a3f-a556-ea82614e3b1f 10 GiB   faulted                    deleted-d2ab411e-0b32-4a3f-a556-ea82614e3b1f                    -                                    
d58b6a40-b5d9-4074-8288-664685f20bdb 10 GiB   faulted                    deleted-d58b6a40-b5d9-4074-8288-664685f20bdb                    -                                    
d6f43faf-9914-4035-80b8-a398e71bf5e3 10 GiB   faulted                    deleted-d6f43faf-9914-4035-80b8-a398e71bf5e3                    -                                    
d7d49285-ef79-4da8-ae22-1a38e9ccb8e4 10 GiB   faulted                    deleted-d7d49285-ef79-4da8-ae22-1a38e9ccb8e4                    -                                    
df8a57a0-2d54-4cf1-bae6-5b8bcce7572d 10 GiB   faulted                    deleted-df8a57a0-2d54-4cf1-bae6-5b8bcce7572d                    -                                    
e1b15808-1bc8-434f-b4a6-7d471aaabc3d 10 GiB   faulted                    deleted-e1b15808-1bc8-434f-b4a6-7d471aaabc3d                    -                                    
e4535581-6cfa-4fc1-9a82-a95cd4de91ae 10 GiB   faulted                    deleted-e4535581-6cfa-4fc1-9a82-a95cd4de91ae                    -                                    
e6f80295-f8e2-4128-a2a9-d54dcfeb73ec 10 GiB   faulted                    deleted-e6f80295-f8e2-4128-a2a9-d54dcfeb73ec                    -                                    
eae964c4-2411-4c5f-a9bb-79e821f1bb9e 10 GiB   faulted                    deleted-eae964c4-2411-4c5f-a9bb-79e821f1bb9e                    -                                    
ef955dc9-a0dd-431b-b952-794dcaa8f516 10 GiB   faulted                    deleted-ef955dc9-a0dd-431b-b952-794dcaa8f516                    -                                    
f1aec42b-00d9-45a2-a26f-1e1dc572732a 10 GiB   faulted                    deleted-f1aec42b-00d9-45a2-a26f-1e1dc572732a                    -                                    
f40dc7d7-7371-4430-a719-44d2c47207d9 10 GiB   faulted                    deleted-f40dc7d7-7371-4430-a719-44d2c47207d9                    -                                    
fd3a1e0d-8fca-40e0-9f5b-ee431b816176 10 GiB   faulted                    deleted-fd3a1e0d-8fca-40e0-9f5b-ee431b816176                    -                                    
ffc00ade-1084-4094-b040-1fd6733ef67b 10 GiB   faulted                    deleted-ffc00ade-1084-4094-b040-1fd6733ef67b                    -                            

[qdzlug]

fwiw, I'm seeing 500's with regularity on dogfood when I'm using terraform to delete disks:

│ Error: Unable to delete disk:
│ 
│ API error: DELETE https://oxide.sys.rack2.eng.oxide.computer/v1/disks/0fa8b206-eb65-464a-8160-739108945ee6
│ ----------- RESPONSE -----------
│ Status: 500 Internal
│ Message: Internal Server Error
│ RequestID: c1cd739c-83cd-4526-aaad-c9fd177552c2
│ ------- RESPONSE HEADERS -------
│ Content-Type: [application/json]
│ X-Request-Id: [c1cd739c-83cd-4526-aaad-c9fd177552c2]
│ Date: [Thu, 10 Apr 2025 14:37:11 GMT]
│ Content-Length: [124]
│ 
╵
make: *** [destroy] Error 1

They delete successfully on a re-run.

[jmpesp]

Taking disk 03a8d85b-2b24-4cd0-acfe-4665e621239c as an example, it failed to delete regions during the disk delete saga:

                             saga id | event time               | sub saga | node id                                                             | event type    | data
------------------------------------ | ------------------------ | -------- | ------------------------------------------------------------------- | ------------- | ---
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.312Z |          |  13: start                                                          | started       | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.321Z |          |  13: start                                                          | succeeded     | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.326Z |          |   0: disk_delete.delete_disk_record                                 | started       | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.341Z |          |   0: disk_delete.delete_disk_record                                 | succeeded     | "deleted_disk" => {"block_size":"AdvancedFormat","create_image_id":null,"create_snapshot_id":null,"identity":{"description":"application Data Disk","id":"03a8d85b-2b24-4cd0-acfe-4665e621239c","name":"app-data-7","time_created":"2025-03-29T05:39:01.817232Z","time_deleted":null,"time_modified":"2025-03-29T05:39:01.817232Z"},"pantry_address":null,"project_id":"fe0da422-5c48-4b52-8010-f2fc401f090f","rcgen":1,"runtime_state":{"attach_instance_id":null,"disk_state":"detached","gen":2,"time_updated":"2025-03-29T05:39:02.959190Z"},"size":10737418240,"slot":null,"volume_id":"6114cc9a-cb4c-442d-a612-1e3547942b68"}
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.345Z |          |   1: disk_delete.space_account                                      | started       | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.364Z |          |   1: disk_delete.space_account                                      | succeeded     | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.368Z |          |   2: params_for_volume_delete_subsaga                               | started       | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.373Z |          |   2: params_for_volume_delete_subsaga                               | succeeded     | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.377Z |        0 |   3: subsaga start volume-delete (params_for_volume_delete_subsaga) | started       | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.382Z |        0 |   3: subsaga start volume-delete (params_for_volume_delete_subsaga) | succeeded     | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:35.385Z |        0 |   4: volume_delete.decrease_crucible_resource_count                 | started       | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:36.302Z |        0 |   4: volume_delete.decrease_crucible_resource_count                 | succeeded     | "crucible_resources_to_delete" => {"V3":{"region_snapshots":[],"regions":["0fd2f3c3-d843-40c3-b1cd-f495960a5e06","01503605-c262-4767-969d-438a08b67ae7","10f2138e-fe09-4d7e-a88c-5a95e00be3e8"]}}
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:36.306Z |        0 |   5: volume_delete.delete_crucible_regions                          | started       | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:36.311Z |        0 |   6: volume_delete.delete_crucible_running_snapshots                | started       | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:44:36.316Z |        0 |   6: volume_delete.delete_crucible_running_snapshots                | succeeded     | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.245Z |        0 |   5: volume_delete.delete_crucible_regions                          | failed        | "no_result_1" => {"ActionFailed":{"source_error":"failed to regions_hard_delete: InternalError { internal_message: \"unexpected database error: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): \\\"sql txn\\\" meta={id=15ef98b0 key=/Table/235/2/\\\"a\\\\x14\\u{31a}\\\\xcbLD-\\\\xa6\\\\x12\\\\x1e5G\\\\x94+h\\\"/\\\"\\\\x01P6\\\\x05\\\\xc2bGg\\\\x96\\\\x9dC\\\\x8a\\\\b\\\\xb6z\\\\xe7\\\"/0 pri=0.10666410 epo=0 ts=1744267503.716118483,0 min=1744267499.698186768,0 seq=12} lock=true stat=ABORTED rts=1744267499.698186768,0 wto=false gul=1744267500.198186768,0\" }"}}
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.265Z |        0 |   6: volume_delete.delete_crucible_running_snapshots                | undo_started  | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.270Z |        0 |   6: volume_delete.delete_crucible_running_snapshots                | undo_finished | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.276Z |        0 |   4: volume_delete.decrease_crucible_resource_count                 | undo_started  | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.282Z |        0 |   4: volume_delete.decrease_crucible_resource_count                 | undo_finished | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.287Z |        0 |   3: subsaga start volume-delete (params_for_volume_delete_subsaga) | undo_started  | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.291Z |        0 |   3: subsaga start volume-delete (params_for_volume_delete_subsaga) | undo_finished | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.296Z |          |   2: params_for_volume_delete_subsaga                               | undo_started  | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.301Z |          |   2: params_for_volume_delete_subsaga                               | undo_finished | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.307Z |          |   1: disk_delete.space_account                                      | undo_started  | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.344Z |          |   1: disk_delete.space_account                                      | undo_finished | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.350Z |          |   0: disk_delete.delete_disk_record                                 | undo_started  | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.374Z |          |   0: disk_delete.delete_disk_record                                 | undo_finished | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.379Z |          |  13: start                                                          | undo_started  | 
042b9e59-3dd4-45cb-879c-bf62782033eb | 2025-04-10T06:45:04.386Z |          |  13: start                                                          | undo_finished |

This is almost certainly due to the changes to regions_hard_delete in #7912. That PR changed the function to use a CTE instead of a for loop during an interactive transaction, and I suspect that the CTEs are all contending with each other.

I'm going to try reproducing locally and will test any fixes here.

[smklein]

I wrote up some helpers in https://github.com/oxidecomputer/omicron/tree/main/nexus/db-queries/benches when I tried to reduce contention in the "sled_reservation" query. You might find that harness useful, if you want to build a benchmark for region deletion.

(The real magic is in https://github.com/oxidecomputer/omicron/blob/main/nexus/db-queries/benches/harness/mod.rs , which can give you a lot of info on "where exactly was there contention" each time you iterate through changes)

[jmpesp]

Thanks, I'll check that out :)

[jmpesp]

Note that the resurrection of previously deleted disks is by design, due to issues in the past exactly like this. It allows users to request deletion again instead of creating a zombie disk :)