Prevent region allocation from filling pools #7912

jmpesp · 2025-04-03T20:42:16Z

Recent customer issues have highlighted problems with storage accounting, namely that while there are quotas and reservations for individual Crucible regions, there's nothing set for the whole Crucible dataset. Crucible could end up using the whole disk, or some large fraction of it, such that other users of the same U2 could be starved out.

This commit adds a buffer to each zpool that the Crucible region allocation query will not allocate into. This overhead will be set to 250G initially (see #7875 for reasoning) but could also be modified with omdb.

Part of this commit's changes include using a CTE with regions_hard_delete, which is much more efficient than the previous for loop but has the effect of overwriting size_used for all datasets, which will undo any time this column value was manually set to prevent allocation for particular datasets / pools. Because of this, this commit also adds a no_provision flag for a Crucible dataset: if it is set, then the region allocation query will not allocate into that dataset. This flag can be toggled with omdb.

Part of the upgrade to R14 will include a support procedure to address if the addition of the control plane storage buffer of 250G causes a Crucible dataset to be "overprovisioned", necessitating manually requested region replacement requests to reduce the size allocated for a particular Crucible dataset. This commit adds an omdb command to show all overprovisioned crucible datasets, and changes the region listing command so it can list regions for a particular dataset.

Fixes #3480

Recent customer issues have highlighted problems with storage accounting, namely that while there are quotas and reservations for individual Crucible regions, there's nothing set for the whole Crucible dataset. Crucible _could_ end up using the whole disk, or some large fraction of it, such that other users of the same U2 could be starved out. This commit adds a buffer to each zpool that the Crucible region allocation query will not allocate into. This overhead will be set to 250G initially (see oxidecomputer#7875 for reasoning) but could also be modified with omdb. Part of this commit's changes include using a CTE with `regions_hard_delete`, which is much more efficient than the previous for loop but has the effect of overwriting `size_used` for all datasets, which will undo any time this column value was manually set to prevent allocation for particular datasets / pools. Because of this, this commit also adds a `no_provision` flag for a Crucible dataset: if it is set, then the region allocation query will not allocate into that dataset. This flag can be toggled with omdb. Part of the upgrade to R14 will include a support procedure to address if the addition of the control plane storage buffer of 250G causes a Crucible dataset to be "overprovisioned", necessitating manually requested region replacement requests to reduce the size allocated for a particular Crucible dataset. This commit adds an omdb command to show all overprovisioned crucible datasets, and changes the region listing command so it can list regions for a particular dataset. Fixes oxidecomputer#3480

leftwo · 2025-04-03T21:24:40Z

Recent customer issues have highlighted problems with storage accounting, namely that while there are quotas and reservations for individual Crucible regions, there's nothing set for the whole Crucible dataset. Crucible could end up using the whole disk, or some large fraction of it, such that other users of the same U2 could be starved out.

This is using the existing zpool size numbers, and then taking 250 off of that right?
And this limitation is a policy one up in Nexus itself, not a quota on the crucible dataset?

leftwo · 2025-04-03T21:26:36Z

... but has the effect of overwriting size_used for all datasets, which will undo any time this column value was manually set to prevent allocation for particular datasets / pools. Because of this, this commit also adds a no_provision flag for a Crucible dataset: if it is set, then the region allocation query will not allocate into that dataset. This flag can be toggled with omdb.

If we had manually set the size_used field as a mechanism to prevent allocations, we would have to record that before applying these changes, then come back and toggle the no_provision flag, right?

jmpesp · 2025-04-03T21:26:38Z

Recent customer issues have highlighted problems with storage accounting, namely that while there are quotas and reservations for individual Crucible regions, there's nothing set for the whole Crucible dataset. Crucible could end up using the whole disk, or some large fraction of it, such that other users of the same U2 could be starved out.

This is using the existing zpool size numbers, and then taking 250 off of that right? And this limitation is a policy one up in Nexus itself, not a quota on the crucible dataset?

Yes, and yes :)

jmpesp · 2025-04-03T21:26:52Z

... but has the effect of overwriting size_used for all datasets, which will undo any time this column value was manually set to prevent allocation for particular datasets / pools. Because of this, this commit also adds a no_provision flag for a Crucible dataset: if it is set, then the region allocation query will not allocate into that dataset. This flag can be toggled with omdb.

If we had manually set the size_used field as a mechanism to prevent allocations, we would have to record that before applying these changes, then come back and toggle the no_provision flag, right?

Correct

leftwo · 2025-04-03T21:30:39Z

This will prevent crucible from using up space (which is a good thing) and I know urgency for R14 too.

This won't prevent some other service from using up all the space in the pool, even if we keep Crucible in check.
So whatever eventual solution we come up with to prevent full disks will need to include preventing the other services from taking more than their fair share of the pool, is that correct?

smklein · 2025-04-03T21:35:43Z

This won't prevent some other service from using up all the space in the pool, even if we keep Crucible in check. So whatever eventual solution we come up with to prevent full disks will need to include preventing the other services from taking more than their fair share of the pool, is that correct?

This is true, but we planned to punt on quotas and reservations because assigning them can fail for deployed racks, and we need a fall-back plan. The "failure mode" introduced in this PR is "we don't overprovision further", which is a good thing, and won't need support staff to remediate.

smklein · 2025-04-03T21:36:22Z

But anyway, yes, definitely do want quotas / reservations to help limit this issue from crossing abstraction boundaries

leftwo · 2025-04-03T21:41:56Z

But anyway, yes, definitely do want quotas / reservations to help limit this issue from crossing abstraction boundaries

Just wanted to be sure we don't forget to do the additional work, and be sure the casual reader did not think that all the problems are now solved. Things will be better with this, and will continue getting better :)

smklein

Looks good, nice job getting the non-provisionable + buffer merged together. And thanks for the tests!

nexus/db-queries/Cargo.toml

nexus/db-queries/src/db/datastore/crucible_dataset.rs

nexus/db-queries/src/db/raw_query_builder.rs

nexus/src/app/background/tasks/physical_disk_adoption.rs

dev-tools/omdb/src/bin/omdb/db.rs

nexus/db-model/src/zpool.rs

smklein · 2025-04-04T17:04:53Z

dev-tools/omdb/src/bin/omdb/db.rs

+    Ok(())
+}
+
+async fn cmd_crucible_dataset_show_overprovisioned(


This is an omdb command, so my bar for testing there is lower than it would be otherwise, but have you tested this API? (Even manually)

(I'm bringing this scrutiny because this command seems really useful, actually, and frankly like something we might want to pull into Nexus in the future)

I tested it by filling up all the space I could, then increasing the storage_buffer:

EVT22200005 # omdb-7912 db crucible-dataset show-overprovisioned 2> /dev/null ID |SIZE_USED |NO_PROVISION |POOL_ID |CONTROL_PLANE_STORAGE_BUFFER |POOL_TOTAL_SIZE -------------------------------------+--------------+-------------+-------------------------------------+-----------------------------+---------------- 12e4105b-3dde-40ff-9f12-5ce5d57f8b4f |2925946470400 |false |02d72adc-f403-4eef-bede-3a2a860a22a3 |375809638400 |3195455668224

leftwo

Still looking, but I wanted these comments posted now.

dev-tools/omdb/src/bin/omdb/db.rs

leftwo · 2025-04-04T17:36:43Z

dev-tools/omdb/src/bin/omdb/db.rs

+    Ok(())
+}
+
+async fn cmd_crucible_dataset_show_overprovisioned(


I tested it by filling up all the space I could, then increasing the storage_buffer:

EVT22200005 # omdb-7912 db crucible-dataset show-overprovisioned 2> /dev/null ID |SIZE_USED |NO_PROVISION |POOL_ID |CONTROL_PLANE_STORAGE_BUFFER |POOL_TOTAL_SIZE -------------------------------------+--------------+-------------+-------------------------------------+-----------------------------+---------------- 12e4105b-3dde-40ff-9f12-5ce5d57f8b4f |2925946470400 |false |02d72adc-f403-4eef-bede-3a2a860a22a3 |375809638400 |3195455668224

leftwo · 2025-04-04T17:49:28Z

Could we have an omdb command somewhere in here that does the TOTAL_SIZE - CONTROL_PLANE_STORAGE_BUFFER math for me, so I can see how much each dataset has left?

leftwo

Just a few questions and comment comments

dev-tools/omdb/src/bin/omdb/db.rs

nexus/db-model/src/zpool.rs

nexus/db-queries/src/db/datastore/zpool.rs

nexus/tests/integration_tests/disks.rs

jmpesp · 2025-04-07T20:46:58Z

Could we have an omdb command somewhere in here that does the TOTAL_SIZE - CONTROL_PLANE_STORAGE_BUFFER math for me, so I can see how much each dataset has left?

Done in 5de0a56

leftwo

Here is the omdb with usage output:

EVT22200005 # omdb-7912 db crucible-dataset list 2> /dev/null
ID                                   |TIME_DELETED |POOL_ID                              |ADDRESS                        |SIZE_USED    |NO_PROVISION |CONTROL_PLANE_STORAGE_BUFFER |POOL_TOTAL_SIZE |SIZE_LEFT     
-------------------------------------+-------------+-------------------------------------+-------------------------------+-------------+-------------+-----------------------------+----------------+--------------
12e4105b-3dde-40ff-9f12-5ce5d57f8b4f |             |02d72adc-f403-4eef-bede-3a2a860a22a3 |[fd00:1122:3344:101::12]:32345 |132875550720 |false        |375809638400                 |3195455668224   |2686770479104 
1e9c36ea-9b9e-4762-b729-01124dc3d56c |             |83cb6813-d89a-4996-bae8-43609c63b1dd |[fd00:1122:3344:101::13]:32345 |71135395840  |false        |268435456000                 |3195455668224   |2855884816384 
238884a7-d934-4325-8d38-d6f5be067b3e |             |fd1796f2-f671-4d60-99dc-e1194024bebb |[fd00:1122:3344:101::18]:32345 |230854492160 |false        |268435456000                 |3195455668224   |2696165720064 
5b1defd7-249c-452b-9d4c-f3464195c659 |             |571bc087-1c0a-4378-ab37-7a42633b05ae |[fd00:1122:3344:101::16]:32345 |175825223680 |false        |268435456000                 |3195455668224   |2751194988544 
8cf5a521-a139-4596-bd3f-079c544381f1 |             |0fd4c7d7-2547-4769-a55f-54c1284f24e4 |[fd00:1122:3344:101::15]:32345 |71135395840  |false        |268435456000                 |3195455668224   |2855884816384 
9d233755-b283-4b00-9f6c-1ee6492ad4d7 |             |0c4db663-9601-45f3-b6b3-bde209e8f7d7 |[fd00:1122:3344:101::17]:32345 |147639500800 |false        |268435456000                 |3195455668224   |2779380711424 
daed3ed0-74ae-4acc-94b5-b8e4d26a97f9 |             |fb3ea440-ff27-43c4-bf37-a3f1d60ae265 |[fd00:1122:3344:101::14]:32345 |49660559360  |false        |268435456000                 |3195455668224   |2877359652864

jmpesp mentioned this pull request Apr 3, 2025

keep the system from running out of disk space #7875

Open

18 tasks

smklein reviewed Apr 3, 2025

View reviewed changes

morlandi7 added this to the 14 milestone Apr 3, 2025

jmpesp added 5 commits April 4, 2025 12:52

Merge branch 'main' into do_not_provision_flag_for_dataset

9c41b55

separate omdb feature not necessary

77c0cf9

remove my dumb

35cffce

check for time_deleted

46cc30b

use ByteCount, and define a const for buffer

cc38e26

smklein reviewed Apr 4, 2025

View reviewed changes

smklein approved these changes Apr 4, 2025

View reviewed changes

leftwo reviewed Apr 4, 2025

View reviewed changes

leftwo approved these changes Apr 7, 2025

View reviewed changes

dev-tools/omdb/src/bin/omdb/db.rs Show resolved Hide resolved

nexus/db-model/src/zpool.rs Outdated Show resolved Hide resolved

nexus/db-queries/src/db/datastore/zpool.rs Show resolved Hide resolved

nexus/tests/integration_tests/disks.rs Show resolved Hide resolved

morlandi7 assigned jmpesp Apr 7, 2025

jmpesp added 3 commits April 7, 2025 20:39

require destructive token for cmd_db_zpool_set_storage_buffer

78abeac

list -i, plus adjust buffer for CI

ac33566

add size_left column, refactor to get_crucible_dataset_rows

5de0a56

jmpesp added 3 commits April 7, 2025 21:06

change pool to zpool

ce059ad

log message when changing control plane storage buffer

2644657

expand comment for control_plane_storage_buffer

0bbf9a7

leftwo approved these changes Apr 7, 2025

View reviewed changes

jmpesp added 2 commits April 8, 2025 00:48

expectorate

bb680d9

no more unwraps

5e0d84f

jmpesp enabled auto-merge (squash) April 8, 2025 01:05

jmpesp added 4 commits April 8, 2025 13:15

try full path to omdb

e295a79

pool -> zpool

24e5693

wait for handoff

651ddb1

wait for all zpools to show up in db

743a45c

jmpesp merged commit c496683 into oxidecomputer:main Apr 9, 2025
16 checks passed

jmpesp deleted the do_not_provision_flag_for_dataset branch April 9, 2025 18:38

This was referenced Apr 9, 2025

crucible zone running out of space earlier than the control plane "thinks" #4234

Closed

Slow disk deletion and resurrection of previously deleted disks #7952

Closed

Prevent region allocation from filling pools #7912

Prevent region allocation from filling pools #7912

Uh oh!

Conversation

jmpesp commented Apr 3, 2025

Uh oh!

leftwo commented Apr 3, 2025

Uh oh!

leftwo commented Apr 3, 2025

Uh oh!

jmpesp commented Apr 3, 2025

Uh oh!

jmpesp commented Apr 3, 2025

Uh oh!

leftwo commented Apr 3, 2025

Uh oh!

smklein commented Apr 3, 2025

Uh oh!

smklein commented Apr 3, 2025

Uh oh!

leftwo commented Apr 3, 2025

Uh oh!

smklein left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smklein Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leftwo Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

leftwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leftwo Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

leftwo commented Apr 4, 2025

Uh oh!

leftwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmpesp commented Apr 7, 2025

Uh oh!

leftwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smklein Apr 4, 2025 •

edited

Loading