test failed in CI: `test_multiple_disks_multiple_snapshots_order_2`

[david-crespo]

This test failed on a CI run on #7067 (which only bumps console assets, changes no actual code):

https://github.com/oxidecomputer/omicron/runs/32966066009

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JCMHTK9FB1PS5VD8TJ2R1YTS/BdASfcE6X0ehlDrcp1UYmDzTDStjPGCGNn105IOfrvuyHqMG/01JCMHVMAJ2QQ624KK2W4TEJQJ#S6606

Excerpt from the log showing the failure:

6625    2024-11-14T06:05:43.589Z	thread 'integration_tests::volume_management::test_multiple_disks_multiple_snapshots_order_2' panicked at nexus/tests/integration_tests/volume_management.rs:719:5:
6626    2024-11-14T06:05:43.589Z	assertion failed: disk_test.crucible_resources_deleted().await

[jmpesp]

The part of the test that's failing is the check at the very end that asserts all Crucible resources have been cleaned up by Nexus:

    // Delete the first snapshot
    NexusRequest::object_delete(client, &get_snapshot_url("first-snapshot"))
        .authn_as(AuthnMode::PrivilegedUser)
        .execute()
        .await
        .expect("failed to delete snapshot");
    
    // Assert everything was cleaned up
    assert!(disk_test.crucible_resources_deleted().await);

The disk_test function that's called checks that the simulated Crucible agent was told to delete all resources:

    pub async fn crucible_resources_deleted(&self) -> bool {
        for (sled_id, state) in &self.sleds {
            for zpool in &state.zpools {
                for dataset in &zpool.datasets {
                    let crucible = self
                        .get_sled(*sled_id)
                        .get_crucible_dataset(zpool.id, dataset.id)
                        .await;
                    if !crucible.is_empty().await {
                        return false;
                    }
                } 
            }
        }
    
        true
    }

It's weird that this is intermittently failing: the disk and snapshot delete calls invoke the corresponding sagas, but the endpoint will wait for them to be complete before returning. Looking at the logs shows that the saga completed:

16:27:10.561Z INFO 9a0aa4f3-583c-44d1-9cb9-7ccc7e5c46a8 (dropshot_external): request completed
    latency_us = 183744
    local_addr = 127.0.0.1:46711
    method = DELETE
    remote_addr = 127.0.0.1:34696
    req_id = 8b0ecd79-3732-4be6-b607-9db3345e74bf
    response_code = 204
    uri = /v1/snapshots/first-snapshot?project=springfield-squidport-disks

and right afterward, the is_empty check logged that a region still existed:

16:27:10.562Z INFO test_multiple_disks_multiple_snapshots_order_2 (omicron_sled_agent::sim::Server): is_empty non_destroyed_regions 1 snapshots 0 running_snapshots 0
    end_port = 1200
    kind = storage
    server = b6d65341-167c-41df-9b5c-41cded99c229
    sled_id = b6d65341-167c-41df-9b5c-41cded99c229
    start_port = 1100

[bnaecker]

Just hit this on #6966

[smklein]

Saw this in https://github.com/oxidecomputer/omicron/pull/6782/checks?check_run_id=33235333336

[jgallagher]

One more https://github.com/oxidecomputer/omicron/pull/7103/checks?check_run_id=33271058412

[jmpesp]

I am unable to reproduce this on top of the changes in #7046, I suspect that the underlying flake in this test was fixed in that PR:

Testing also revealed that there were scenarios where find_deleted_volume_regions would return volumes for hard-deletion prematurely. The function now returns a struct instead of a list of tuples, and in that struct, regions freed for deletion are now distinct from volumes freed for deletion. Volumes are now only returned for hard-deletion when all associated read/write regions have been (or are going to be) deleted.

[karencfv]

👋 https://github.com/oxidecomputer/omicron/runs/34636067643

[jgallagher]

👋 https://github.com/oxidecomputer/omicron/runs/34636067643

I think this doesn't have the changes in #7046, which just landed on main yesterday, but please double check me!