Tracking issue for saga unwind safety

[smklein]

This issue tracks testing that if we fail a particular action in a saga, we can safely unwind (performing undo actions) and leave the system in a "clean" state.

• Instance create: [nexus] Add tests for instance_create unwind safety #1843
• Instance delete
• Disk create: [nexus] Make 'disk_create' saga unwind safe, add tests for it #1835
• Disk Delete
• Snapshot Create: [nexus] Make snapshot-create saga unwind-safe #2093
• Snapshot Delete
• Volume Delete
• Volume Remove ROP
• Project Creation: [nexus] Make project creation unwind safe, add tests #2087
• Project Deletion
• VPC Creation: [nexus] Make the VPC creation saga unwind-safe, add tests #2086
• VPC Deletion
• Instance Migration

If other sagas are added without unwind safety tests, they should be added to this list.

Related to #1799, #2094

[askfongjojo]

We've seen at least two incidents of failed instance_create unwind on rack2 resulting in panic. One was #3212 and another was caused by a networking issue which prevented the vnic from being cleaned up:

BRM42220031 # cat /var/svc/log/oxide-sled-agent\:default.log | looker -l error
WARNING: Failed to delete OPTE port overlay VNIC while dropping port. The VNIC will not be cleaned up properly, and the xde device itself will not be deleted. Both the VNIC and the xde device must be deleted out of band, and it will not be possible to recreate the xde device until then. Error: DeleteVnicError { name: "vopte4", err: CommandFailure(CommandFailureInfo { command: "/usr/sbin/dladm delete-vnic vopte4", status: ExitStatus(unix_wait_status(256)), stdout: "", stderr: "dladm: vnic deletion failed: link busy\n" }) }
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: AdmError { op: Uninstall, zone: "oxz_propolis-server_2df3f0da-9b07-47bd-85de-a89fc640383f", err: CommandOutput(CommandOutputError("exit code 1\nstdout:\n\nstderr:\nzoneadm: zone 'oxz_propolis-server_2df3f0da-9b07-47bd-85de-a89fc640383f': uninstall operation is invalid for shutting_down zones.")) }', sled-agent/src/instance.rs:476:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ May 26 17:59:33 Stopping because all processes in service exited. ]
[ May 26 17:59:33 Executing stop method (:kill). ]
[ May 26 17:59:33 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]