Mark orphaned sagas as abandoned

[karencfv]

Due to a few bugs in Steno, we ended up in dogfood (an potentially customer racks) with a handful of sagas in an "unwinding" state whose current_sec is a Nexus instance that no longer exists.

Initially, we assumed we could mark these sagas as abandoned during the saga recovery process, but later found that the initial orphaned sagas were still there. This was because during saga recovery, each Nexus only attempts to recover sagas that are assigned to itself. This means that these sagas whose current_sec no longer exists, will never go through the saga recovery process, and therefore are never marked as abandoned.

An idea was floated around to somehow reassign these sagas to a Nexus so they can be processed during the saga recovery task and be marked as abandoned there. I will look into this and see how feasible it is.

[karencfv]

Bad idea 1

During re-assignment assign these orphaned sagas to a live Nexus so that during the saga recovery task they get abandoned.

This is a bad idea because only sagas from the same generation as the target Nexus are allowed to be transferred to it. Orphaned sagas could be assigned to a Nexus of any generation.

Bad idea 2

Have a background task that abandons any saga in Unwinding state who's current_sec is a Nexus that no longer exists.

This is a bad idea because of possible race conditions with Nexuses in the process of being expunged. Sagas are only reassigned when their Nexus no longer exists.

This option also adds a lot of complexity.

Plausible(?) idea

Based on the target blueprint, decide whether a saga is re-assignable or not.

Abandon a saga iff its current_sec is in state BlueprintZoneDisposition::Expunged { ready_for_cleanup: true } and its generation is not the same as the Nexuses with state BlueprintZoneDisposition::InService.

Deciding whether a saga is re-assignable, and abandoning if not would take place during blueprint execution ( realize_blueprint())

Also wondering, if a saga is orphaned does it matter what state it's in? Or should I only abandon those in Unwinding state? 🤔

[smklein]

To be super pedantic about the lifetime of the en-route-to-dead Nexus, they goes through the following lifetime phases:

1. In the blueprint, it is marked expunged, as a policy (but it might still be alive)
2. In the blueprint, is marked expunged, and marked "ready_for_cleanup: true", meaning that it is confirmed "not running"
3. In the blueprint, the zone no longer exists. It might have been fully cleaned up (pruned from the blueprint), but we could still have a saga referencing this no-longer-alive Nexus if it was expunged a long time ago.

Here's the thing, I think the "phase 3" here doesn't exist today, because I don't think we're pruning yet? but I assume we will at some point, because this is basically necessary garbage collection to do eventually. (see: #5552)

I hear you on the point that "we also need to check for the generation number" not matching the in-service Nexuses -- I think that's right, that makes sense to me as the decision point to "orphan" instead of "adopt" -- but wanted to make sure we're on the same point about what counts as a "sufficiently dead Nexus".

So, in summary, I think the criteria should be:

• We read the target blueprint into memory as B
• For any unfinished saga S(N), assigned to a nexus N...
• ... if N has a generation < the currently running Nexus AND N is expunged + ready_for_cleanup in B
• ... then the saga S(N) should be marked abandoned

AND ALSO to make this work, we need to assert that "before a zone N is removed from B, it must not have any unfinished sagas" (which, tbh, is sorta re-stating what #5552 asserts, but it's an assertion we're depending on here)

And to be even more explicit: If this "saga abandoner" happens to see an unfinished saga S(N) assigned to a nexus N which is not in target blueprint B, we should leave it alone. It's tempting to view this as "maybe the nexus N is super old and got pruned?", but unfortunately, it's possible that our "target blueprint" became out-of-date, and a new Nexus got deployed. And if that nexus isn't in our blueprint, we don't have a generation number to help us distinguish these cases.

[karencfv]

Thanks for taking a look @smklein

Here's the thing, I think the "phase 3" here doesn't exist today, because I don't think we're pruning yet? but I assume we will at some point, because this is basically necessary garbage collection to do eventually. (see: #5552)

Yeah, my initial idea was to have sagas whose nexus didn't exist in the blueprint be marked as orphaned, but I double checked on dogfood to see if old nexuses were pruned, and they definitely were not:

[BRM42220051]root@oxz_switch0:~# omdb nexus blueprints show target | grep -i nexus
<...>
    nexus               c053e7af-80de-4a43-89d1-232776290a02   artifact: version 18.0.0-0.ci+git3b3f937d05f   expunged ✓     fd00:1122:3344:109::40
    nexus               c54479be-b734-4e3d-abc0-36145e5fadc1   artifact: version 19.0.0-0.ci+gitd3ef1db023b   expunged ✓     fd00:1122:3344:109::59
    nexus               c880d4a7-78da-46f3-8df1-e01ca137899b   artifact: version 20.0.0-0.ci+git3dc79fe53b1   expunged ✓     fd00:1122:3344:109::6e
    nexus               e4df8bba-edfa-49ba-a903-c74148628e2d   artifact: version 20.0.0-0.ci+gita3fee0ec59c   expunged ✓     fd00:1122:3344:109::75
    nexus               f2aa12f4-7c8e-4516-b836-debc60fffdcd   artifact: version 18.0.0-0.ci+git0ee7d730282   expunged ✓     fd00:1122:3344:109::52
    nexus               fa0ff897-c9d2-4e10-ba73-e5383f4426e8   artifact: version 17.0.0-0.ci+gitf83a43dbb42   expunged ✓     fd00:1122:3344:109::39
    oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crypt/zone/oxz_nexus_03e0589d-96df-4fdc-8e37-96d9509bf53f             d80d3207-3f80-43b2-a15d-c968632667eb   expunged      none      none          off        
    oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crypt/zone/oxz_nexus_081d0bf1-2b82-40f2-9167-64a1ed3cd5a0             555d61cb-5937-4bc0-a478-8f39dec920cc   expunged      none      none          off        
    oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crypt/zone/oxz_nexus_0ab489a1-758f-4210-b351-277679534806             9d4fdbcf-4fae-4acb-ade2-0a52f5e9d51e   expunged      none      none          off        
    oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crypt/zone/oxz_nexus_0c4838be-48cf-4077-92a5-91e0432f5846             cec7a7ad-cd2a-482b-b138-db1138159f13   expunged      none      none          off        
    oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crypt/zone/oxz_nexus_344e9423-5b88-4605-a694-ff5117b3b552             e7153d3d-07df-4fd8-9fe5-beb611ef923b   expunged      none      none          off    
<...>

That's when I decided to rely on generation number instead.

Thanks for the added context. I'll make sure to include all those safeguards

[karencfv]

And to be even more explicit: If this "saga abandoner" happens to see an unfinished saga S(N) assigned to a nexus N which is not in target blueprint B, we should leave it alone. It's tempting to view this as "maybe the nexus N is super old and got pruned?", but unfortunately, it's possible that our "target blueprint" became out-of-date, and a new Nexus got deployed. And if that nexus isn't in our blueprint, we don't have a generation number to help us distinguish these cases.

But wouldn't this new Nexus that got deployed be of a newer generation than the ones in the blueprint?

[jgallagher]

Here's the thing, I think the "phase 3" here doesn't exist today, because I don't think we're pruning yet? but I assume we will at some point, because this is basically necessary garbage collection to do eventually. (see: #5552)

This is correct. I started on this project in #9730, but it got sidelined to do all the networking cleanup / reconcilers work in the wake of R18. We do still need to do this eventually; the pruner is responsible for knowing all the reasons an expunged zone might not yet be pruneable, which also means anything that would prevent pruning an expunged zone needs to have a way to know for sure that such prevention is no longer necessary.

And to be even more explicit: If this "saga abandoner" happens to see an unfinished saga S(N) assigned to a nexus N which is not in target blueprint B, we should leave it alone. It's tempting to view this as "maybe the nexus N is super old and got pruned?", but unfortunately, it's possible that our "target blueprint" became out-of-date, and a new Nexus got deployed. And if that nexus isn't in our blueprint, we don't have a generation number to help us distinguish these cases.

But wouldn't this new Nexus that got deployed be of a newer generation than the ones in the blueprint?

I think one way to get to the case Sean's describing is:

• Blueprint B1 has Nexuses A, B, C at some generation N
• The disk or sled hosting Nexus C is expunged
• We generate blueprint B2 that expunges Nexus C and adds Nexus D. This will be at the same generation N as Nexuses A and B because it's a replacement on the same version, not an update.
• Nexus B executes B2 which deploys Nexus D
• Nexus D gets some assigned saga
• Due to unfortunate timing, Nexus A tries to execute blueprint B1. Its saga abandoner will see the sagas assigned to Nexus D, but Nexus D isn't in blueprint B1 at all. It must leave these alone.

[smklein]

100%, @jgallagher , that's the case I was thinking of, where:

Due to unfortunate timing, Nexus A tries to execute blueprint B1. Its saga abandoner will see the sagas assigned to Nexus D, but Nexus D isn't in blueprint B1 at all. It must leave these alone.

When Nexus A is trying to execute this blueprint, it does not see "Nexus D" in the blueprint, so it does not know the generation number. Without that number, the "saga abandoner" cannot identify if it's a newer Nexus, or if it's an older Nexus, or if it's the same generation as existing Nexuses.

I think, however, this is okay. If we really try to uphold the property that "a Nexus should not be removed from the blueprint until it has no unfinished sagas", then these not-in-blueprint Nexuses which own sagas should be either "newer" or "the same generation as the current set". And in that case, if we wait for the abandoner to take another lap, re-load a newer blueprint, and try again, it should eventually see a blueprint that contains that Nexus - which means we can make a definitive choice about "should we abandon or re-assign those sagas".