fm sitreps should be able to request a support bundle collection

[hawkw]

We would like a way for the fault management subsystem to be able to request that, for a particular FM case, a support bundle should be collected.

#9552 adds a mechanism for the fault management subsystem to request alerts, by adding a set of alert requests to cases in a sitrep, and a fm_rendezvous background task which then attempts to ensure that all alerts requested by cases in the current sitrep exist in the alerts table. We should do a similar thing for support bundle requests.

In particular:

• Cases should have a set of support bundle requests, analogous to the alert requests in [nexus] FM alert requests and rendezvous task #9552.
• The support bundle requests should be able to contain whatever ways of deciding what information should/should not be included in the bundle currently exists in the support bundle APIs --- a diagnosis engine should be able to request targeted support bundles, or at least as much of a targeted support bundle the support bundle system can currently do. If we make support bundles more able to be targeted in the future, we should be able to make the FM requests express whatever additional ways of targeting things there are.¹
• Something (probably the fm_rendezvous task, but perhaps not if that doesn't make sense due to a reason I have failed to anticipate...) should then make sure that all support bundles requested by cases in the current sitrep will be/are being/have been requested.
  • This must be idempotent; I expect that this will mean that (like alerts) the primary keys for whatever database records you are creating will come from the sitrep's requests rather than being made up by the rendezvous thingy
• There should maybe be a way to say "please list for me the support bundles in the latest version of a case"?
  • Although we don't need this part right away, it will probably be useful to have if we are going to then be able to have Active Problems that say "here are the support bundle(s) that are related to me"?
• I think there should also be a way for the support bundle to record which case requested it and, ideally, why it was requested
  • like a comment field or something
  • this is, at least, probably useful for OMDB so that you can figure out why a support bundle was collected
  • i don't remember if Sean already added this to support bundles or not?
• Also go do all the OMDB and such to make it nice

Footnotes

1. I realize this part doesn't make a ton of sense, I haven't actually looked at the interfaces to support bundle collection in a little while. Perhaps @smklein can correct my language here? ↩

[hawkw]

Oh, one other complicating factor I just realized: this will need to interact with #9662, which adds automatic deletion of support bundles if there are too many. In particular, we will need to make sure that if a sitrep requests a support bundle, the support bundle is collected, and then later, the support bundle is deleted, we don't go and immediately re-create that support bundle. Perhaps that means we have to change cleanup_bundle_from_database to tombstone the deleted bundle rather than hard-deleting it, so that the primary key still prevents us from re-creating the bundle? Probably talk to @smklein about this.

[smklein]

• The support bundle requests should be able to contain whatever ways of deciding what information should/should not be included in the bundle currently exists in the support bundle APIs --- a diagnosis engine should be able to request targeted support bundles, or at least as much of a targeted support bundle the support bundle system can currently do. If we make support bundles more able to be targeted in the future, we should be able to make the FM requests express whatever additional ways of targeting things there are.

We have a struct called BundleRequest that's the front-end for bundle collection. right now, the support bundle background task is using ``BundleRequest::default()`, but we could (should?) pull that out so we can request more specific data here, as appropriate to the case.

See also: BundleDataSelection, which specifies categories of information to be included within the bundle. It's a list that's meant to be expanded!

• Something (probably the fm_rendezvous task, but perhaps not if that doesn't make sense due to a reason I have failed to anticipate...) should then make sure that all support bundles requested by cases in the current sitrep will be/are being/have been requested.

That makes sense - also be aware of / cautious of #9662 - it is very possible that an old case might collect a bundle, but if it stays around for long enough, that bundle might get deleted.

(It's also possible I need to modify 9662 to handle this, and avoid auto-deleting a bundle if it's still referenced by a case? More room for discussion here)

• This must be idempotent; I expect that this will mean that (like alerts) the primary keys for whatever database records you are creating will come from the sitrep's requests rather than being made up by the rendezvous thingy
• There should maybe be a way to say "please list for me the support bundles in the latest version of a case"?

Bundles have UUID from the moment their db records are created, before they are actually collected. This should help?

(This is what the front-end API does for bundle collection does - it makes a new database row for a bundle, with a UUID, but doesn't actually start collection until the background task runs)

• I think there should also be a way for the support bundle to record which case requested it and, ideally, why it was requested

There is a reason_for_creation string on Support Bundles, so, it's unstructured but it does exist.

Could add a more strongly-typed (or indexable) type to bundles for them to reference cases too.

[hawkw]

• The support bundle requests should be able to contain whatever ways of deciding what information should/should not be included in the bundle currently exists in the support bundle APIs --- a diagnosis engine should be able to request targeted support bundles, or at least as much of a targeted support bundle the support bundle system can currently do. If we make support bundles more able to be targeted in the future, we should be able to make the FM requests express whatever additional ways of targeting things there are.

We have a struct called BundleRequest that's the front-end for bundle collection. right now, the support bundle background task is using ``BundleRequest::default()`, but we could (should?) pull that out so we can request more specific data here, as appropriate to the case.

See also: BundleDataSelection, which specifies categories of information to be included within the bundle. It's a list that's meant to be expanded!

Yeah. I think @mergeconflict will need to figure out a nice way to stuff all that stuff in CRDB for the FM bundle request table. Have fun!

• Something (probably the fm_rendezvous task, but perhaps not if that doesn't make sense due to a reason I have failed to anticipate...) should then make sure that all support bundles requested by cases in the current sitrep will be/are being/have been requested.

That makes sense - also be aware of / cautious of #9662 - it is very possible that an old case might collect a bundle, but if it stays around for long enough, that bundle might get deleted.

(It's also possible I need to modify 9662 to handle this, and avoid auto-deleting a bundle if it's still referenced by a case? More room for discussion here)

Yeah, I also left a comment about making sure that we don't re-create deleted bundles (#10062 (comment)), I had suggested tombstoning the db records when they're deleted, but I think your proposal about having a nullable FM case reference in the bundle table, and just not delete things referenced by cases that still exist in the current sitrep is also good. Especially because we may want to add a structured reference to the case ID for other reasons anyway. Whatever you think is better 🤷‍♀️

[smklein]

One other thing to note, regarding the "support bundle collection via FM":

Right now, bundles can be requested/deleted through the public-facing API. So, if FM requested bundles too, they could be deleted through the external API. We'll want to deal with that: either allow it, and make sure FM can cope, or prevent operators from deleting "requested-by-FM" bundles.

[hawkw]

One other thing to note, regarding the "support bundle collection via FM":

Right now, bundles can be requested/deleted through the public-facing API. So, if FM requested bundles too, they could be deleted through the external API. We'll want to deal with that: either allow it, and make sure FM can cope, or prevent operators from deleting "requested-by-FM" bundles.

Yeah, that might be a case where tombstoning makes sense --- I think we should be able to respect the operator's intent to delete the thing if we soft-deleted it in a way where we would stop the reconciler from going and re-creating it...

[hawkw]

(and then i guess that maybe deleting the case could go and hard-delete any tombstoned support bundles associated with it, so that we can not store them in CRDB until The End of Time? perhaps we could have the behavior of the current support bundle deletion be a hard-delete if the fm_case column is NULL and a soft delete if it isn't, or something?)

[rmustacc]

One other thing to note, regarding the "support bundle collection via FM":

Right now, bundles can be requested/deleted through the public-facing API. So, if FM requested bundles too, they could be deleted through the external API. We'll want to deal with that: either allow it, and make sure FM can cope, or prevent operators from deleting "requested-by-FM" bundles.

FWIW, my default is that the operator should be able to delete it. I would expect the flow to be that a a targeted support bundle is generated in response to something that includes a lot of the additional information that we might want like say vendor-specific NVMe logs for a disk failure, and that they would send it to us. Once they have that there's no reason they shouldn't be able to delete it and reclaim that space.

[hawkw]

One other thing to note, regarding the "support bundle collection via FM":
Right now, bundles can be requested/deleted through the public-facing API. So, if FM requested bundles too, they could be deleted through the external API. We'll want to deal with that: either allow it, and make sure FM can cope, or prevent operators from deleting "requested-by-FM" bundles.

FWIW, my default is that the operator should be able to delete it. I would expect the flow to be that a a targeted support bundle is generated in response to something that includes a lot of the additional information that we might want like say vendor-specific NVMe logs for a disk failure, and that they would send it to us. Once they have that there's no reason they shouldn't be able to delete it and reclaim that space.

Yeah, I agree --- I would really rather not have automatically collected support bundles remain un-deletable as long as the FM case is open. I think we should probably tombstone them in the DB so that we don't keep trying to collect a new one, and clean up the tombstones when the case is deleted.

[hawkw]

A near-term product goal here is that when we see a Hubris task fault ereport (added in oxidecomputer/hubris#2341) from an SP, we ought to be able to go and make a support bundle with any dumps from that SP.

I imagine that we would probably do this by just making a case for the first fault ereport, and then waiting for some denouncing time window, so that subsequent fault ereports from the same SP get grouped together into the case, and then eventually making a support bundle and closing the case. The idea here is that we don't want to spew out a whole bunch of mostly-duplicative support bundles when a task is crash looping.

(I suspect that this kind of debouncing/coalescing pattern will be pretty widespread in control plane FM analysis. We might even consider a conventional rule like "cases should generally not be closed in the same sitrep in which they were opened" to try and allow time for additional events to be observed/grouped together before making a diagnosis...)