Cleanup new excludeOnUpgrade Saved Object API

[joshdover]

As a follow up to #106534, there are several tasks we'd like to complete to "lock down" the API and improve the test coverage:

• Cleanup
  • Move integration test to actions plugin + add simple integration test for core + shrink test payload (Add ability for types to define an excludeOnUpgrade hook #106534 (comment) Add ability for types to define an excludeOnUpgrade hook #106534 (comment) Add ability for types to define an excludeOnUpgrade hook #106534 (comment))
  • Remove the unusedTypesQuery and move implementations to plugins
• Robustness Improvements
  • Handle case where other objects have references to excluded objects
  • Switch to KQL filter w/ field re-writing to prevent plugins from excluding objects of other types
  • Add documentation on when to use this new API vs. regular migrations
  • Generate task & action types from task manager registry instead of using hard-coded ones Update getOldestActionTask and excludeOnUpgrade hook to read action types from task registry #107012

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[joshdover]

@elastic/kibana-alerting-services Would it make sense for you all to take the last item in this here, "Generate task & action types from task manager registry instead of using hard-coded ones"? I imagine it's a pretty small change and you all know better than I about how to do it in the most robust way.

[mikecote]

@joshdover, yup, that makes sense to me 👍 we can prioritize a separate issue on our side. Let me know if you want me to create it.

[joshdover]

@mikecote Created a new issue here: #107012

[joshdover]

Alternatively, we may be able to remove this API from master/8.0. Reason being that our recommended upgrade path is to go through 7.16 first, use the Upgrade Assistant, and then upgrade to 8.0. For users who follow the guidance, this should result in these old tasks & action_task_params getting filtered out on their upgrade to 7.16. For users who don't follow the guidance and migration from < 7.13 to 8.0, they may experience a slower upgrade, but it should still work.

Any thoughts on this this @mikecote?

[mikecote]

@joshdover There is still a backlog item on the alerting side to clean up documents as failures are encountered (#55340). Until this is complete, there will be scenarios (not sure how likely) where the user could end up in the same state as before, with a lot of documents to clean up.

We did create a background cleanup task in 7.13 (#96971) to mitigate this problem, but there is still a chance a scenario like the above could happen. The user would need to have > 1,200 actions failing per hour, which can be possible.

With that said, it sounds preferable to solve #55340 and remove this core API at the same time, I'm not sure if it can be deferred to 8.x instead of 8.0?

[joshdover]

With that said, it sounds preferable to solve #55340 and remove this core API at the same time, I'm not sure if it can be deferred to 8.x instead of 8.0?

I think we can really only remove this in the short-term if we fix #55340 for the 7.16 release since that is a release we'd expect users to upgrade to before upgrading to 8.x. If #55340 is fixed in a later release, it's still possible for their 7.16 -> 8.x upgrade to be slow since there may be many tasks objects created in 7.16 that will have to processed in the migration.

If it's not feasible to fix #55340 for 7.16 then I think we're stuck with supporting this for some time and should proceed with the items in this issue.

[mikecote]

If it's not feasible to fix #55340 for 7.16 then I think we're stuck with supporting this for some time and should proceed with the items in this issue.

Thanks for the insight, I'll have to default on the team not foreseeing capacity to complete this in time for 7.16. Though, PRs are welcome if the ROI on the core team is worth it.

[mikecote]

In the meantime, I've added #55340 to our 7.16/8.0 candidates list so we don't lose sight of the issue.

[mikecote]

@lukeelmers good news, from the conversation above ^^, we've fixed #55340 so it should unblock this issue.