[FEATURE] Support optional force deletion of certain resources

[ohltyler]

Currently, certain resources are blocked from being deleted during deprovisioning, including indices, as a precaution to prevent users from accidentally deleting potential data.

By allowing the ability to delete such resources if the user explicitly wants to, this gives users more flexibility. It simplifies the experience of performing iterative testing / rapid prototyping on a workflow, where users may be constantly deprovisioning / reprovisioning resources.

On the frontend, the UX is focused on iterative testing of workflows, most of which involve index creation. Currently, this causes errors during reprovisioning due to the created index resource existing. There are frontend-only solutions for this problem, but ideally this is allowed directly from the backend for consistency and ease of use for API users.

Proposal: an allow_delete list param - a fine-grained parameter for deleting any specially-handled resources specified in the list of resource types passed

POST .../_deprovision?allow_delete=index,ingest_pipeline,foo
{
"workflow_id" : "abc"
}

Internally, we can add a delete index workflow step (and for any other resources that have this guardrail) that will only be executed when the parameter logic specifies as such.

Additionally, we may want some notification / visibility to users on such scenarios. See #579 for details.

[dbwiddis]

Do you think matching the workflow step type (e.g., delete_index) or the workflow step id (e.g., foo_index) is better?

[ohltyler]

Do you think matching the workflow step type (e.g., delete_index) or the workflow step id (e.g., foo_index) is better?

From user-facing view, the latter makes more sense to me. They may not know / care about the internal workflow step type, but just know they want to delete foo_index. Additionally, specifying the particular resource may prevent more complexity / logic on the backend to try to determine which resource a user actually wants to delete (e.g., only specify delete_index but have a workflow with 3 index creations, do we delete one? all? etc.)

[dbwiddis]

From user-facing view, the latter makes more sense to me. They may not know / care about the internal workflow step type, but just know they want to delete foo_index.

That was my thinking as well. It's also easily obtainable on the resources list that they're deprovisioning so both backend code and frontend code (hey here's the resources, check the "force delete" box on the ones you want) can see them.

Only remaining complication I see is how to indicate which resources require force delete. I'll think about that....

[minalsha]

@dbwiddis could you please link doc issue created for this feature?

[ohltyler]

@dbwiddis could you please link doc issue created for this feature?

It is linked in the linked PR #763 : opensearch-project/documentation-website#7639

[minalsha]

ahh my bad. Thank you