[Transform] Improve reporting status of the transform that is about to finish

[przemekwitek]

This PR fixes 2 places that may yield issues with status reporting when the transform finishes:

1. it makes the TransformTask.shutdown method wait until the persistent task disappears.
2. It makes the _stats action report stopping rather than stopped state if there is no persistent task seen by the TaskManager but there still is a task in the ClusterState.

Additionally, this PR adds a test (TransformRobustnessIT.testCreateAndDeleteTransformInALoop) which shows that every time we are about to delete the transform, the persistent task is already gone.

Relates #95327

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @przemekwitek, I've created a changelog YAML for you.

[droberts195]

LGTM, although I think it would be clearer if you renamed one variable.

[hendrikmuhs]

2. It makes the `_stats` action report `stopping` rather than `stopped` state if there is no persistent task seen by the `TaskManager` but there still is a task in the `ClusterState`.

This is great.

1. it makes the `TransformTask.shutdown` method wait until the persistent task disappears.

The stop API has a wait_completion query parameter, the added code is a duplication of that in a different place. You find it in the transport implementation of _stop.

The change breaks wait_completion=false.

I therefore think we should revert that part. It doesn't solve the fundamental problem that if the indexer is running _stop does not unregister the persistent task. That means that with a hanging indexer thread it is not possible to remove the p-task using _stop.

[przemekwitek]

2. It makes the `_stats` action report `stopping` rather than `stopped` state if there is no persistent task seen by the `TaskManager` but there still is a task in the `ClusterState`.

This is great.

1. it makes the `TransformTask.shutdown` method wait until the persistent task disappears.

The stop API has a wait_completion query parameter, the added code is a duplication of that in a different place. You find it in the transport implementation of _stop.

The change breaks wait_completion=false.

I therefore think we should revert that part. It doesn't solve the fundamental problem that if the indexer is running _stop does not unregister the persistent task. That means that with a hanging indexer thread it is not possible to remove the p-task using _stop.

Right, removing the p-task using stop is a separate problem that I'd like to tackle.
I've verified (using the TransformRobustnessIT.testCreateAndDeleteTransformInALoop test case) that this change was indeed unnecessary.

The partial revert PR is ready for review: #96060