[ST] Fix MigrationST

[im-konge]

Type of change

• Bugfix

Description

This PR fixes few things failing in the STs in the MigrationST class:

• from time to time, producer fails to send one or two messages, so I extended the delivery.timeout.ms (together with request.timeout.ms) to 30s, which should be sufficient in case of disconnection from the node
• long wait before the continuous clients are finished - changed the delay to 500 ms (from 1000ms)
• issue with deletion of KafkaTopic(s) -> the issue is only with topics from the ZK mode, when we migrated to fully KRaft, from time to time UTO wasn't able to delete the KafkaTopic(s) and because of that the clean-up of our STs was failing. For now I just turned off finalizers in UTO, but this is being investigated
• small timeout for movement of Kafka cluster state to KRaftDualWriting and other states -> extended to 5 minutes, however it's still not enough from time to time, depending on the KafkaRoller and when the Pods are rolled. This is being investigated as well
• small timeout for migration AZP -> increased to 240minutes.

Other than this I decreased the wait timeout for "default" resource deletion -> from 3 minutes to 2, as it should be sufficient timeout for resources like CRDs, KafkaTopic, KafkaUser, Secret, ConfigMap, etc.

Checklist

• Make sure all tests pass

[im-konge]

/azp run migration

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[im-konge]

/azp run migration

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[im-konge]

/azp run migration

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[im-konge]

/azp run migration

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[henryZrncik]

Good Job! Increase in request.timeout.ms and delivery.timeout.ms seems reasonable to me. Also the cleaning logs seems to be clean of any errors now, so thanks for looking on that.

[ppatierno]

LGTM assuming it works.
At the same time you should plan investigations on the stuff already mentioned in the description. This PR sounds like a temporary workaround to me if I got it correctly.

[im-konge]

At the same time you should plan investigations on the stuff already mentioned in the description. This PR sounds like a temporary workaround to me if I got it correctly.

The only thing that needs investigation is the deletion of topics by UTO.
The timeout for movement to KRaftDualWriting state is due to KAFKA-16563.
I will check the issue with topics further - to track it, I created #9975.