Fix test flakes in TestKafkaTransportProvider

[akaritakai]

Checking for the topic's existence using AdminUtils.topicExists(..) isn't enough to guarantee that the topic is ready for production, allowing a race condition to exist.

When we lose the race condition, the test fails like this:

WARN  NetworkClient:968 - [Producer clientId=testClient] Error while fetching metadata with correlation id 2 : {testTopic_1=UNKNOWN_TOPIC_OR_PARTITION}
INFO  Metadata:285 - Cluster ID: ...
INFO  Log:66 - [Log partition=testTopic_1-0, dir=...] Loading producer state till offset 0 with message format version 2
INFO  Log:66 - [Log partition=testTopic_1-0, dir=...] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in ... ms
INFO  LogManager:66 - Created log for partition testTopic_1-0 in ... with properties ...
INFO  Partition:66 - [Partition testTopic_1-0 broker=2] No checkpointed highwatermark is found for partition testTopic_1-0
WARN  KafkaTransportProvider:test:0:261 - sent failure due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

As we can see, the broker is still setting everything up when the producer makes the metadata request, so the produce request fails. This failure also results in the test's KafkaProducer closing, making retries not possible in the current configuration.

A simple fix is to ensure that the topic is consumable (and thus is set up on the broker) before we try to produce to it, which is what this PR does.

[groelofs]

LGTM

[ahmedahamid]

LGTM. Thank you, @akaritakai.

[ahmedahamid]

Thanks again

[lhaiesp]

LGTM