KAFKA-8073: Fix flaky BaseQuotaTest.testThrottledProducerConsumer

[nileshkumar3]

Ref: https://issues.apache.org/jira/browse/KAFKA-8073

Causes

1. produceUntilThrottled sent tiny payloads, so on slow hosts the
   1000 sends could stay under the 8000 B/s producer quota and never
   trigger a throttle.
2. The broker throttle-time metric was cached once before the send
   loop, so the lazily-registered sensor could be missed.
3. verifyThrottleTimeRequestChannelMetric asserted on the broker
   ThrottleTimeMs histogram immediately, before the broker had
   recorded the value.

Changes

• Use a fixed 2 KB payload in produceUntilThrottled so the quota is
  reliably exceeded.
• Re-read the broker throttle-time metric inside the send loop.
• Poll the RequestChannel ThrottleTimeMs via TestUtils.waitUntilTrue.

[muralibasani]

@nileshkumar3 can you pls add some loop(may be 50) run results executed locally ? this could help if the flakiness is gone.

[nileshkumar3]

@nileshkumar3 can you pls add some loop(may be 50) run results executed locally ? this could help if the flakiness is gone.
@muralibasani I ran the ClientIdQuotaTest.testThrottledProducerConsumer locally 50 times and all passed.

[github-actions]

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.