[BEAM-14104] Support shard aware aggregation in Kinesis writer.

[mosche]

This PR introduces shard aware aggregation to achieve better aggregation results (to better max out Kinesis API limits).

If random partitioning is sufficient, this can already be achieved using the explicit random partitioner. The benefit of this approach is that it doesn't require pulling shard details through the API.
However, the random partitioner uses a static configuration that doesn't handle any resharding, that may occur.

Shard aware aggregation is implemented as follows:

• Periodically the writer pulls hash key ranges assigned to shards. These are statically shared using an ObjectPool to minimize the necessary API calls.

• The configured partitioner is used to generate the partitionKey and the optional explicitHashKey for each record. If explicitHashKey is set, it is used as is with no change in behavior. Otherwise, however, the lower bound of the hash key range of the target shard (based on the hashed partitionKey) is chosen for aggregation and set as explicitHashKey.

This makes sure aggregated records always contain only records that match the shard they are routed to even if the stream is in the process of resharding.

To simplify statically sharing the shard state, ClientPool was generalized to ObjectPool. This should be fine as the code wasn't released yet.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[mosche]

R: @aromanenko-dev

[mosche]

@aromanenko-dev Not sure if feasible in terms of reviewing, though would be great to get this into the 2.38 release as well :)

[mosche]

Run Java PostCommit

[mosche]

Run Java PostCommit

[mosche]

Same BigQuery it tests failing

[aromanenko-dev]

Thanks, sorry for delay.
LGTM, just several minor notes.

[aromanenko-dev]

Run Java PostCommit

[aromanenko-dev]

LGTM, just a very minor note on the not obvious constant.
Please, merge it by your self once it will be fixed and Java PostCommit passed

[mosche]

All significant tests passed, failures are unrelated ...