Implement the KPL Aggregated Record Format for Kinesis indexing service

[dclim]

The Kinesis indexing service (#6431) supports de-aggregation of multiple rows packed into a single record by the Kinesis Producer Library's aggregate method for more efficient data transfer. Currently, enabling the de-aggregate functionality requires the user to manually provide the Kinesis Client Library on the classpath, since this library has a license not compatible with Apache projects.

See: https://github.com/apache/incubator-druid/pull/6431/files#diff-88b66c67f91d2ab9d26a23abf0543bfbR312

A better solution would be for the Druid project to implement the protocol ourselves according to the specification here: https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md

For legal reasons, this should be done by a contributor who has not read through the Amazon licensed implementation so that there is no case for copyright infringement (https://en.wikipedia.org/wiki/Clean_room_design).

[justinborromeo]

Currently working on this.

[himanshug]

https://lists.apache.org/thread.html/95aec3e271903320994fba321c58aff66b739b1f68796c253e430469@%3Cdev.druid.apache.org%3E means we can now package the AWS lib itself instead of implementing the protocol ?

[justinborromeo]

@himanshug yes. Closing #6917 since it's no longer necessary.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.