Skip to content
Permalink
Browse files
Package kinesis client jar within the extension (#12370)
amazon-kinesis-client was not covered undered the apache license and required separate insertion in the kinesis extension.
This can now be avoided since it is covered, and including it within druid helps prevent incompatibilities.

Allows enabling of deaggregation out of the box by packaging amazon-kinesis-client (1.14.4) with druid for kinesis ingestion.
  • Loading branch information
AmatyaAvadhanula committed Apr 4, 2022
1 parent 9849047 commit 067254b7782cad0b27abf5815771981a4fcf3cda
Show file tree
Hide file tree
Showing 3 changed files with 23 additions and 7 deletions.
@@ -572,12 +572,9 @@ determines the number of records to be processed per each ingestion loop in the
See [issue](https://github.com/apache/druid/issues/6714)

The Kinesis indexing service supports de-aggregation of multiple rows packed into a single record by the Kinesis
Producer Library's aggregate method for more efficient data transfer. Currently, enabling the de-aggregate functionality
requires the user to manually provide the Kinesis Client Library on the classpath, since this library has a license not
compatible with Apache projects.
Producer Library's aggregate method for more efficient data transfer.

To enable this feature, add the `amazon-kinesis-client` (tested on version `1.9.2`) jar file ([link](https://mvnrepository.com/artifact/com.amazonaws/amazon-kinesis-client/1.9.2)) under `dist/druid/extensions/druid-kinesis-indexing-service/`.
Then when submitting a supervisor-spec, set `deaggregate` to true.
To enable this feature, set `deaggregate` to true when submitting a supervisor-spec.

## Resharding

@@ -134,8 +134,17 @@
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>1.9.2</version>
<scope>provided</scope>
<version>1.14.4</version>
<exclusions>
<exclusion>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-cloudwatch</artifactId>
</exclusion>
<exclusion>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
</exclusion>
</exclusions>
</dependency>

<!-- Tests -->
@@ -3479,6 +3479,16 @@ libraries:

---

name: Kinesis Client
license_category: binary
version: 1.14.4
module: extensions/druid-kinesis-indexing-service
license_name: Apache License version 2.0
libraries:
- com.amazonaws: amazon-kinesis-client

---

name: Kafka Client
version: 5.5.1-ccs
license_category: binary

0 comments on commit 067254b

Please sign in to comment.