kinesis-client-buildpack for Heroku

This is a Heroku buildpack that pulls in JAR files used to make use the Amazon Kinesis Client Library. Since Java 8 is required to make use of those JAR files, this buildpack is built atop the official Heroku JVM buildpack.

Compatability

This buildpack is compatible with version 1.0.1 of the aws-kclrb gem. Be sure to pin your gem version accordingly:

gem 'aws-kclrb', '= 1.0.1'

In general, this buildpack will be versioned in parallel with aws-kclrb so version numbers should match to guarantee compatibility.

Usage

Since the JAR files in the buildpack have to correspond with the gem version, you probably want to pin buildpack versions you want to use when adding the buildpack:

$ heroku buildpacks:set https://github.com/apartmentlist/kinesis-client-buildpack.git#v1.0.1.2

$ heroku buildpacks -a my-ruby-app
=== my-ruby-app Buildpack URLs
1. heroku/ruby
2. https://github.com/apartmentlist/kinesis-client-buildpack.git#v1.0.1.2

Details

The jar list is the most interesting part of this buildpack. The list was generated by creating an empty maven project with a pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.apartmentlist.kcl</groupId>
  <artifactId>kcl-app</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>kcl-app</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>amazon-kinesis-client</artifactId>
      <version>1.8.1</version>
    </dependency>
  </dependencies>
</project>

You can find the latest version of the amazon-kinesis-client in its Maven repository listing.

Once a new project is created with this file, the dependencies can be downloaded to a directory. From the project root, one can simply:

mkdir jars
mvn dependency:copy-dependencies -DoutputDirectory=jars

The group ID, artifact ID and versions for each can be extracted using mvn:

mvn dependency:list

The result shows all dependencies and their versions:

com.amazonaws:aws-java-sdk-kinesis:jar:1.11.171:compile
com.fasterxml.jackson.core:jackson-databind:jar:2.6.7.1:compile
com.amazonaws:aws-java-sdk-kms:jar:1.11.171:compile
com.fasterxml.jackson.core:jackson-core:jar:2.6.7:compile
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.6.7:compile
commons-codec:commons-codec:jar:1.9:compile
com.amazonaws:aws-java-sdk-cloudwatch:jar:1.11.171:compile
com.amazonaws:amazon-kinesis-client:jar:1.8.1:compile
com.amazonaws:aws-java-sdk-dynamodb:jar:1.11.171:compile
com.google.guava:guava:jar:18.0:compile
joda-time:joda-time:jar:2.8.1:compile
commons-lang:commons-lang:jar:2.6:compile
commons-logging:commons-logging:jar:1.1.3:compile
software.amazon.ion:ion-java:jar:1.0.2:compile
com.amazonaws:aws-java-sdk-s3:jar:1.11.171:compile
com.amazonaws:aws-java-sdk-core:jar:1.11.171:compile
org.apache.httpcomponents:httpclient:jar:4.5.2:compile
org.apache.httpcomponents:httpcore:jar:4.4.4:compile
com.fasterxml.jackson.core:jackson-annotations:jar:2.6.0:compile
com.amazonaws:jmespath-java:jar:1.11.171:compile
com.google.protobuf:protobuf-java:jar:2.6.1:compile

This listing can then be split out to update the list of jars and versions that should be downloaded in the compile script.

Testing

If you want to modify the buildpack (probably the compile file) and test it, you can run it on a development machine:

mkdir tmp
STACK=heroku-16 ./bin/compile tmp tmp

This will produce two directories cache and jars that contain the JARs. You can remove them when you're finished testing.