KCL Library doesn't work with the newest Kinesis SDK

[Nath5]

When using the Kinesis SDK version 1.10.1 the KCL library throws the following error:

    java.lang.NoSuchMethodError: com.amazonaws.AmazonWebServiceRequest.copyPrivateRequestParameters()Ljava/util/Map;
       at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:969)
       at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(AmazonCloudWatchClient.java:370)
       at com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher.publishMetrics(DefaultCWMetricsPublisher.java:63)
       at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.runOnce(CWPublisherRunnable.java:144)
       at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.run(CWPublisherRunnable.java:90)
       at java.lang.Thread.run(Thread.java:745)

[kevincdeng]

Hi Nath5,

Can you provide some information about your setup?

1. Which version of the KCL are you using?
2. How did you upgrade to the 1.10.1 SDK? Did you change the pom.xml of the KCL?
3. Which version of Java are you using?

[Nath5]

Hey Kevin,

1. We are using version 1.4.0 of the KCL
2. We are using gradle as the build tool, we are using several other
   libraries which bring in the latest version of the AWS SDK. Gradle by
   default will use the latest version of a library if multiple dependencies
   require it. So since other libraries are including 1.10.1 of the kinesis
   sdk that is what is being used.
3. We are using Oracle's version of Java 8

Let me know if you need any more details.

Thanks,
Nathan

On Tue, Jun 23, 2015 at 2:52 PM Kevin Deng notifications@github.com wrote:

Hi Nath5,

Can you provide some information about your setup?

1. Which version of the KCL are you using?
2. How did you upgrade to the 1.10.1 SDK? Did you change the pom.xml
   of the KCL?
3. Which version of Java are you using?

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

[kevincdeng]

Looks like the implementation of AmazonWebServiceRequest changed between 1.9 and 1.10. The copyPrivateRequestParameters method no longer exists. This then means that the aws-java-sdk-cloudwatch package you're using is not the same version as the aws-java-sdk-core being used.

Is this because your other projects do not have a dependency on aws-java-sdk-cloudwatch, causing gradle to have upgraded aws-java-sdk-core to 1.10, but not aws-java-sdk-cloudwatch? Is there a way you can override the version of all the sdk components used?

[ebolinger]

Please update the POM file to build KCL against aws-java-sdk.version = 1.10.8.
It seems that the cloudwatch / core dependency did not follow Semantic Versioning -- specifically the part about maintaining backwards-compatibility between version 1.9 and 1.10. That API change should have incremented the MAJOR version number because removing the AmazonWebServiceRequest copyPrivateRequestParameters() method was not backwards-compatible. Below is my dependency graph showing:

• amazon-kinesis-client 1.5.1 depends on:
• aws-java-sdk-cloudwatch 1.9.37, which depends on:
• aws-java-sdk-core 1.9.37, which is evicted by:
• aws-java-sdk-core 1.10.2, because my app includes:
• another AWS library that depends on 1.10.2.

[info]   +-com.amazonaws:amazon-kinesis-client:1.5.1
[info]   | +-com.amazonaws:aws-java-sdk-cloudwatch:1.9.37
[info]   | | +-com.amazonaws:aws-java-sdk-core:1.10.2
[info]   | | | +-commons-logging:commons-logging:1.1.3
[info]   | | |
[info]   | | +-com.amazonaws:aws-java-sdk-core:1.9.37 (evicted by: 1.10.2)

My work-around was to add an application dependency on the latest AWS Java SDK. In SBT syntax this was:

 libraryDependencies ++= Seq(
  "com.amazonaws" % "amazon-kinesis-client" % "1.5.1",
  "com.amazonaws" % "aws-java-sdk" % "1.10.8"
 )

[stephenh]

FWIW (and apologies for perhaps restating the obvious) this is not an issue with KCL's metadata itself, it's just because of a deficiency in Maven/Ivy/etc. metadata to represent a "upgrade this group of parent/child of dependencies in lock step".

We had filed a ticket against the main aws-java-sdk:

aws/aws-sdk-java#384

But it was closed, admittedly because there isn't much projects like aws-java-sdk can do, other than be dogmatic about treating internal/cross-module breaking API changes as severe as external breaking API changes.

We've had similar issues with slf4j (api/impl), httpclient (core/client), scala (library/compiler), and jackson (core/modules) all doing the same "breaking internal changes between related modules", and finally had to put a build rule in place (using Ivy, but could be expressed in Gradle/sbt/etc.) that fails the build if we detect mismatched versions of certain groups of modules, e.g. aws-java-sdk-* jars/slf4j/etc.

E.g. all of our aws-java-sdk-* jars have to match to the specific x.y.z version.

So, I don't have much to add I guess, other than to commiserate that it's affected us too, and will be an ongoing issue for the Java/Maven ecosystem.

[pfifer]

It looks like this is fixed, feel free to reopen if you're still having problems.