[SPARK-35930][BUILD] Upgrade kinesis-client to 1.14.4

[sarutak]

What changes were proposed in this pull request?

This PR upgrades kinesis-client from 1.14.0 to 1.14.4

Why are the changes needed?

Upgrading to 1.14.1 or newer is recommended by the community for users who use kinesis-client 1.14.0 due to a bug.
https://github.com/awslabs/amazon-kinesis-client/tree/master#recommended-upgrade-for-all-users-of-the-1x-amazon-kinesis-client

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Ran *Kinesis*Suite with ENABLE_KINESIS_TESTS=1 and they passed on an AWS environment

[dongjoon-hyun]

Thank you for making a PR, @sarutak . BTW, is it tested manually because this is not covered by CIs?

[sarutak]

@dongjoon-hyun Thank you, I forgot that kinesis tests don't run on CI.
I ran *Kinesis*Suite with ENABLE_KINESIS_TESTS=1 and they passed on my AWS environment.
But kinesis-client seems to need newer Guava regardless of 1.14.4 or currently used 1.14.0.
Otherwise, we'll get like the following exception.

java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createUnstarted()Lcom/google/common/base/Stopwatch;
        at com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager.<init>(LeaseCleanupManager.java:74)
        at com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager.createOrGetInstance(LeaseCleanupManager.java:105)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:579)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:537)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:406)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:294)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:351)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:198)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:178)
        at org.apache.spark.streaming.kinesis.KinesisReceiver.onStart(KinesisReceiver.scala:192)
        at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
        at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
        at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint.$anonfun$startReceiver$1(ReceiverTracker.scala:596)
        at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint.$anonfun$startReceiver$1$adapted(ReceiverTracker.scala:586

So, I tried with -Dguava.version=27.0-jre. Is this expected?

I also ran kinesis tests on the commit a9e8e05 which upgraded kinesis-client to 1.14.0 but this also seems to need newer Guava.
On the other hand, the commit 82267ac which is the parent commit of a9e8e05 doesn't need newer Guava.

[SparkQA]

Test build #140357 has finished for PR 33133 at commit ba8fc24.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

-Dguava.version=27.0-jre sounds like a regression because the vanilla Apache Spark 3.2.x doesn't work while 3.1.x works.

Since there are some previous works to address the Guava restriction, ping @viirya , @sunchao , @HyukjinKwon in order to get some advice.

[sarutak]

-Dguava.version=27.0-jre sounds like a regression because the vanilla Apache Spark 3.2.x doesn't work while 3.1.x works.

Yeah, a9e8e05 was not merged to branch-3.1 so I think that is the reason it works with 3.1.x while doesn't with 3.2.x.

[sunchao]

Does it mean we'd need to upgrade Guava version in order for kinesis-client to work in Spark 3.2.0?

[viirya]

FYI, I re-opened #29326 to try upgrading guava again.

[HyukjinKwon]

Shall we revert a9e8e05 first?

[dongjoon-hyun]

Since SPARK-34549 is reverted, I'll close this PR, @sarutak . Thank you for checking.

After Guava issue is resolved, feel free to reopen this.