[SPARK-10969] [Streaming] [Kinesis] Allow specifying separate credentials for Kinesis and DynamoDB

[kaklakariada]

KinesisUtils.createStream() allows specifying only one set of AWS credentials that will be used by Amazon Kinesis Client Library (KCL) for accessing Kinesis, DynamoDB and CloudWatch.

Motivation

In a scenario where one needs to read from a Kinesis Stream owned by a different AWS account the user usually has minimal rights (i.e. only read from the stream). In this case creating the DynamoDB table in KCL will fail. Instead the DynamoDB table should be created in your own AWS account.

Proposal

This PR adds a new serializable class AWSCredentialPool containing credentials for Kinesis, DynamoDB and CloudWatch that is passed as a parameter to KinesisUtils.createStream(). These credentials are then passed to the constructor of KinesisClientLibConfiguration.

@tdas @cfregly please review if this PR is OK.

See SPARK-10969

This contribution is @ajayrammohan and my original work and we license the work to the project under the project's open source license.

[AmplabJenkins]

Can one of the admins verify this patch?

[soulcutter]

This sentence looks like it got mangled during editing somehow.

[soulcutter]

I like the idea of being able to provide separate credentials for services, however it seems to me like a more-common use case is still going to be providing a single set of credentials for all services. It would be nice if the way to do so was still as straightforward as it is currently.

[kaklakariada]

@soulcutter Thank you very much for your review and sorry for the delay!
Please correct me if I am wrong, but in test testKinesisStreamWithSimpleAwsCredentials we test, that you still can use a single set of credentials with the old API.

[soulcutter]

Welp I think you're right :)

[kaklakariada]

Solved with https://issues.apache.org/jira/browse/SPARK-19911 / #17250, see this comment.