SPARK-5960 allow aws credentials to be passed to KinesisUtils.createStream

[mce]

No description provided.

[srowen]

(Why is this needed?)

[mce]

tests are failing on kinesis-acl, at least on my local. Error message is saying a method is missing from a class in jackson library. I don't know the exact reason but adding this dependency fixes the error

[srowen]

Hm, I doubt this is the right way to fix it, since you are declaring a compile-time dependency on something that you don't use. Surely it's because there is a missing dep on some Kinesis component at runtime scope? you also shouldn't manage versions here.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Nit: space after import.

I know we have all lived to regret including third-party classes as part of the API. Although it seems maybe essential here, I wonder if there is any reasonable way to not accept this as AWSCredentials but as access keys or something. I suppose that this could work but would assume a certain type of credentials are being used.

[mce]

Updated code a bit after your comments.

Created Credentials class to avoid third-party import in api.
Moved credentials parameter to existing constructor as an optional parameter.

[cfregly]

my initial attempt at adding this support introduced a BasicAWSCredentialsProvider as follows:

https://github.com/cfregly/spark/blob/kinesis-dbc/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/BasicAWSCredentialsProvider.scala

highlights:

1. takes AccessKey and SecretKey as Strings
2. follows the same hierarchy as the rest of the AWSCredentialsProviders (including not being Serializable)

[srowen]

ok to test

[SparkQA]

Test build #30028 has started for PR 5439 at commit d93dcb6.

[SparkQA]

Test build #30028 has finished for PR 5439 at commit d93dcb6.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Credentials(accessKeyId: String, secretKey: String) extends AWSCredentials
• This patch does not change any dependencies.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30028/
Test FAILed.

[SparkQA]

Test build #30115 has started for PR 5439 at commit bb36da0.

[SparkQA]

Test build #30115 has finished for PR 5439 at commit bb36da0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30115/
Test PASSed.

[mce]

@srowen tests passed, fyi

[cfregly]

collapse the IRecordProcessor and IRecordProcessorFactory into a single import, as well?

[mce]

done

[SparkQA]

Test build #30334 has started for PR 5439 at commit 5cfd437.

[SparkQA]

Test build #30334 has finished for PR 5439 at commit 5cfd437.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30334/
Test FAILed.

[SparkQA]

Test build #30335 has started for PR 5439 at commit 9877a98.

[SparkQA]

Test build #30335 has finished for PR 5439 at commit 9877a98.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30335/
Test PASSed.

[SparkQA]

Test build #30406 has started for PR 5439 at commit 7789df7.

[SparkQA]

Test build #30406 has finished for PR 5439 at commit 7789df7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30406/
Test PASSed.

[mce]

@cfregly Moved initialization code into onStart method to fix serialization error. I tested by running example application and it runs without any problem.

[mce]

@cfregly any other comments?

[AmplabJenkins]

Can one of the admins verify this patch?

[tdas]

I am really not sure how this will work. AWSCredentials is not serializable, while the Receiver has to be serializable (as the receiver object is sent to the executors from the driver). Did you test whether this works?

[tdas]

I am taking a look at this PR a little late. But I am not sure how this work at all. AWSCredentials is not serializable.

[tdas]

@mce Are you able to update this? If you are not able to work on this, mind closing this PR?

[mce]

@tdas sorry for the late response, i'm closing the pr now as i don't have any time to work on this currently.