Expose regionName setting in Kinesis receiver configuration

[kopiczko]

Hi,

I'd like to have that setting exposed to be able to set DynamoDB table (for KCL) in the same region as the Kinesis stream.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Isn't this changing the API?
I believe you should make a JIRA along with this PR. It's not trivial.
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark

[cfregly]

@srowen @kopiczko

This is part of a larger effort to overhaul Kinesis-based streaming slated for 1.4. Lots of API changes including region, AWS credentials, and application name - as well as upgrading both the AWS Java SDK and the KCL.

Here's the parent jira: https://issues.apache.org/jira/browse/SPARK-6599.

Here's the related jira that covers the region portion: https://issues.apache.org/jira/browse/SPARK-6514.

We should definitely try to be backward-compatible even though the API is Experimental.

[kopiczko]

Thanks guys for your response. @cfregly I've already answered your comment on Jira. I guess we should move our discussion there. I can improve this implementation to meet requirements in SPARK-6514 and rename PR according to guidelines in wiki.

[cfregly]

might want to add a note similar to the KCL README.md when describing this param:

"The Amazon DynamoDB table and Amazon CloudWatch metrics associated with your application will also use this region setting."

[tdas]

Any updates on this patch? If you are not able to work on it, mind closing it?

[tdas]

This is not needed any more as Spark 1.4.0 has fixed this issue. Mind closing this PR?

[kopiczko]

Sorry guys, I currently have no time to work on it. @tdas: Would you mind give a reference how this is solved?

[tdas]

Sure, take a look at the new KinesisUtils API.

https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.kinesis.KinesisUtils$