[SPARK-8833][STREAMING][WIP] Kafka Direct API support offset in zookeeper

[guowei2]

Kafka Direct API support offset in zookeeper

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[guowei2]

I will add test case for this later

[SparkQA]

Test build #36564 has started for PR 7235 at commit 1941d7f.

[srowen]

CC @koeninger but I don't think this class is intended to manage Kafka offsets by itself

[srowen]

That's a good bug fix

[AmplabJenkins]

Merged build triggered.

[SparkQA]

Test build #36564 has finished for PR 7235 at commit 1941d7f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #36566 has started for PR 7235 at commit e322623.

[SparkQA]

Test build #36566 has finished for PR 7235 at commit e322623.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test PASSed.

[koeninger]

The containsKey thing is a good catch.

But this basic idea has been discussed before and rejected:

https://issues.apache.org/jira/browse/SPARK-6249

I don't think the direct stream should be implicitly doing anything at all with zookeeper. You can handle writing to / reading from ZK yourself explicitly if you need to.

[koeninger]

@guowei2 do you want to open a separate jira / pull request for that containsKey fix?

[guowei2]

ok