-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-8865][STREAMING] FIX BUG: check key in kafka params #7254
Conversation
Could you add a test case to |
Also, could you add a brief description? |
Test build #36648 has finished for PR 7254 at commit
|
LGTM. The explanation is that contains == containsValue |
@koeninger Please take a look at this. I am curious as to how it fails. |
"zookeeper.connect" and "group.id" aren't necessary for anything in the kafka direct stream. But they're expected to be present in a kafka consumer config, and overriding that behavior wasn't possible. So as a workaround, we set them to a blank string. That way users don't have to define unnecessary settings in the kafka param map passed to the KafkaUtils constructor. We talked through that during the original development of the direct stream. The code as it is released today is almost always going to set a blank string, regardless of what users pass in, because contains on a java property object is not the equivalent of containsKey, it is containsValue. This just fixes it so that if they do pass in values for those keys, for whatever personal reasons they have, the values won't get overwritten with a blank string. TL;DR this should have no impact, but is a good bugfix so that the code does what it looks like it's supposed to. |
In case it wasn't clear, LGTM I don't think it needs a test case, because it would be testing behavior that spark doesn't care about (we don't deal with group id or zookeeper connect) |
@koeninger Thank you very much for the explanation. I am going to add this to the JIRa. |
Author: guowei2 <guowei@growingio.com> Closes #7254 from guowei2/spark-8865 and squashes the following commits: 48ca17a [guowei2] fix contains key (cherry picked from commit 8977003) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Author: guowei2 <guowei@growingio.com> Closes #7254 from guowei2/spark-8865 and squashes the following commits: 48ca17a [guowei2] fix contains key (cherry picked from commit 8977003) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
No description provided.