Apache Flink module #92
Conversation
| .builder((eventMap: SimpleEvent) => new DateTime()) | ||
| .curator(curator) | ||
| .discoveryPath(discoveryPath) | ||
| .location(DruidLocation(indexService, firehosePattern, dataSource)) |
There was a problem hiding this comment.
firehosePattern is no longer needed and most users will find it easier to not worry about it. Could you remove that from the example?
| val aggregators = Seq(new LongSumAggregatorFactory("baz", "baz")) | ||
|
|
||
| DruidBeams | ||
| .builder((eventMap: SimpleEvent) => new DateTime()) |
There was a problem hiding this comment.
Using new DateTime for the timestamper is going to lead to mysterious drops (when tranquility thinks an event is ok and reports it sent, but druid silently drops it). I guess you copied this from the spark example, so we should fix both. I don't think that has to happen as part of this PR though.
Filed #93 to remember to do this.
There was a problem hiding this comment.
Yes. Just copied the Spark example. And I think it makes sense to fix both examples at the same time. Won't fix it in this PR.
|
@gesundkrank thanks for the contribution! Could you please take a look at the comments on the patch, squash your commits (see https://github.com/druid-io/druid/blob/master/CONTRIBUTING.md), and fill out the Druid CLA at http://druid.io/community/cla.html? |
|
@gianm Thank you very much for your review! I squashed and commited the fixups. |
|
👍 looks good! @gesundkrank I don't see you in the CLA list- do you fall under a corporate CLA? If so which one? |
|
Yes. My company is mbr targeting. |
|
@gesundkrank thank you, received the CLA |
This module adds support for Apache Flink. Works and looks a lot like the Spark module. But instead of the
BeamRDDit containsBeamSink.Had some struggle with the Flink dependency imports since they are not following the Scala naming conventions.
Would be great to get some feedback on this and even greater to get it merge.