avro-extensions -- feature to specify avro reader schema inline #3249

himanshug · 2016-07-15T05:07:08Z

in the task json for all events.

See https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/druid-development/tg2q3VHN5rQ/5kUuiTouEQAJ

skoppar · 2016-08-04T21:44:47Z

This is something we need in our set up. Thanks Himanshu. Which release is this expected to be a part of ?

drcrallen · 2016-08-04T21:47:00Z

...ions-core/avro-extensions/src/main/java/io/druid/data/input/avro/SimpleAvroBytesDecoder.java

+
+  private final Schema schemaObj;
+
+  private Map<String, Object> schema;


drcrallen · 2016-08-04T22:09:58Z

This should be reasonable to be in 0.9.2, and if there's a community need even more so

himanshug · 2016-08-05T04:50:02Z

@skoppar do you have single schema for all the events in the kafka topic or multiple ? is it possible for you to test this patch?

are you already trying to use avro extension with schema_repo and facing some error?

nishantmonu51 · 2016-08-05T05:39:56Z

LGTM, 👍

himanshug · 2016-08-05T14:35:03Z

updated the docs.

skoppar · 2016-08-05T15:23:55Z

Hi Himanshu,
I havent tried Schema repo and was about to extend the AvroBytesDecoder myself and thats when I saw your comment on another google thread. https://groups.google.com/forum/#!searchin/druid-user/avro$20druid%7Csort:relevance/druid-user/72NKnm6eBRY/VCgyaTKEAwAJ

I am testing your patch now. Will keep you posted

…e task json for all events

skoppar · 2016-08-08T18:28:20Z

publicAvroKafka.txt

Hi Himanshu,

I am not able to get the avro stream to work with the new jar. I have
packaged your changes which produced
"druid-avro-extensions-0.9.2-SNAPSHOT.jar".
I moved the original jar into a subdirectory so that there are no conflicts
and copied the snapshot jar in current directory (imply-1.3.0/dist/druid/
extensions/druid-avro-extensions). Also changed imply-1.3.0/conf-
quickstart/druid/_common/common.runtime.properties to include avro library:

druid.extensions.loadList=["druid-avro-extensions", "druid-histogram",
"druid-datasketches", "druid-kafka-indexing-service"]

I've modified kafka.json in imply-1.3.0/conf-quickstart/tranquility to the
one attached.

Upon starting imply:
bin/supervise -c conf/supervise/quickstart.conf

I do not see any topics being picked in the tranquility logs
[imply-1.3.0/var/sv/tranquility-kafka.log]:
2016-08-08 17:27:56,913 [main] INFO k.c.ZookeeperConsumerConnector -
[cust1-tranquility_skoppar-mac15.com-1470677276325-37ef097f], Creating
topic event watcher for topics ((?!))
2016-08-08 17:27:56,929 [main] INFO k.c.ZookeeperConsumerConnector -
[cust1-tranquility_skoppar-mac15.com-1470677276325-37ef097f], Topics to
consume = List()

Doesn't it mean that no topics have been registered?

Only relevant entries in the log are:
2016-08-08 17:27:56,151 [main] INFO c.m.t.kafka.writer.WriterController -
Ready: [topicPattern] -> dataSource mappings:
2016-08-08 17:27:56,152 [main] INFO c.m.t.kafka.writer.WriterController -
[(?!)] -> AvroSource (priority: 1)

I am new to Druid, so please pardon my ignorance. Is the
publicAvrokafka.json file right? At this point I havent added any business
functionality. The derived fields are coming from an ETL process outside of
Druid. I just want to be able to ingest avro data through kafka and that
doesnt seem to work. To be sure its not a kafka issue, I have tried using
kafka-console-consumer and I can receive events there.

regards
Sunita

On Fri, Aug 5, 2016 at 7:35 AM, Himanshu notifications@github.com wrote:

updated the docs.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3249 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoyRcBJ9n4QvVBCCUoH05HJTDWJh38ks5qc0olgaJpZM4JNFH8
.

himanshug · 2016-08-08T18:47:00Z

@skoppar i haven't really used kafka with tranquility and information provided does not look like avro parsing related failures.
i don't see any kafka topic etc input information in your spec , not sure how you provide it to tranquility... @gianm can you spot something?

skoppar · 2016-08-08T19:14:31Z

Thanks for quick response Himanshu.
From what I understand,
"properties": {
"topicPattern": "raw-cust1"
}

Provides the topic which is per datasource level and below properties provide overall cluster connection. Adding group_id to data source specs might be more advisable, but I haven't tried that yet. I am using all default ports.

"kafka.group.id":"cust1-tranquility",
"zookeeper.connect":"localhost",
"kafka.zookeeper.connect":"localhost"

regards
Sunita

On Mon, Aug 8, 2016 at 11:47 AM, Himanshu notifications@github.com wrote:

@skoppar https://github.com/skoppar i haven't really used kafka with
tranquility and information provided does not look like avro parsing
related failures.
i don't see any kafka topic etc input information in your spec , not sure
how you provide it to tranquility... @gianm https://github.com/gianm
can you spot something?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3249 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoycaGfewz_18wXsiKchgnJC_MDnRYks5qd3mygaJpZM4JNFH8
.

skoppar · 2016-08-09T03:30:49Z

Another fact is, Zookeeper is able to recognize the consumer group and the
group is visible upon executing zookeeper shell. Also from tranquility
side, zookeeper connection seems to be present. The Kafka broker connection
is what seems to be missing and the topic list is empty as mentioned in the
previous comment. Appreciate any help @gianm https://github.com/gianm
I also tried with 0.8.2.0-kafka-1.3.0 instead of 0.9.0-kafka-2.0.1

nishantmonu51 · 2016-08-09T17:24:24Z

@skoppar seems like you have specified the properties at wrong place in json spec file,
the properties needs to be defined outside the spec block in json as done here -
https://github.com/druid-io/tranquility/blob/master/distribution/src/universal/conf/kafka.json.example

skoppar · 2016-08-09T17:32:07Z

Thanks Nishant for bringing it to my notice.
Yea I see there is some discrepancy in the spec now. It indeed was the issue w.r.t recognizing the topic. After putting the topic details out of spec, tranquility is able to find the kafka topic.

Currently getting some class loading issues with the new jar. Probably I need to double check the extension loadlist. Will update here.

skoppar · 2016-08-09T18:22:22Z

Hi Himanshu,

Looks like there is a dependency on io.druid.druid-api-0.9.1.jar based on the errors I see. imply-1.3.0/dist/tranquility/lib has a huge list of jars. I hope I have to replace fewer jars. Are there any documented dependencies? Attaching the exception I get now.
WithPR.txt

gianm · 2016-08-09T18:26:28Z

@skoppar what is the tranquility config you are using now?

skoppar · 2016-08-09T18:37:25Z

@gianm attaching the current tranquility-kafka json file.
publicAvroKafka.txt

himanshug · 2016-08-09T18:44:06Z

@skoppar @gianm given that exception came from StringInputRowParser , I believe avro_stream parser did not get recognized and that probably means avro-extensions module wasn't loaded. did you see any avro-extension loading failures or a message regarding loading of avro-extension in the logs?

skoppar · 2016-08-09T19:01:18Z

@himanshug doesnt look like loading the jar is the issue. I could be wrong though. Attaching the indexing-log in imply-1.3.0/var/druid/indexing-logs which shows the druid-avro-extensions-0.9.2-SNAPSHOT.jar being added. Please let me know if I am looking at a wrong place.
index_realtime_metrics-kafka.txt

himanshug · 2016-08-09T20:14:02Z

@skoppar one of the problems in your json is that "parseSpec" is inside "avroBytesDecoder" while it should be a level up inside "parser" .

skoppar · 2016-08-09T20:32:59Z

oops ! let me relook at the complete json

On Tue, Aug 9, 2016 at 1:14 PM, Himanshu notifications@github.com wrote:

@skoppar https://github.com/skoppar one of the problems in your json is
that "parseSpec" is inside "avroBytesDecoder" while it should be a level up
inside "parser" .

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3249 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoyXAMybK3Ysq-K-Ou47LgWSrF7UY4ks5qeN-YgaJpZM4JNFH8
.

skoppar · 2016-08-09T21:36:36Z

Its working now. 2 more changes I had to do was:

Specify the format in parseSpec
Add the path to druid extensions manually in quickStart.conf

With that done to the attached conf, I am able to process records with schema inline. However, I do not see the data source in druid console or pivot. I can see metrics-kafka (had run the example), so I was expecting this should show up as well.

Using topic name - raw-avro for this test. Below is what the tranquility-kafka log shows - the avro schema and then the below message
2016-08-09 21:34:23,038 [KafkaConsumer-CommitThread] INFO c.m.tranquility.kafka.KafkaConsumer - Flushed {raw-avro={receivedCount=800, sentCount=0, droppedCount=0, unparseableCount=800}} pending messages in 0ms and committed offsets in 3ms.

himanshug · 2016-08-10T17:47:31Z

@skoppar glad to know that it worked out for you and thanks for testing the patch.

fjy · 2016-08-10T17:49:17Z

👍

skoppar · 2016-08-10T17:50:24Z

Well not completely. Trying to figure out why segments are not getting
created. But the patch did work well. This must be something else I am
missing. I have 800 messages received and unparseableCount=800. I tried
changing timestamp to be current, but still get the same results.
Appreciate any inputs you have on this, however, it does not seem to be
related to the changes in the PR.

On Aug 10, 2016, at 10:47 AM, Himanshu notifications@github.com wrote:

@skoppar https://github.com/skoppar glad to know that it worked out for
you and thanks for testing the patch.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3249 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoyTP1qdVFbw4wClZT9FgayJfKcbtOks5qeg7BgaJpZM4JNFH8
.

skoppar · 2016-08-10T17:52:20Z

Well not completely. Trying to figure out why segments are not getting
created. But the patch did work well in terms of pulling the schema and
getting right count of events. This must be something else I am missing. I
have 800 messages received and unparseableCount=800. I tried changing
timestamp to be current, but still get the same results. Appreciate any
inputs you have on this, however, it probably is not related to the changes
in the PR.

himanshug · 2016-08-10T18:25:25Z

@skoppar enable debug logging and that would show you more info.

skoppar · 2016-08-10T21:06:56Z

Figured it out with the help of a kafka consumer class. The console consumer showed data just fine, so couldn't spot the error. The rootcause is, I was sending the avro message using StringEncoder. Using DefaultEncoder on producer side fixed the issue. Thanks @himanshug and @gianm. Awaiting 9.2 release :) 👍

himanshug added the Improvement label Jul 15, 2016

drcrallen reviewed Aug 4, 2016
View reviewed changes

drcrallen added this to the 0.9.2 milestone Aug 4, 2016

himanshug force-pushed the avro_schema branch from 2a83f5d to 97fa774 Compare August 5, 2016 04:47

himanshug changed the title ~~WIP: avro-extensions -- feature to specify avro reader schema inline~~ avro-extensions -- feature to specify avro reader schema inline Aug 5, 2016

himanshug force-pushed the avro_schema branch from 97fa774 to 66cde68 Compare August 5, 2016 14:34

avro-extensions -- feature to specify avro reader schema inline in th…

599a17d

…e task json for all events

himanshug force-pushed the avro_schema branch from 66cde68 to 599a17d Compare August 5, 2016 21:46

fjy merged commit 46da682 into apache:master Aug 10, 2016

himanshug mentioned this pull request Aug 16, 2016

avro-extension -- feature to specify multiple avro reader schemas inline #3368

Merged

himanshug deleted the avro_schema branch January 3, 2017 16:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

avro-extensions -- feature to specify avro reader schema inline #3249

avro-extensions -- feature to specify avro reader schema inline #3249

himanshug commented Jul 15, 2016

skoppar commented Aug 4, 2016

drcrallen Aug 4, 2016

himanshug Aug 5, 2016

drcrallen commented Aug 4, 2016

himanshug commented Aug 5, 2016

nishantmonu51 commented Aug 5, 2016

himanshug commented Aug 5, 2016

skoppar commented Aug 5, 2016 •

edited

skoppar commented Aug 8, 2016 •

edited

himanshug commented Aug 8, 2016

skoppar commented Aug 8, 2016 •

edited

skoppar commented Aug 9, 2016 •

edited

nishantmonu51 commented Aug 9, 2016 •

edited

skoppar commented Aug 9, 2016 •

edited

skoppar commented Aug 9, 2016 •

edited

gianm commented Aug 9, 2016

skoppar commented Aug 9, 2016

himanshug commented Aug 9, 2016

skoppar commented Aug 9, 2016 •

edited

himanshug commented Aug 9, 2016

skoppar commented Aug 9, 2016

skoppar commented Aug 9, 2016 •

edited

himanshug commented Aug 10, 2016

fjy commented Aug 10, 2016

skoppar commented Aug 10, 2016

skoppar commented Aug 10, 2016 •

edited

himanshug commented Aug 10, 2016

skoppar commented Aug 10, 2016


		private final Schema schemaObj;

		private Map<String, Object> schema;

avro-extensions -- feature to specify avro reader schema inline #3249

avro-extensions -- feature to specify avro reader schema inline #3249

Conversation

himanshug commented Jul 15, 2016

skoppar commented Aug 4, 2016

drcrallen Aug 4, 2016

Choose a reason for hiding this comment

himanshug Aug 5, 2016

Choose a reason for hiding this comment

drcrallen commented Aug 4, 2016

himanshug commented Aug 5, 2016

nishantmonu51 commented Aug 5, 2016

himanshug commented Aug 5, 2016

skoppar commented Aug 5, 2016 • edited

skoppar commented Aug 8, 2016 • edited

himanshug commented Aug 8, 2016

skoppar commented Aug 8, 2016 • edited

skoppar commented Aug 9, 2016 • edited

nishantmonu51 commented Aug 9, 2016 • edited

skoppar commented Aug 9, 2016 • edited

skoppar commented Aug 9, 2016 • edited

gianm commented Aug 9, 2016

skoppar commented Aug 9, 2016

himanshug commented Aug 9, 2016

skoppar commented Aug 9, 2016 • edited

himanshug commented Aug 9, 2016

skoppar commented Aug 9, 2016

skoppar commented Aug 9, 2016 • edited

himanshug commented Aug 10, 2016

fjy commented Aug 10, 2016

skoppar commented Aug 10, 2016

skoppar commented Aug 10, 2016 • edited

himanshug commented Aug 10, 2016

skoppar commented Aug 10, 2016

skoppar commented Aug 5, 2016 •

edited

skoppar commented Aug 8, 2016 •

edited

skoppar commented Aug 8, 2016 •

edited

skoppar commented Aug 9, 2016 •

edited

nishantmonu51 commented Aug 9, 2016 •

edited

skoppar commented Aug 9, 2016 •

edited

skoppar commented Aug 9, 2016 •

edited

skoppar commented Aug 9, 2016 •

edited

skoppar commented Aug 9, 2016 •

edited

skoppar commented Aug 10, 2016 •

edited