Support Avro IDL

[mariussoutier]

Currently decoders must be .avsc files, but more complex schemas are usually written in .avdl files and should be supported by Trifecta.

[ldaniels528]

Sorry for the late reply. I absolutely agree and I'm working on a bug fix release at present that will hopefully be completed within the next couple of days. After which, I can look into supporting .avdl. However, if you've already made changes for supporting .avdl, please submit a pull request, and I'll merge it.

[mariussoutier]

I haven't, but I'll look into it and get back to you tomorrow!

[ldaniels528]

Now that I've finally release v0.19.0, I plan to incorporate the feature you've requested into the next release (v0.19.1)

[mariussoutier]

Nice, thank you!

[mariussoutier]

Hi, how is this coming forward? If you can't find the time, can you just give me a hint where in the codebase to implement this, then I'll try to submit a PR.

[ldaniels528]

I'm planning to work on the feature this week. I'll keep you posted.

[ldaniels528]

The class TxConfig is responsible for retrieving the decoders (currently
just AvroDecoder (avsc) instances). In order to implement a different type,
you'll have to change a few things:

Take a look at TxConfig.getDecodersByTopic(topic: String): Seq[TxDecoder].
It's responsible for returning the collection of decoders. Currently,
TxDecoder accepts just an AvroDecoder instance. It makes the most sense to
change AvroDecoder into a trait with two sub-classes: AvscDecoder and
AvIDLDecoder.

The TxDecoder definition:

case class TxDecoder(topic: String, name: String, lastModified: Long,
decoder: Either[AvroDecoder, TxFailedSchema])

You'll also need to change the following method:

TxConfig.getDecoderFromFile(topic: String, decoderFile: File)

You can use the file's extension to determine which type of Avro definition
you're dealing with so that you can return the appropriate implementation
(as above: AvscDecoder and AvIDLDecoder).

On Fri, Aug 5, 2016 at 9:01 AM, Marius Soutier notifications@github.com
wrote:

Hey, I can do it for you, then you can work on something else. I
definitely have time for it this week.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#16 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEc3pbX_c8emm57P-cOOrUMonrFtwqQ_ks5qc15qgaJpZM4GrSks
.

Best Regards,

Lawrence Daniels

[mariussoutier]

Ok I have a first implementation in the linked fork (branch avdl). So it seems I only really had to touch the AvroDecoder companion object. I'm not sure the two AvroDecoder subclasses are even necessary.

The main thing to watch now is that if the Schema is passed as a String, newlines must not be omitted or Avro cannot parse it correctly.

Although it might be better to only allow passing it as a File or IS so the parser has control over it - what do you think?

[ldaniels528]

Thanks, I'll take a look.

On Wed, Aug 10, 2016 at 8:10 AM, Marius Soutier notifications@github.com
wrote:

Ok I have a first implementation in the linked fork (branch avdl). So it
seems I only really had to touch the AvroDecoder companion object. I'm
not sure the two AvroDecoder subclasses are even necessary.

The main thing to watch now is that if the Schema is passed as a String,
newlines may not be omitted or Avro cannot parse it correctly.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#16 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEc3pWj6lOZTeYS5hNEOgvvpbJ8SnrfMks5qeenbgaJpZM4GrSks
.

Best Regards,

Lawrence Daniels

[mariussoutier]

So this time on the right branch and I now only use the companion object to differentiate between the two.

[ldaniels528]

Submit a pull request and I'll merge it.