Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for protobuf input format #5269

Closed
npawar opened this issue Apr 17, 2020 · 4 comments
Closed

Add support for protobuf input format #5269

npawar opened this issue Apr 17, 2020 · 4 comments
Assignees

Comments

@npawar
Copy link
Contributor

npawar commented Apr 17, 2020

Currently the supported input formats for batch ingestion are CSV, AVRO, JSON, Parquet, Thrift, ORC.

Add support for Protobuf.

@reallocf
Copy link
Contributor

Hey @npawar - I'm interested in working on this one. Would be my first time contributing to the project, would love to help! 🎉

I'm guessing I'll create another pinot-input-format and mirror pinot-json or pinot-csv or pinot-parquet. Then update RecordReaderFactory in pinot-spi and probably the CreateSementCommand in pinot-tools. Where else should I be updating? I'm sure I'm missing a bunch of places :)

@kishoreg
Copy link
Member

You pretty much got everything right. I think @KKcorps has already started working on this. Can you pick up the support for filtering during ingestion? #5268

@reallocf
Copy link
Contributor

@kishoreg - thanks for letting me know! Definitely don't want to duplicate work 👍 will take a look at #5268

@KKcorps
Copy link
Contributor

KKcorps commented Apr 21, 2020

Taking this up!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants