Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contribute to Drill? #2

Open
cgivre opened this issue Mar 4, 2020 · 7 comments
Open

Contribute to Drill? #2

cgivre opened this issue Mar 4, 2020 · 7 comments

Comments

@cgivre
Copy link

cgivre commented Mar 4, 2020

Hello there,
Nice work on your Drill storage plugin! I was wondering if you might consider submitting this as a pull request to be included in the next version of Drill?
If you are open to it, I can assist with the review process.
Thanks!

@dbw9580
Copy link
Collaborator

dbw9580 commented Mar 7, 2020

Hi, thank you for contacting! We would love to see Minerva become part of Drill!
We had been working on other projects and haven't updated it for quite a while. There are still performance issues and bugs here and there. So I'm not sure if it meets the standards of Drill to be merged, and if there is anything else that should be done on our side?

@cgivre
Copy link
Author

cgivre commented Mar 8, 2020

@dbw9580
I can help you through the review process. It's ok if it isn't perfect. We can get it into Drill and then refine it. One thing we will need however are unit tests. There is a current PR for a storage plugin for Apache Druid 1 where the author wrote a very thorough suite of unit tests. You can use that as a reference for how to do it.

@dbw9580
Copy link
Collaborator

dbw9580 commented Mar 8, 2020 via email

@bmcnabb25
Copy link

@cgivre @dbw9580 out of curiosity, was this plugin ever included in Drill?

@cgivre
Copy link
Author

cgivre commented Apr 26, 2022

@bmcnabb25 There was a pull request started (apache/drill#2084) but it hasn't been merged yet. My sense is that it was close and if someone was willing to pick up the work, we could get it merged.

@nashshapley
Copy link

@cgivre Thanks for the invitation. Would you mind listing the remaining questions that need to be solved? Bowen and our new team members will work together to get it merged. If possible, would you please send the todo list to my email: ydxu@fudan.edu.cn.

@cgivre
Copy link
Author

cgivre commented Oct 20, 2022

Hi @nashshapley
Please take a look here: apache/drill#2084.

The reason this wasn't merged was that in its current implementation, the IPFS/Drill plugin was creating new Drillbits and tearing them down as needed. This is likely to cause serious issues in a distributed system such as Drill because there is no way to control what query goes to which Drillbit.

With all that said, I think there is a MUCH simpler way of implementing this plugin. In Drill 1.20 we added a file system extension for Dropbox. It reuses all the existing Drill file readers. Basically, you'd have to implement methods that get an InputStream of the file. You can also write by doing the same thing. It should be fairly straightforward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants