Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C] SQLite3 driver using nanoarrow #120

Closed
Tracked by #76
paleolimbot opened this issue Sep 12, 2022 · 5 comments · Fixed by #196
Closed
Tracked by #76

[C] SQLite3 driver using nanoarrow #120

paleolimbot opened this issue Sep 12, 2022 · 5 comments · Fixed by #196
Assignees
@lidavidm
Milestone
ADBC Libraries 0.1.0

Comments

@paleolimbot
Copy link
Member

I did a little prototyping around this to try to hit nanoarrow with some more real-world data...my interest in SQLite3 is mostly around the GeoPackage (https://geopackage.org), which is a common spatial file format based on SQLite3. The experiment is currently living in its own repo ( https://github.com/paleolimbot/minigpkg/blob/main/src/minigpkg/nanoarrow_sqlite3.h#L112-L127 ), and while I don't think I currently have the bandwidth to implement the ADBC API, I'd be happy to circle back to this at some point since an SQLite3 driver that depends only on sqlite3 seems like it would be useful here.

As noted in the current driver (https://github.com/apache/arrow-adbc/blob/main/c/drivers/sqlite/sqlite.cc#L139-L141), SQLite really needs an adaptive builder or an adaptive schema guesser to be useful.
@lidavidm
Copy link
Member

Yeah, I do want to rework the driver at some point, and nanoarrow would be part of that (the adaptive builder would be the other part). But so far, SQLite is the easiest one for me to use as a testbed (since I can pull in whatever junk I need from libarrow) so I've resisted refactoring. (Also, because I'm focusing on the Postgres and Flight SQL drivers as 'real' drivers.) But having SQLite be a 'real' driver would be useful!

@paleolimbot paleolimbot mentioned this issue Oct 4, 2022
Closed
@lidavidm lidavidm self-assigned this Oct 5, 2022
@lidavidm
Copy link
Member

lidavidm commented Oct 5, 2022

I'm going to try to pick this up soon - it'll be generally useful especially for docs and demos, just because SQLite is small

@paleolimbot
Copy link
Member Author

Just a heads up that I added investigating this as part of my sprint task for the next two weeks ("actually try to use nanoarrow")...I figured this would be a good practical outlet for that and it seems like it might help with the R adbc work.

@lidavidm
Copy link
Member

@paleolimbot I have a branch available: https://github.com/lidavidm/arrow-adbc/tree/sqlite-nanoarrow

@lidavidm lidavidm mentioned this issue Nov 17, 2022
Closed
12 tasks
@lidavidm lidavidm moved this from Todo to In Progress in Arrow ADBC Initial Release Nov 17, 2022
@lidavidm lidavidm mentioned this issue Nov 27, 2022
Merged
@lidavidm
Copy link
Member

PR is finally up at #196. Just need to shepherd it through CI (only tested so far on a macOS dev environment)

lidavidm added a commit that referenced this issue Nov 28, 2022
@lidavidm @paleolimbot
feat(c/driver/sqlite): port SQLite driver to nanoarrow (#196)
3b11e33 
Also related: #144.

Fixes #120.

Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net>
Repository owner moved this from In Progress to Done in Arrow ADBC Initial Release Nov 28, 2022
@lidavidm lidavidm added this to the 0.1.0 milestone Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lidavidm lidavidm

Labels
None yet
Projects
No open projects
Arrow ADBC Initial Release
Status: Done
Milestone
ADBC Libraries 0.1.0
Development

Successfully merging a pull request may close this issue.

feat(c/driver/sqlite): port SQLite driver to nanoarrow lidavidm/arrow-adbc
2 participants
@lidavidm @paleolimbot