Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Support globbing for multiple parquet files? #24
Is there a way I can load multiple parquet files?
My first guess brings back the following IOError from Arrow:
CREATE VIRTUAL TABLE trips USING parquet('parquet/*');
For the record there are two parquet files in that folder.
$ ls -l parquet/00000*
I'd like to support this more seamlessly in the future, either by supporting a glob or, like Hive, taking a directory to query. There are some internal design things I'd have to think about first, though.
Until then, the best I can suggest is to create N tables, one per parquet file, then create a view that UNION ALLs the tables. If you invoke sqlite like