Prototype: delegation to Dataframe library

[alxmrs]

Is there a way to quickly get feature complete with xql — without implementing a SQL engine?Beyond using just DataFusion, can we be lazier?

Here’s my pitch: we provide a core operation to translate a chunked xr.Dataset to a dd.DataFrame. Then, we let dask-sql handle the heavy lifting of performing SQL queries.

Why would this work?

Xarray chunks have an analog to Dask chunked arrays. This is the core issue to solve to get decent performance, not necessarily the SQL engine. If we use dask.dataframe.io.from_map to translate a xr.Dataset chunk into a deferred, chunked dask array — and the dask DataFrame has sufficient metadata to compute indexes s.t. values can be lazily evaluated, then it should work and be performant.

Why is this preferred?

Let’s let the dask-sql project be on the hook for maintenance, for implementing the SQL spec (including hairy cases like nested queries and joins).
Plus, we may get the ability to join raster and tabular data together “for free.” This would be a killer use case for xql.

Why is this not preferred?

This ties our implementation to Dask. (Mitigation: maybe we could make the Dataframe transformation use a “portable” dataframe library, like Fugue.) Does coupling the implementation to a dataframe prevent future plans for distributed Xarray backends (Cubed, xbeam with Stephan’s Dataframe refactor)?

Unknowns

• Would we lose out on efficiencies from compact data representation (in contiguous arrays)?
• Does the math check out? Specifically, the chunking math? For example, are there obvious corner cases when we perform standard queries (select..wheres, joins) given that we’re translating from raster chunks to table chunks? We should be able to mathematically model this (or simulate it in a program). Does the arithmetic work out? Can we get lazy evaluation when it’s needed (it’s critical that we avoid loading the whole dataset to perform operations that just use a subset)? Will we always fit within memory a la the guarantees of Dask and Xarray (this is a must to me)? If not, what cases wont we, and how do we mitigate that?

Implementation notes

Assuming we use the implementation defined here in qarray: we need to update the core translate operation to work on a range of data rather than a single index (i.e. vectorized. Maybe operate on a whole Dataset at once? These need to be chunked ofc.). Then, we would be able to add the function to “from_map” or “from_defferred”, applied on a iterable of the xr.Dataset chunks. (This should also help provide clarity on the chunking math, defined above.)