Conversation
Warning: this is an untested sketch! This change provides a few humble functions to try to adapt the Xarray model to Dask's dataframe model. The conversion is more or less an itertools.product and index operation. The translation to dataframes honor's Xarray's chunks. I've copied weather-tools' `ichunked` function just in case we need that layer of chunking of iterables (it's not used now). There area a few next steps. From here, we can write unit tests to prove out the conversion to Dask Dataframe. Further, we can then add dask-sql to this module and see how it works on real SQL queries. I'm pretty sure before applying `unravel` to `form_map`, we'll need to convert the output to a Pandas dataframe.
Collaborator
Author
|
CC: @mahrsee1997 -- Feel free to contribute to this branch if this is interesting to you! |
It lacks support for dataframe features. Nor is it performant. But at a base level, we can convert Xarray datasets to dataframes.
1. Compute slices late 2. Adding `divisions` metadata (via an integer index).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change provides a few humble functions to try to adapt the Xarray model to Dask's dataframe model. The conversion is more or less an itertools.product and index operation. The translation to dataframes honor's Xarray's chunks.
There area a few next steps. From here, we can add dask-sql to this module and see how it works on real SQL queries.