Split input into multiple partitions on request?

[mrocklin]

There is the use case of "please partition my data by hour, regardless of how much data I give to you". is this something that we want to support on top of castra?

[esc]

Yes, and support efficient buffering and appends, like bcolz. But I would say, that is a feature for the future. It also depends on what our 'clients' want.

[cpcloud]

wouldn't this be better suited for dask?

something like

df.repartition(by='H')

[cpcloud]

currently repartition is a bit more generic, but i could imagine an index-type-sensitive version of it

[mrocklin]

This would still be important for data access. The idea here is that one might want data that is accessible in hourly chunks (or some other fixed period.) While dask could repartition this data it would still have to read possibly larger chunks at a time.

[cpcloud]

hm thinking a bit more i can see the value of having the splitting done as soon as you call extend

[jcrist]

Fixed by #40. Closing.