Indexing and time series

[filmackay]

Eagerly awaiting the indexing additions in the (hopefully) near future. How similar to Pandas will this implementation be?

Interested to see how this will intersect with time series functionality - will this just be left to TimeSeries.jl - or will there be embedded capability in DataFrames.jl? I found the Pandas model a bit restrictive - a single index column being a timestamp. It would be nice to have a bit more flexibility - such as multi-level indices that can use a time index amongst others.

[johnmyleswhite]

To be honest, this is at least 12 months away.

[kmsquire]

I'd say that's somewhat pessimistic, in that there's a possibility that we
pick up new developers between now and then and can accelerate progress.
(But I understand the sentiment, John, in that you've been taking a lot of
this on yourself.)

Cheers! Kevin

On Thu, Feb 6, 2014 at 7:58 PM, John Myles White
notifications@github.comwrote:

To be honest, this is at least 12 months away.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/532#issuecomment-34402257
.

[johnmyleswhite]

I'm going to close this issue for the moment since we need to focus on core functionality for the next 6 months. When that's done, we can reopen this issue.

[johnmyleswhite]

Kevin, you're right: if someone comes onboard who can reliably knock out this kind of work, we could make progress a lot faster.

[filmackay]

Issue closed or open, I'd like to have a discussion on the general design of indexing here. John do you have thoughts on the high-level API, and it's interaction with time series functionality?

[filmackay]

I'd be interested in helping, but first I would need to get the design/objectives established. Either that or this would be useful for the poor sucker who does get to this in a year :)

[johnmyleswhite]

I don't really have any opinions on time series functionality. I don't generally work with time series and think someone with more expertise needs to take a management role since my views are not well-informed.

[filmackay]

OK, what about indexing in general - got any views on it where you want to depart from Pandas take on the subject?

[johnmyleswhite]

I think our indexing operations should try to emulate the behavior of a system like SQLite. In general, my preference is always to push for behavior closer and closer to that associated with traditional databases except when we've made a conscious decision to break with the database standards. I don't know enough about the internals of Pandas to make a comparison there, but my (not very informed) sense is that Pandas offers functionality that's much less general than what you'd expect from SQLite.

[HarlanH]

To expand on John's take, Fil, the goal of Julian DataFrames is precisely
to optimize for relational database-like operations. We definitely want to
include indexing (and have made halfhearted stabs in the past that didn't
work very well), but we don't want to implement the sort of rowindex-based
indexing that Pandas uses. In fact, as you may have noticed, there are no
row names in DataFrames.jl, intentionally. We definitely want there to be a
separate implementation of perpetually-sorted time series data that is
distinct from DataFrames (but likely shares at least some internal parts,
such as support of DataArrays). There are other folks working on exactly
that right now. There isn't a sense, in our minds, that the two issues
ought to go together.

On Thu, Feb 6, 2014 at 11:50 PM, John Myles White
notifications@github.comwrote:

I think our indexing operations should try to emulate the behavior of a
system like SQLite. In general, my preference is always to push for
behavior closer and closer to that associated with traditional databases
except when we've made a conscious decision to break with the database
standards. I don't know enough about the internals of Pandas to make a
comparison there, but my (not very informed) sense is that Pandas offers
functionality that's much less general than what you'd expect from SQLite.

Reply to this email directly or view it on GitHubhttps://github.com//issues/532#issuecomment-34404088
.

[filmackay]

Thanks for the replies everyone.

I guess my thinking was not that the two out to go together, as in be implemented together - but that DataFrames will need indexing - and that time series will need to use it.

I had noticed the lack of row names, what were you thinking for indexing in DataFrames if not designating one or more columns as an 'index' for the frame - and then displaying them as row names? I'm interested in how else indexing could work, without at least something similar to this (?)

Or are you saying that DataFrames will never have indexes (row names), and that will always be external in another package that may use DataFrames...

[milktrader]

The current thinking about time series in Julia is that there is room for more than one data structure. TimeSeries initially used the DataFrame data structure, with the first column taking on the role of being the index. Now, TimeSeries has its own data structure called a TimeArray and methods in TimeSeries are being written to support this data structure instead of DataFrames/DataArrays. So far, the data structure is showing some promise as being a lightweight one-type time series structure. Here are some links to recent discussions about time series data structures and DataFrames with time series data. Besides these issues, there is also the newly-registered TimeData package by @cgroll that implements a data structure whose elements include a DataFrame https://github.com/cgroll/TimeData.jl

Time-based DataFrame general discussion
JuliaStats/Roadmap.jl#10

TimeArray behaviors
JuliaStats/Roadmap.jl#7

TImeSeries and TimeModels
JuliaStats/Roadmap.jl#6

[milktrader]

Also, there was some interesting discussion about using DataFrames as an element in a new type in the METADATA PR for TimeData.jl JuliaLang/METADATA.jl#540

[HarlanH]

Indexes in relational databases can be defined for any column, or set of
columns, and a given table can have multiple indexes, not just the one that
the table is ordered by. That's the far-off goal for Julian DataFrames too,
as it gives you maximal flexibility.

We also, and relatedly, want to support memory-mapped and distributed
DataFrames, which would require a blocked structure that would necessarily
have to interact with (and constrain the design of) the indexes.

If you're curious about possible architectures, I'd encourage you to do
some reading in the design and implementation of columnar OLAP database
systems.

On Fri, Feb 7, 2014 at 7:53 AM, Fil Mackay notifications@github.com wrote:

Thanks for the replies everyone.

I guess my thinking was not that the two out to go together, as in be
implemented together - but that DataFrames will need indexing - and that
time series will need to use it.

I had noticed the lack of row names, what were you thinking for indexing
in DataFrames if not designating one or more columns as an 'index' for the
frame - and then displaying them as row names? I'm interested in how else
indexing could work, without at least something similar to this (?)

Or are you saying that DataFrames will never have indexes (row names), and
that will always be external in another package that may use DataFrames...

Reply to this email directly or view it on GitHubhttps://github.com//issues/532#issuecomment-34434044
.