First impression feedback

[mrocklin]

Just took this for a spin, here is some feedback. Some of this is administrative and probably obvious to you. I'm listing it here just for completeness, not to prioritize:

Administrative

• Needs nicer __repr__ implementation
• Should move code out of __init__.py
• Should maybe move tests within parquet/tests ?
• Should probably keep test data outside of the package (I think that this is already done, just wanted to check)
• Need to remove references to personal namespace (mdurant shows up in the tests)
• How are we going to package this long-term? Do we need a different name than the existing parquet project?

Performance

• The from_delayed call needs to be passed metadata to avoid triggering a local computation
• The from_delayed call would like to be passed divisions= information if we can get it from the parquet file. We might want to sniff statistics within the parquet file for sorted columns and, if we find exactly one, make it the index automatically?

[martindurant]

I now pass divisions= based on the row counts in each row_group. The information on whether there are sorting columns and statistics in each piece are also available, so I can implement what you suggest.

[martindurant]

Suggestions for what should should be in __repr__/__str__? I have an info, columns, dtypes, divisions (i.e., row numbers) and cats (i.e., partitions) properties currently available.

[mrocklin]

Location, size in bytes, columns or number of columns, dtypes? I wouldn't
worry about it near term though. I think we'll have more to say on this
topic after the project sees more use.

On Wed, Oct 26, 2016 at 9:46 AM, Martin Durant notifications@github.com
wrote:

Suggestions for what should should be in repr/str? I have an
info, columns, dtypes, divisions (i.e., row numbers) and cats (i.e.,
partitions) properties currently available.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#2 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AASszNhUMuZrhXIXYI5ttzCNKYVBEKfXks5q31nCgaJpZM4Kf9JM
.