ENH: Parquet, HDF5, and CSV interfaces #1194

jreback · 2017-10-27T00:55:56Z

closes #1175
closes #1165

on top of #1167

jreback · 2017-10-27T00:58:47Z

@wesm is it better map parquet logical types directly to ibis types, or convert to arrow types which then almost trivially map to ibis types?

(Pdb) p parquet_file.schema
<pyarrow._parquet.ParquetSchema object at 0x10bcf6c48>
ticker: BYTE_ARRAY UTF8
time: INT64 TIMESTAMP_MICROS
open: DOUBLE
__index_level_0__: INT64
 
(Pdb) p list(parquet_file.schema)[0].logical_type
'UTF8'

wesm · 2017-10-27T02:39:49Z

@jreback it's probably better to use the Arrow types, which will be more consistent with the underlying execution.

How large do we anticipate each of the file implementations being? Any reason to have the module vs. ibis/file/parquet.py?

jreback · 2017-10-27T22:25:19Z

@jreback it's probably better to use the Arrow types, which will be more consistent with the underlying execution.

I added an arrow converter (not used, but its in the tests) & a parquet one; certainly can change to just convert parquet types to arrow to ibis.

How large do we anticipate each of the file implementations being? Any reason to have the module vs. ibis/file/parquet.py?

was organizing like the pandas backend. but yes we could combine all of these into a single file. though this is not user visible anyhow.

jreback · 2017-10-27T23:40:05Z

ok revamped to use a single file per (csv, hdf5, parquet).

wesm · 2017-10-30T00:41:05Z

👍

jreback · 2017-10-31T14:35:10Z

ibis/file/parquet.py

+
+def parquet_types_to_ibis_schema(schema):
+    pairs = []
+    for cs in schema:


@wesm is there (maybe internal) function to convert a ParquetColumnSchema to an Arrow schema?
would prefer to not have these parquet routines directly exposed here, rather have arrow types (which trivially convert to ibis types)

The code exists, but needs to be exposed: https://issues.apache.org/jira/browse/ARROW-1759

jreback · 2017-10-31T15:45:26Z

@wesm @cpcloud I am not sure how parquet represents in python2 but this seems odd

using the example table from https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_parquet.py#L72

I am not seeing a logical_type for strings in py2; in py3 these are UTF8 (both are BYTE_ARRAY as physical_type). note that the last field is passed in as bytes, so this looks as expected.

PY2

<pyarrow._parquet.ParquetSchema object at 0x1047bf908>
uint8: INT32 UINT_8
uint16: INT32 UINT_16
uint32: INT64
uint64: INT64 UINT_64
int8: INT32 INT_16
int16: INT32 INT_16
int32: INT32
int64: INT64
float32: FLOAT
float64: DOUBLE
bool: BOOLEAN
datetime: INT64 TIMESTAMP_MICROS
str: BYTE_ARRAY
str_with_nulls: BYTE_ARRAY
empty_str: BYTE_ARRAY
bytes: BYTE_ARRAY
__index_level_0__: INT64

PY3

uint8: INT32 UINT_8
uint16: INT32 UINT_16
uint32: INT64
uint64: INT64 UINT_64
int8: INT32 INT_16
int16: INT32 INT_16
int32: INT32
int64: INT64
float32: FLOAT
float64: DOUBLE
bool: BOOLEAN
datetime: INT64 TIMESTAMP_MICROS
str: BYTE_ARRAY UTF8
str_with_nulls: BYTE_ARRAY UTF8
empty_str: BYTE_ARRAY UTF8
bytes: BYTE_ARRAY
__index_level_0__: INT64

cpcloud · 2017-10-31T15:52:13Z

Hm I would expect strings to be UTF8 there for both Pythons. That seems like a bug.

jreback · 2017-11-01T11:54:18Z

revised, I guess I missed parquet_file.schema.to_arrow_types()

xhochy · 2017-11-01T17:20:19Z

unicode in Python 2 should be UTF8, str has no encoding and does is only regarded as a byte array.

cpcloud · 2017-11-01T18:23:37Z

Yep, @xhochy when I said "strings" I meant "string types coming from parquet". All string types should be represented by unicode in python2 and str in python3.

jreback · 2017-11-01T20:24:53Z

note that .to_arrow_types() handles this the correct way. I am thinking that there might be not enough meta-data in parquet and arrow is doing some kind of work-around here.

cpcloud · 2017-11-01T20:53:36Z

ibis/expr/datatypes.py

    'float': float,
+    'halffloat': float16,


Do we need to have two spellings of float16 or can we just leave it as float16?

i can take it out, but this is a common spelling, e.g. like float, double.

Ok, that's fine.

cpcloud · 2017-11-01T20:53:55Z

ibis/file/client.py

+class FileClient(ibis.client.Client):
+
+    def __init__(self, root):
+        super(FileClient, self).__init__()


Don't need this call here.

cpcloud · 2017-11-01T20:54:47Z

ibis/file/client.py

+    def __dir__(self):
+        dbs = self.list_databases(path=self.path)
+        tables = self.list_tables(path=self.path)
+        return sorted(list(set(dbs).union(set(tables))))


Can just call sorted here.

cpcloud · 2017-11-01T20:55:09Z

ibis/file/client.py

+
+        new_name = "{}.{}".format(name, self.extension)
+        if (self.root / name).is_dir():
+            path = path / name


You can do augmented assignment here as: path /= name

cpcloud · 2017-11-01T20:56:01Z

ibis/file/client.py

+
+    def __getattr__(self, name):
+        try:
+            return object.__getattribute__(self, name)


A call to __getattribute__ isn't necessary because you're only ever inside __getattr__ if a call to __getattribute__ has raised AttributeError

cpcloud · 2017-11-01T21:04:45Z

ibis/file/parquet.py

+
+
+_ARROW_DTYPE_TO_IBIS_TYPE = {
+    'int8': dt.int8,


Can we use the arrow objects here rather than strings?

This seems not implemented :>

In [5]: hash(pa.int8()) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-5-bca0e6e2f6af> in <module>() ----> 1 hash(pa.int8()) TypeError: unhashable type: 'pyarrow.lib.DataType'

cpcloud · 2017-11-01T21:05:16Z

ibis/file/parquet.py

+}
+
+
+def arrow_types_to_ibis_schema(schema):


Can you make a note somewhere that this doesn't handle complex types yet?

cpcloud · 2017-11-01T21:07:44Z

ibis/file/parquet.py

+        return t
+
+    def list_tables(self, path=None):
+        # tables are files in a dir


Again this looks pretty similar, I think there's some opportunity to factor this code out.

cpcloud · 2017-11-01T21:07:54Z

ibis/file/parquet.py

+
+
+@pre_execute.register(ParquetTable, ParquetClient)
+def parquet_data_preload_uri_client(op, client, scope=None, **kwargs):


See csv pre_execute comment

see my comment above

cpcloud · 2017-11-01T21:08:55Z

ibis/file/utils.py

@@ -0,0 +1,6 @@
+try:
+    import pathlib  # noqa


Can you put the specific flake8 error code here?

i remove this file, only used in 1 place

cpcloud · 2017-11-02T15:00:49Z

ibis/expr/datatypes.py

@@ -415,6 +415,32 @@ def valid_literal(self, value):
        return isinstance(value, six.string_types + (datetime.datetime,))


+class SignedInteger(Integer):


The Int8, Int16, Int32, and Int64 classes should inherit from this.

closes ibis-project#1165

closes ibis-project#1175

jreback · 2017-11-07T13:41:08Z

@cpcloud looks good to go

jreback · 2017-11-14T17:35:41Z

@cpcloud this looks ready to merge.

jreback · 2017-11-19T18:27:43Z

ping

cpcloud · 2017-11-15T12:50:13Z

ibis/expr/types.py

+    pass
+
+
+class HalffloatScalar(ScalarExpr, FloatValue):


This needs to subclass from halffloat value not float value

cpcloud · 2017-11-15T12:50:29Z

ibis/expr/types.py

+    pass
+
+
+class HalffloatColumn(NumericColumn, FloatValue):


Same as previous comment

cpcloud · 2017-11-19T22:32:24Z

ibis/file/csv.py

+                usecols = None
+
+        else:
+


Can you remove some of this extra whitespace?

cpcloud · 2017-11-19T22:34:15Z

ibis/file/client.py

+                    if str(d).endswith(self.extension):
+                        tables.append(d.stem)
+        elif path.is_file():
+            # by definition we are at the db level at this point


Why is this true?

for HDF5 a file is a database, as well as a dir.

Should this be in the HDF client then?

no, if you look above you requested that the common code be moved here.

Right, I definitely remember that request, but this method is only used in one place and makes assumptions that are specific to HDF5 which is why I'm asking. This isn't a blocker though.

remove halffloatingvalue superclass

cpcloud · 2017-11-20T03:56:06Z

Merging! Thanks @jreback!

jreback added the feature Features or general enhancements label Oct 27, 2017

jreback force-pushed the parquet branch from 6be2ce2 to dc02d00 Compare October 27, 2017 22:21

jreback force-pushed the parquet branch from dc02d00 to b786d88 Compare October 27, 2017 22:31

jreback changed the title ~~WIP: Parquet file interface~~ ENH: Parquet file interface Oct 27, 2017

jreback force-pushed the parquet branch from b786d88 to 99e11c5 Compare October 27, 2017 23:34

jreback mentioned this pull request Oct 27, 2017

feat: file back end enhancements #1195

Closed

3 tasks

jreback self-assigned this Oct 27, 2017

jreback added this to the 0.12 milestone Oct 27, 2017

jreback force-pushed the parquet branch 3 times, most recently from 62b8051 to 9a4080e Compare October 28, 2017 00:47

jreback force-pushed the parquet branch from 9a4080e to 3e49b37 Compare October 30, 2017 10:28

jreback modified the milestones: 0.12, 0.13 Oct 30, 2017

jreback force-pushed the parquet branch from 3e49b37 to 5812d8d Compare October 31, 2017 14:32

jreback commented Oct 31, 2017

View reviewed changes

cpcloud changed the title ~~ENH: Parquet file interface~~ ENH: Parquet, HDF5, and CSV interfaces Oct 31, 2017

cpcloud requested changes Nov 1, 2017

View reviewed changes

cpcloud reviewed Nov 2, 2017

View reviewed changes

jreback force-pushed the parquet branch from 42ef51d to 6b37516 Compare November 3, 2017 23:02

jreback added 12 commits November 7, 2017 08:04

ENH: initial support a file-like backend with csv & hdf5 implementations

762ef32

closes ibis-project#1165

ENH: parquet backend implementation

acb83b9

closes ibis-project#1175

compat for strings in py2

bf3077e

require pyarrow 0.7.1; use parquet_file.schmea.to_arrow_types()

c32c7a9

review comments

05ceb27

review comments

655039f

fixed types

620a41d

handle multiple physical tables

e00a34b

update recipe for newer pyarrow

e87a756

only need pyarrow 0.6.0

49989f8

re-add scope caching

58a6c80

bump 2.7 build to numpy 1.11 to allow pyarrow to install

33cd1da

jreback force-pushed the parquet branch from 6b37516 to 33cd1da Compare November 7, 2017 13:04

jreback mentioned this pull request Nov 10, 2017

Unified ibis <-> pandas type conversions #1221

Closed

cpcloud requested changes Nov 19, 2017

View reviewed changes

jreback added 2 commits November 19, 2017 16:33

subclass halffloat from halffloatvalue

04d9483

fix up casting tests

dba6609

cpcloud reviewed Nov 19, 2017

View reviewed changes

jreback added 2 commits November 19, 2017 17:39

remove whitespace

e5a838a

remove implicit casting tests

6c53bdf

remove halffloatingvalue superclass

cpcloud approved these changes Nov 20, 2017

View reviewed changes

cpcloud closed this Nov 20, 2017

cpcloud reopened this Nov 20, 2017

cpcloud closed this in 7b777fd Nov 20, 2017



		@pre_execute.register(ParquetTable, ParquetClient)
		def parquet_data_preload_uri_client(op, client, scope=None, **kwargs):

		@@ -415,6 +415,32 @@ def valid_literal(self, value):
		return isinstance(value, six.string_types + (datetime.datetime,))


		class SignedInteger(Integer):

ENH: Parquet, HDF5, and CSV interfaces #1194

ENH: Parquet, HDF5, and CSV interfaces #1194

Conversation

jreback commented Oct 27, 2017 • edited Loading

jreback commented Oct 27, 2017

wesm commented Oct 27, 2017

jreback commented Oct 27, 2017

jreback commented Oct 27, 2017

wesm commented Oct 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Oct 31, 2017

cpcloud commented Oct 31, 2017 • edited Loading

jreback commented Nov 1, 2017

xhochy commented Nov 1, 2017

cpcloud commented Nov 1, 2017

jreback commented Nov 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback Nov 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud Nov 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 7, 2017

jreback commented Nov 14, 2017

jreback commented Nov 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Nov 20, 2017 • edited Loading

jreback commented Oct 27, 2017 •

edited

Loading

cpcloud commented Oct 31, 2017 •

edited

Loading

jreback Nov 2, 2017 •

edited

Loading

cpcloud Nov 2, 2017 •

edited

Loading

cpcloud commented Nov 20, 2017 •

edited

Loading