ENH: allow categoricals in msgpack #12573

pwaller · 2016-03-09T16:59:14Z

Supercedes #12191.

I've made a best-effort to rebase this against master.

It seems to work here.

jreback · 2016-03-09T17:27:59Z

add a note in whatsnew/0.18.1

chris-b1 · 2016-03-09T17:29:45Z

pandas/io/packers.py

+        ctor_dtype = np_dtype
+        if is_categorical_dtype(pd_dtype):
+            # Series ctor doesn't take dtype with categorical
+            ctor_dtype = None


@jreback - should the Series constructor accept a Categorical with dtype=category? This part felt hacky.

it already does.
However in this case want to construct a pd.Categorical as you can pass all of the attributes thru

In [1]: Series(list('aabc'),dtype='category') Out[1]: 0 a 1 a 2 b 3 c dtype: category Categories (3, object): [a, b, c]

but you don't need this here at all. the dtype is not present so its None, the Series will just take a Categorical directly.

Can I conclude that no change is required here in scope of this PR?

no, this needs to be reworked a bit, seem my 2nd comment.

hmm, that might be a bug as in this case it is ok to specify the dtype (but it should be ignored as this is pretty generic as it doesn't specify anything in particular about the dtype, so its compatible). can you open a separate issue for that.

I need to start a new thread on this line of code.

So, if I understood, it shouldn't be necessary to special case the dtype? If I remove the dtype=ctor_dtype change (and leave it set to dtype=np_dtype), then it does not work. I get:

ValueError: cannot specify a dtype with a Categorical unless dtype='category'

I don't currently understand what is being asked for here.

you need to rebase on master, that's the issue that was just merged

This was after rebasing.

so that's what I was first going to write. This is not a numpy-dtype and cannot be treated like one. Instead put this code in encode/decode (e.g. check isinstance(obj, Categorical)). Then you won't have this issue at all.

pwaller · 2016-03-09T17:32:41Z

add a note in whatsnew/0.18.1

Done. Please let me know if it is to taste.

jreback · 2016-03-09T17:40:36Z

doc/source/whatsnew/v0.18.1.txt

@@ -23,6 +23,9 @@ Enhancements
 ~~~~~~~~~~~~


+- ``read_msgpack`` now supports categoricals (:issue:`12573`)
+


support for serializing and de-serializing categoricalls with msgpack

pwaller · 2016-03-09T17:48:53Z

pandas/io/tests/test_packers.py

@@ -341,6 +342,7 @@ def setUp(self):
        self.d['mixed'] = Series(data['E'])
        self.d['dt_tz_mixed'] = Series(data['F'])
        self.d['dt_tz'] = Series(data['G'])
+        self.d['categorical'] = Series(data['H'])


@chris-b1 is this correct? Why is it Series and not Categorical?

It wouldn't make a difference here, e.g. df['c'] = Categorical(..) and df['c'] = Series(Categorical(..)) do the same thing

jreback · 2016-03-16T14:00:20Z

ok #12574 has been merged. pls rebase/update.

pwaller · 2016-03-16T15:07:38Z

Rebased.

jreback · 2016-03-16T15:18:13Z

doc/source/whatsnew/v0.18.1.txt

@@ -30,7 +30,7 @@ Enhancements
 ~~~~~~~~~~~~


-
+- support for serializing and de-serializing categoricalls with msgpack (:issue:`12573`)



Support.........categoricals (sp)

pwaller · 2016-03-16T16:34:57Z

Wow, I think I managed to clear all of your comments. Everything suddenly made sense in the end. Please let me know if I missed something and if I should squash or reword my commits.

pwaller · 2016-03-16T16:36:03Z

I broke the NDFrame tests. Investigating.

jreback · 2016-03-16T16:38:09Z

pandas/io/packers.py

+        return {u'typ': u'categorical',
+                u'klass': u(obj.__class__.__name__),
+                u'name': getattr(obj, 'name', None),
+                u'codes_dtype': u(obj._codes.dtype.name),


I don't think you actually need to specify the codes/catgories dtypes. Those will be automatic. IOW, don't use convert here, just same codes : obj.codes and categories : obj.categories. These are recursive encoder/decoders.

jreback · 2016-03-16T16:48:56Z

pandas/io/packers.py

+        return {u'typ': u'categorical',
+                u'klass': u(obj.__class__.__name__),
+                u'name': getattr(obj, 'name', None),
+                u'codes': obj._codes,


prob use obj.codes here (same but _codes is internal)

pwaller · 2016-03-16T16:50:46Z

pandas/io/tests/test_packers.py

-            'G': [Timestamp('20130603', tz='CET')] * 5
+            'G': [Timestamp('20130603', tz='CET')] * 5,
+            'H': Categorical(['a', 'b', 'c', 'd', 'e']),
+            'I': Categorical(['a', 'b', 'c', 'd', 'e'], ordered=True),


I'm in a strange position here. I added this test - it passes, but I didn't add the relevant code in convert/unconvert to pass the ordered parameter through. What gives?

that check is not used at all! I was just going to tell you to take it out. a Categorical is fully serialized/deserialized via encode/decode the dtype is NEVER category. except when its a series but that is already handled.

jreback · 2016-04-12T20:11:31Z

@pwaller can you rebase/update

jreback · 2016-04-17T15:51:12Z

this looks pretty good code wise. can you squash everything.

jreback · 2016-04-17T15:51:18Z

cc @kawochen

pwaller · 2016-04-17T16:23:18Z

Squashed.

jreback · 2016-04-17T18:36:50Z

pandas/io/packers.py

@@ -262,6 +265,9 @@ def convert(values):
    v = values.ravel()

    # convert object
+    if is_categorical_dtype(values):
+        return v
+
    if dtype == np.object_:


change line 271 to:

elif is_object_dtype(dtype)

pwaller · 2016-04-24T09:03:15Z

Rebased and comments addressed.

pwaller · 2016-04-24T09:23:48Z

I'm getting a test failure I'm not sure how to address and I don't have much time to look into it:

======================================================================
ERROR: test_basic_frame (pandas.io.tests.test_packers.TestNDFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/io/tests/test_packers.py", line 410, in test_basic_frame
    i_rec = self.encode_decode(i)
  File "/home/travis/build/pydata/pandas/pandas/io/tests/test_packers.py", line 75, in encode_decode
    return read_msgpack(p, **kwargs)
  File "/home/travis/build/pydata/pandas/pandas/io/packers.py", line 203, in read_msgpack
    return read(fh)
  File "/home/travis/build/pydata/pandas/pandas/io/packers.py", line 188, in read
    l = list(unpack(fh, encoding=encoding, **kwargs))
  File "_unpacker.pyx", line 459, in pandas.msgpack._unpacker.Unpacker.__next__ (pandas/msgpack/_unpacker.cpp:4709)
  File "_unpacker.pyx", line 390, in pandas.msgpack._unpacker.Unpacker._unpack (pandas/msgpack/_unpacker.cpp:3843)
  File "/home/travis/build/pydata/pandas/pandas/io/packers.py", line 639, in decode
    blocks = [create_block(b) for b in obj[u'blocks']]
  File "/home/travis/build/pydata/pandas/pandas/io/packers.py", line 637, in create_block
    dtype=b[u'dtype'])
  File "/home/travis/build/pydata/pandas/pandas/core/internals.py", line 2518, in make_block
    return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)
  File "/home/travis/build/pydata/pandas/pandas/core/internals.py", line 1903, in __init__
    placement=placement, **kwargs)
  File "/home/travis/build/pydata/pandas/pandas/core/internals.py", line 1314, in __init__
    raise TypeError("values must be {0}".format(self._holder.__name__))
TypeError: values must be Categorical

jreback · 2016-04-25T14:45:43Z

pandas/io/packers.py

@@ -393,6 +402,16 @@ def encode(obj):
                    u'dtype': u(obj.dtype.name),
                    u'data': convert(obj.values),
                    u'compress': compressor}
+
+    elif isinstance(obj, Categorical):
+        return {u'typ': u'categorical',


this needs to be 'category'

pwaller · 2016-04-25T14:59:31Z

Updated again. Thanks for your patience.

jreback · 2016-04-25T15:14:09Z

To fix that error

diff --git a/pandas/io/packers.py b/pandas/io/packers.py
index ba38c8e..f009793 100644
--- a/pandas/io/packers.py
+++ b/pandas/io/packers.py
@@ -266,7 +266,7 @@ def convert(values):

     # convert object
     if is_categorical_dtype(values):
-        return v
+        return values

     if dtype == np.object_:
         return v.tolist()

DOC: support for categoricals in read_msgpack Add TestCategorical test cases Add Catecorical ordered=True ndframe test

pwaller · 2016-04-25T15:35:08Z

Forced again.

jreback · 2016-04-25T15:41:44Z

ok looks good. ping when green.

jreback · 2016-04-25T22:06:59Z

thanks!

pwaller mentioned this pull request Mar 9, 2016

WIP/ENH: allow categoricals in msgpack #12191

Closed

jreback added Enhancement Msgpack Categorical Categorical Data Type labels Mar 9, 2016

jreback added this to the 0.18.1 milestone Mar 9, 2016

chris-b1 reviewed Mar 9, 2016
View reviewed changes

jreback reviewed Mar 9, 2016
View reviewed changes

pwaller force-pushed the msgpack-categorical branch from aa487d1 to 6a3cdce Compare March 9, 2016 17:43

pwaller reviewed Mar 9, 2016
View reviewed changes

chris-b1 mentioned this pull request Mar 9, 2016

BUG: Series with Categorical and dtype #12574

Closed

pwaller force-pushed the msgpack-categorical branch from 6a3cdce to 59cf8fd Compare March 16, 2016 14:50

jreback reviewed Mar 16, 2016
View reviewed changes

pwaller force-pushed the msgpack-categorical branch 2 times, most recently from d5d077b to ecab0d4 Compare March 16, 2016 16:34

jreback reviewed Mar 16, 2016
View reviewed changes

pwaller force-pushed the msgpack-categorical branch from 84c4b6c to a2a0693 Compare March 16, 2016 16:48

jreback reviewed Mar 16, 2016
View reviewed changes

pwaller force-pushed the msgpack-categorical branch from a2a0693 to ae97751 Compare March 16, 2016 16:49

pwaller reviewed Mar 16, 2016
View reviewed changes

pwaller force-pushed the msgpack-categorical branch from 5205852 to 330d327 Compare March 16, 2016 17:02

pwaller force-pushed the msgpack-categorical branch from 116e3d8 to d1792d9 Compare April 12, 2016 20:19

pwaller force-pushed the msgpack-categorical branch 3 times, most recently from fc7e1f1 to 52ea6e4 Compare April 17, 2016 16:23

jreback reviewed Apr 17, 2016
View reviewed changes

pwaller force-pushed the msgpack-categorical branch from 52ea6e4 to e7c4d9d Compare April 24, 2016 09:03

jreback reviewed Apr 25, 2016
View reviewed changes

pwaller force-pushed the msgpack-categorical branch from e7c4d9d to c257021 Compare April 25, 2016 14:59

ENH: allow categoricals in msgpack

ff87282

DOC: support for categoricals in read_msgpack Add TestCategorical test cases Add Catecorical ordered=True ndframe test

pwaller force-pushed the msgpack-categorical branch from c257021 to ff87282 Compare April 25, 2016 15:35

jreback closed this in 2fd0a06 Apr 25, 2016

jankatins mentioned this pull request Apr 25, 2016

ENH: Categorical serialized #7621

Closed

4 tasks

		@@ -23,6 +23,9 @@ Enhancements
		~~~~~~~~~~~~


		- ``read_msgpack`` now supports categoricals (:issue:`12573`)

		@@ -30,7 +30,7 @@ Enhancements
		~~~~~~~~~~~~



		- support for serializing and de-serializing categoricalls with msgpack (:issue:`12573`)

ENH: allow categoricals in msgpack #12573

ENH: allow categoricals in msgpack #12573

Conversation

pwaller commented Mar 9, 2016

jreback commented Mar 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwaller commented Mar 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Mar 16, 2016

pwaller commented Mar 16, 2016

Choose a reason for hiding this comment

pwaller commented Mar 16, 2016

pwaller commented Mar 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Apr 12, 2016

jreback commented Apr 17, 2016

jreback commented Apr 17, 2016

pwaller commented Apr 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwaller commented Apr 24, 2016

pwaller commented Apr 24, 2016

jreback Apr 25, 2016 • edited Loading

Choose a reason for hiding this comment

pwaller commented Apr 25, 2016

jreback commented Apr 25, 2016

pwaller commented Apr 25, 2016

jreback commented Apr 25, 2016

jreback commented Apr 25, 2016

jreback Apr 25, 2016 •

edited

Loading