ENH: add to_xarray conversion method #11972

jreback · 2016-01-06T16:19:35Z

supersedes #11950
xref #10000

using xarray >= 07.0

In [10]: Series([1,2,3]).to_xarray()
Out[10]: 
<xarray.DataArray (index: 3)>
array([1, 2, 3])
Coordinates:
  * index    (index) int64 0 1 2

In [11]: DataFrame({'A' : [1,2,3], 'B' : ['foo','bar','baz']}).to_xarray()
Out[11]: 
<xarray.Dataset>
Dimensions:  (index: 3)
Coordinates:
  * index    (index) int64 0 1 2
Data variables:
    A        (index) int64 1 2 3
    B        (index) object 'foo' 'bar' 'baz'

In [14]: Panel(np.random.randn(2, 3, 4)).to_xarray()
Out[14]: 
<xarray.DataArray (items: 2, major_axis: 3, minor_axis: 4)>
array([[[ 0.23726039,  0.44636322,  0.04425575,  1.06178388],
        [-0.58236405,  0.62602167,  0.36156612, -0.12687913],
        [-0.67854107,  0.72270844, -1.15402631, -1.89758909]],

       [[ 0.42948622,  0.93227075, -0.13943692,  0.83043343],
        [ 1.0355157 , -0.30532004, -0.16337369, -0.29283026],
        [ 0.65199775, -0.11131818,  0.37853134,  1.56620844]]])
Coordinates:
  * items       (items) int64 0 1
  * major_axis  (major_axis) int64 0 1 2
  * minor_axis  (minor_axis) int64 0 1 2 3

TODO:

examples in doc-string
how-to-convert/use xarray

jreback · 2016-01-06T16:20:24Z

since I renamed the branch now really sure if there was a way to avoid creating a new PR. oh well.

max-sixty · 2016-01-06T17:09:22Z

pandas/core/generic.py

+        if self.ndim == 1:
+            return xray.DataArray.from_series(self)
+        elif self.ndim == 2:
+            return xray.Dataset.from_dataframe(self)


@shoyer should this be different from just xray.DataArray(series) or xray.Dataset(df)?
This might be an xray change; seems a bit off to have special handling in pandas though

best would be: xarray.from_pandas(...) (and you guys handle the construction).

Though ok here as well. as > 2 nd we want to have easy transition for current Panel users.

jreback · 2016-01-06T17:09:59Z

@shoyer ping when releasing new xarray. I am going to min-version this to what you release.

jreback · 2016-01-06T17:13:49Z

@MaximilianR it would be quite helpful if you can post / write a mini-doc (where we can incorporate in a doc-string / document) on how to migrate Panel -> xarray, with use-cases / how-tos etc.

can obviously also add to here: http://xray.readthedocs.org/en/stable/pandas.html

max-sixty · 2016-01-06T17:47:40Z

@jreback Yes I can have a go at that. I could see that being fairly short given the existing docs - just a couple of examples of Panel migration. Is that what you envision?

jreback · 2016-01-06T17:49:16Z

@MaximilianR yep, the migration part as well as 'working' with them (which might require another section). E.g. imagine you have Panels and now you have xarray objects. So they want to perform 'typical' operations there (we will prob need a section in the pandas docs when I do the deprecation PR's) as well (so will effectively just copy what is there).

jreback · 2016-01-09T00:23:16Z

pandas/core/generic.py

+
+        # > 2 dims
+        coords = [(a, self._get_axis(a)) for a in self._AXIS_ORDERS]
+        return xray.DataArray(self,


@shoyer you didn't like this?

as an aside, do you want to put this routine in xarray and I'll just call xray.DataArray.from_pandas(self)?

jreback · 2016-01-26T21:25:27Z

@shoyer @MaximilianR

updated to use xarray 0.7.0

works with all index types including MultiIndex! nice!
Panel <-> xarray <-> Panel works nicely!
>3 dim doesn't convert back to pandas (but no problem)

jreback · 2016-01-26T21:26:27Z

I think you need to have RTD use xarray? (and have a pointer from xray)?

shoyer · 2016-01-26T21:33:07Z

@jreback I'm suggesting http://xarray.pydata.org/ instead... it turns out you can't change RTD stubs, so we're stuck with http://xray.readthedocs.org for now.

shoyer · 2016-01-26T21:34:50Z

pandas/core/generic.py

+
+        # > 2 dims
+        coords = [(a, self._get_axis(a)) for a in self._AXIS_ORDERS]
+        return xarray.DataArray(self,


I wonder if we should add Panel4D support to the DataArray constructor so this could just be xarray.DataArray(self)... OTOH I'm pretty sure Panel4D is barely used

yeh its no big deal. you could always add later if its real useful.

jreback · 2016-01-27T01:14:54Z

comments

@wesm @jorisvandenbossche @TomAugspurger @sinhrks @cpcloud

jreback · 2016-01-27T01:19:44Z

pandas/tests/test_generic.py

+                               expected,
+                               check_index_type=False)
+
+        # not implemented


@shoyer

what am I doing wrong here?

In [1]: df = DataFrame({'a': list('abc'), ...: 'b': list(range(1, 4)), ...: 'c': np.arange(3, 6).astype('u1'), ...: 'd': np.arange(4.0, 7.0, dtype='float64'), ...: 'e': [True, False, True], ...: 'f': pd.Categorical(list('abc')), ...: 'g': pd.date_range('20130101', periods=3), ...: 'h': pd.date_range('20130101', ...: periods=3, ...: tz='US/Eastern')} ...: ) In [2]: df.to_xarray() Out[2]: <xarray.Dataset> Dimensions: (index: 3) Coordinates: * index (index) int64 0 1 2 Data variables: a (index) object 'a' 'b' 'c' b (index) int64 1 2 3 c (index) uint8 3 4 5 d (index) float64 4.0 5.0 6.0 e (index) bool True False True f (index) category 'a' 'b' 'c' g (index) datetime64[ns] 2013-01-01 2013-01-02 2013-01-03 h (index) datetime64[ns] 2013-01-01T05:00:00 2013-01-02T05:00:00 ... In [3]: df = DataFrame({'a': list('abc'), ...: 'b': list(range(1, 4)), ...: 'c': np.arange(3, 6).astype('u1'), ...: 'd': np.arange(4.0, 7.0, dtype='float64'), ...: 'e': [True, False, True], ...: 'f': pd.Categorical(list('abc')), ...: 'g': pd.date_range('20130101', periods=3), ...: 'h': pd.date_range('20130101', ...: periods=3, ...: tz='US/Eastern')} ...: ) In [4]: df.index = pd.MultiIndex.from_product([['a'], range(3)], ...: names=['one', 'two']) In [5]: df Out[5]: a b c d e f g h one two a 0 a 1 3 4 True a 2013-01-01 2013-01-01 00:00:00-05:00 1 b 2 4 5 False b 2013-01-02 2013-01-02 00:00:00-05:00 2 c 3 5 6 True c 2013-01-03 2013-01-03 00:00:00-05:00 In [6]: df.to_xarray() ValueError: dimensions ('one', 'two') must have the same length as the number of data dimensions, ndim=1

I'm going to blame this one (in part) on pandas's Categorical:

ipdb> series.values [a, b, c] Categories (3, object): [a, b, c] ipdb> series.values.reshape(shape) [a, b, c] Categories (3, object): [a, b, c] ipdb> shape [1, 3]

Instead of erroring, it ignores the reshape argument (to 2D).

This certainly needs a fix in xarray, too, though -- we should use np.asarray when converting DataFrame columns rather than .values.

ok, going to merge as is then. do you want me to create an issue on xarray for this?

pydata/xarray#737

yes, let's merge -- I'll fix that in the next xarray bug fix release

jreback · 2016-01-27T14:26:33Z

@shoyer note that conda has updated to include xarray

jorisvandenbossche · 2016-01-27T23:30:44Z

doc/source/install.rst

@@ -245,6 +245,7 @@ Optional Dependencies
 * `Cython <http://www.cython.org>`__: Only necessary to build development
  version. Version 0.19.1 or higher.
 * `SciPy <http://www.scipy.org>`__: miscellaneous statistical functions
+* `xarray <http://xarray.readthedocs.org>`__: pandas like handling for > 2 dims. Version 0.7.0 or higher is recommeded.


recommeded -> recommended

maybe also mention for what functionality this optional dependency is needed

max-sixty · 2016-01-28T22:15:04Z

@jreback @shoyer I've done a (currently very short & basic) draft of how to move from pandas to xarray, with the intention you guys can offer some feedback and we can iterate.

Do you want me to do a separate PR into pandas? Or should this go into the 'from Pandas' section of the xarray docs, with a link from pandas' What's New?

jreback · 2016-01-28T22:18:30Z

@MaximilianR I think best is to add to the xray docs, which we will link to.

jorisvandenbossche · 2016-01-28T22:35:37Z

Personally I think this maybe more belongs in the pandas docs. I mean, the xarray docs could certainly have a section about how it interplays with pandas (as it already has, http://xarray.pydata.org/en/stable/pandas.html), but as this will also be about how to move from the deprecated panel to xarray, it feels more at place in the pandas docs (I don't think xarray docs should handle about deprecated pandas features).

Anyhow, not that important, as just link to wherever it is located.

jreback · 2016-01-28T22:38:44Z

@jorisvandenbossche right, I have a note about that above, e.g. about how to transition from using a Panel to a DataArray.

So @MaximilianR can certainly add to the pandas docs (but should obviously add to xarray as well)

closes pandas-dev#11972

shoyer · 2016-02-10T17:43:46Z

doc/source/whatsnew/v0.18.0.txt

@@ -271,6 +271,7 @@ In addition, ``.round()``, ``.floor()`` and ``.ceil()`` will be available thru t
   s
   s.dt.round('D')

+<<<<<<< 6693a723aa2a8a53a071860a43804c173a7f92c6


merge conflict

yep fixing as I am merging
thxs

jreback · 2016-02-10T17:46:24Z

@shoyer was just about to merge. anything else besides those 2 comments?

(and the linked issue for more docs)

jreback · 2016-02-10T21:30:00Z

@jorisvandenbossche I think I broke the doc build, but not sure how: https://travis-ci.org/pydata/pandas/jobs/108337474

jorisvandenbossche · 2016-02-10T22:07:32Z

pandas/core/generic.py

+
+        See Also
+        --------
+        `xarray docs <http://xarray.pydata.org/en/stable/>`__


Possibly it is this line where it is choking on, as in a See also should come a python object

I would just make a 'Note' section of it instead of See also, and then say "See also the xarray docs .."

supersedes pandas-dev#11950 xref pandas-dev#10000 Author: Jeff Reback <jeff@reback.net> Closes pandas-dev#11972 from jreback/xarray and squashes the following commits: 85de0b7 [Jeff Reback] ENH: add to_xarray conversion method

jreback added Enhancement Reshaping Concat, Merge/Join, Stack/Unstack, Explode Compat pandas objects compatability with Numpy or Python functions labels Jan 6, 2016

jreback added this to the 0.18.0 milestone Jan 6, 2016

max-sixty reviewed Jan 6, 2016
View reviewed changes

jreback force-pushed the xarray branch 2 times, most recently from c03bdd4 to 2a007b5 Compare January 7, 2016 01:17

jreback reviewed Jan 9, 2016
View reviewed changes

jreback force-pushed the xarray branch from 2a007b5 to c3bc471 Compare January 17, 2016 21:54

jreback force-pushed the xarray branch 2 times, most recently from bea8f9d to 2307129 Compare January 26, 2016 21:20

shoyer reviewed Jan 26, 2016
View reviewed changes

jreback force-pushed the xarray branch from 2307129 to 9dd52e8 Compare January 26, 2016 21:45

jreback reviewed Jan 27, 2016
View reviewed changes

jreback force-pushed the xarray branch from f7c03d4 to 07e1780 Compare January 27, 2016 14:25

jorisvandenbossche reviewed Jan 27, 2016
View reviewed changes

jreback force-pushed the xarray branch from 07e1780 to 59cde69 Compare January 28, 2016 22:39

jreback force-pushed the xarray branch from 59cde69 to 2be091f Compare January 29, 2016 17:04

ENH: add to_xarray conversion method

85de0b7

closes pandas-dev#11972

jreback force-pushed the xarray branch from 2be091f to 85de0b7 Compare February 9, 2016 23:39

jreback mentioned this pull request Feb 10, 2016

DOC: show how to use to_xarray then use xarray to perform analysis on Panels #12283

Closed

shoyer reviewed Feb 10, 2016
View reviewed changes

jreback closed this in 358da56 Feb 10, 2016

jorisvandenbossche reviewed Feb 10, 2016
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add to_xarray conversion method #11972

ENH: add to_xarray conversion method #11972

jreback commented Jan 6, 2016

jreback commented Jan 6, 2016

max-sixty Jan 6, 2016

jreback Jan 6, 2016

jreback commented Jan 6, 2016

jreback commented Jan 6, 2016

max-sixty commented Jan 6, 2016

jreback commented Jan 6, 2016

jreback Jan 9, 2016

jreback commented Jan 26, 2016

jreback commented Jan 26, 2016

shoyer commented Jan 26, 2016

shoyer Jan 26, 2016

jreback Jan 26, 2016

jreback commented Jan 27, 2016

jreback Jan 27, 2016

jreback Jan 30, 2016

shoyer Jan 31, 2016

jreback Feb 9, 2016

shoyer Feb 10, 2016

jreback Feb 10, 2016

jreback commented Jan 27, 2016

jorisvandenbossche Jan 27, 2016

jorisvandenbossche Jan 27, 2016

jreback Jan 28, 2016

max-sixty commented Jan 28, 2016

jreback commented Jan 28, 2016

jorisvandenbossche commented Jan 28, 2016

jreback commented Jan 28, 2016

shoyer Feb 10, 2016

jreback Feb 10, 2016

jreback commented Feb 10, 2016

jreback commented Feb 10, 2016

jorisvandenbossche Feb 10, 2016

jreback Feb 10, 2016

ENH: add to_xarray conversion method #11972

ENH: add to_xarray conversion method #11972

Conversation

jreback commented Jan 6, 2016

jreback commented Jan 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jan 6, 2016

jreback commented Jan 6, 2016

max-sixty commented Jan 6, 2016

jreback commented Jan 6, 2016

Choose a reason for hiding this comment

jreback commented Jan 26, 2016

jreback commented Jan 26, 2016

shoyer commented Jan 26, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jan 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jan 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

max-sixty commented Jan 28, 2016

jreback commented Jan 28, 2016

jorisvandenbossche commented Jan 28, 2016

jreback commented Jan 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 10, 2016

jreback commented Feb 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment