BUG: fix json segfaults

[Komnomnomnom]

closes #11473
closes #10778
closes #11299

• tests added / passed
• vbench / asv ok
• windows tests pass
• valgrind clean
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

This fixes several potential segfaults in the json code:

• time objects were freed and potentially garbage collected before their data had been read to_json segfaults with timezone-aware datetimes #11473. Was also segfaulting if an exception was raised during conversion (e.g. when using dateutil timezones).
• 0d arrays were not being handled corectly (needed goto INVALID) pandas.io.tests.test_json.test_ujson.NumpyJSONTests testOdArray segfaults under python2.7/3.5 in 0.17.0 #11299
• all blocks were assumed to be ndarrays df.to_json segfaults with categorical column types #10778

Fixing #10778 means non-ndarray blocks are now supported (although I think at present category is the only one?).

Not tested on windows yet. Seeing some unrelated travis failures on my fork (msgpack), is that normal?

[jreback]

non-ndarray values are the extension types

• categorical
• datetime w/tz
• sparse (which are actually a ndarray sub-class); not a big deal ATM.

[jreback]

for tz (that's on 0.18.0), but add as a test for your PR.

In [5]: df = DataFrame({'A' : pd.date_range('20130101',periods=3,tz='US/Eastern'), 'B' : pd.date_range('20130101',periods=3)})

In [6]: df
Out[6]: 
                          A          B
0 2013-01-01 00:00:00-05:00 2013-01-01
1 2013-01-02 00:00:00-05:00 2013-01-02
2 2013-01-03 00:00:00-05:00 2013-01-03

In [7]: df.dtypes
Out[7]: 
A    datetime64[ns, US/Eastern]
B                datetime64[ns]
dtype: object

In [8]: df.to_json()
/Users/jreback/miniconda/envs/coursebuild/bin/python.app: line 3: 49679 Segmentation fault: 11  /Users/jreback/miniconda/envs/coursebuild/python.app/Contents/MacOS/python "$@"

[Komnomnomnom]

welp, ok I'll take another look.

[jreback]

@Komnomnomnom thanks!

[jreback]

pls update when you can

[Komnomnomnom]

The json code assumed in several places that the .values attribute would always return an ndarray, which of course is not the case for any of the extension types. It then makes extensive use of the numpy C-API to iterate over the assumed ndarray, which is a terrible idea for non-ndarray objects (or even ndarray subclasses in the case of sparse).

I added a helper function to handle this a bit better. It initially attempts to use the .values attribute and falls back to get_values() if necessary. If get_values() fails an exception is raised. Subclasses of ndarray are not accepted. Seems to work ok now for sparse and categorical, but it could be better as some coercion / copying is needed.

datetime w/tz no longer segfaults but it is not JSON serializable as both values and get_values() do not return an ndarray. I'm inclined to think this is a bug?

In [35]: df
Out[35]: 
                          A
0 2013-01-01 00:00:00-05:00
1 2013-01-02 00:00:00-05:00
2 2013-01-03 00:00:00-05:00

In [36]: df.values
Out[36]: 
DatetimeIndex(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00',
               '2013-01-03 00:00:00-05:00'],
              dtype='datetime64[ns, US/Eastern]', freq='D')

In [37]: df.get_values()
Out[37]: 
DatetimeIndex(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00',
               '2013-01-03 00:00:00-05:00'],
              dtype='datetime64[ns, US/Eastern]', freq='D')

In [38]: df._data.blocks[0].values
Out[38]: 
DatetimeIndex(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00',
               '2013-01-03 00:00:00-05:00'],
              dtype='datetime64[ns, US/Eastern]', freq='D')

In [39]: df._data.blocks[0].get_values()
Out[39]: 
DatetimeIndex(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00',
               '2013-01-03 00:00:00-05:00'],
              dtype='datetime64[ns, US/Eastern]', freq='D')

In [40]: type(df.values)
Out[40]: pandas.tseries.index.DatetimeIndex

In [41]: df.dtypes
Out[41]: 
A    datetime64[ns, US/Eastern]
dtype: object

In [42]: type(df._data.blocks[0])
Out[42]: pandas.core.internals.DatetimeTZBlock

In [43]: df._data.blocks[0].dtype
Out[43]: datetime64[ns, US/Eastern]

[jreback]

@Komnomnomnom no that is all as expected. Internally the DatetimeTZBlock holds a DatetimeIndex rather than a ndarray (though it is duck-like a ndarray).

How do you think we should serialize this?

In general converting to a ndarray is not a great thing as these are now UTC times (don't be fooled as to the repr, they have been converted).

In [44]: np.array(df._data.blocks[1].values)
Out[44]: 
array(['2013-01-01T05:00:00.000000000', '2013-01-02T05:00:00.000000000',
       '2013-01-03T05:00:00.000000000'], dtype='datetime64[ns]')

converting to a list will preserve everything

In [42]: list(df._data.blocks[1].values)
Out[42]: 
[Timestamp('2013-01-01 00:00:00-0500', tz='US/Eastern', offset='D'),
 Timestamp('2013-01-02 00:00:00-0500', tz='US/Eastern', offset='D'),
 Timestamp('2013-01-03 00:00:00-0500', tz='US/Eastern', offset='D')]

This converts to string, not sure how useful this is to you (we use this for csv writing).

In [54]: df._data.blocks[1].values.to_native_types()
Out[54]: 
array(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00',
       '2013-01-03 00:00:00-05:00'], dtype=object)

This is probably the most general thing you can do (and will coerce all things correctly). So if you find a block that doesn' thave .values as something you recoginize you can just do this.

In [57]: df._data.blocks[1].get_values(dtype=object)
Out[57]: 
array([Timestamp('2013-01-01 00:00:00-0500', tz='US/Eastern'),
       Timestamp('2013-01-02 00:00:00-0500', tz='US/Eastern'),
       Timestamp('2013-01-03 00:00:00-0500', tz='US/Eastern')], dtype=object)

[jreback]

lmk when you are ready for review

[jreback]

@Komnomnomnom hows this coming

[Komnomnomnom]

@jreback should have some time this evening

[Komnomnomnom]

(on UK time)

[jreback]

@Komnomnomnom great thanks!

[Komnomnomnom]

@jreback TZ block is supported now. Note atm the JSON handler always converts to UTC when encoding and this seems to be the case for any time object. It would be nice if it honoured TZ timestamps (see #12997) but I've added some regression tests to ensure this behaviour remains consistent in the meantime.

So the behaviour now is to repeatedly call .values on an object until a concrete ndarray is returned with a fall-back to get_values() if that doesn't work. So the resulting calls end up being:

• DatetimeTZBlock -> tzblock.values.values
• SparseBlock -> spblock.values.values
• CategorocalBlock -> catblock.get_values()

I just want to give it a run through vbench, otherwise I'm happy with it.

Let me know if you have any comments. Am I missing something regarding the TZ behaviour?

[jreback]

@Komnomnomnom no problem with coercing to UTC, that's what we do anyone when transforming to any non-tz supported.

note that this is numpy 1.11, under older ones you would get a differernt DISPLAY (but the actual values are correct).

In [1]: s = Series(pd.date_range('20130101',periods=3,tz='US/Eastern'))

In [2]: s
Out[2]: 
0   2013-01-01 00:00:00-05:00
1   2013-01-02 00:00:00-05:00
2   2013-01-03 00:00:00-05:00
dtype: datetime64[ns, US/Eastern]

In [3]: s.values
Out[3]: 
array(['2013-01-01T05:00:00.000000000', '2013-01-02T05:00:00.000000000',
       '2013-01-03T05:00:00.000000000'], dtype='datetime64[ns]')

[jreback]

add the issue for the datetime w/tz fixes?

[Komnomnomnom]

DatetimeTZBlock was covered by #10778 mentioned here. (#11473 is a separate issue related to datetime objects)

[Komnomnomnom]

@jreback is there CI support for windows? If not is there any chance you could run the JSON tests on your windows box?

[jreback]

yes, you can sign up for http://www.appveyor.com/.

we run this on master, but you can run on a personal (free) accct.

I will run now (on my local vm).

[jreback]

@Komnomnomnom passes!

[jreback]

side issue. for conformity I was going to rename:

``pandas\io\tests\test_json` to pandas\io\tests\json\ (also have to make a tiny change in `setup.py`. Want to do that here?

[jreback]

or I can do it in a followup.

[Komnomnomnom]

Great!

np I'll rename it now

[jreback]

@Komnomnomnom this looks fine. ping me when your travis run is green (no need to wait for master)

[jreback]

@Komnomnomnom

also move the test_json_norm.py in there as well.

[jreback]

@Komnomnomnom looks good. ping when green.

[Komnomnomnom]

Some unrelated msgpack and pickle failures (during version comparison?), not sure what's going on there. Otherwise clean:

https://travis-ci.org/Komnomnomnom/pandas/builds/125890385

[jreback]

@Komnomnomnom thanks. yeah those happen when it doesn't get the tag information, not really sure why it doesnt' always work.

[jreback]

thanks for the fixes!