ARROW-1997: [C++/Python] Ignore zero-copy-option in to_pandas when strings_to_categorical is True

[Licht-T]

This closes ARROW-1997.

The problem is

>>> import pandas as pd
>>> import pyarrow as pa
>>> 
>>> df = pd.DataFrame({
... 'Foo': ['A', 'A', 'B', 'B']
... })
>>> table = pa.Table.from_pandas(df)
>>> df = table.to_pandas(strings_to_categorical=True)
<class 'pandas.core.categorical.Categorical'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "table.pxi", line 1043, in pyarrow.lib.Table.to_pandas
  File "pyarrow/pandas_compat.py", line 535, in table_to_blockmanager
    blocks = _table_to_blocks(options, block_table, nthreads, memory_pool)
  File "pyarrow/pandas_compat.py", line 629, in _table_to_blocks
    return [_reconstruct_block(item) for item in result]
  File "pyarrow/pandas_compat.py", line 436, in _reconstruct_block
    ordered=item['ordered'])
  File "/home/rito/miniconda3/envs/pyarrow-dev-27/lib/python2.7/site-packages/pandas/core/categorical.py", line 624, in from_codes
    raise ValueError("codes need to be between -1 and "
ValueError: codes need to be between -1 and len(categories)-1

When strings_to_categorical=True, the categorical index is newly created in to_pandas procedure.
But, this passes data to Python by zero-copy, so the array is deallocated.
https://github.com/Licht-T/arrow/blob/be58af6dd0333652abbe2333ee5968df3f2e371f/cpp/src/arrow/python/arrow_to_pandas.cc#L1040

[xhochy]

@Licht-T The code looks good but I fail to understand the initial problem and thus cannot really understand what the change should actually do. Can you explain it a bit more?

[Licht-T]

Thanks @xhochy, added the PR comment!

[xhochy]

Now I get it 👍

[wesm]

Is it possible to add a unit test that hits this code path, or is there a test already that does?

[wesm]

This error message is a little bit unclear, I will tweak

[Licht-T]

Thanks @wesm, added tests!

[wesm]

+1. Will merge once the build is passing