Parquet, boolean dataframes & reindex

[0x0L]

Hello

Code Sample, a copy-pastable example if possible

# pandas 0.25.0
# pyarrow 0.14.1
import pandas as pd

df = pd.DataFrame(dtype=bool, index=range(100))
df['A'] = True
df['B'] = False
df['C'] = True
df['D'] = False
df['E'] = True
df['F'] = False

index = [0, 2, 3, 4, 5, 6, 7, 8, 10, 11]
columns = ['A', 'C', 'D', 'F']
dk = df.reindex(columns=columns, index=index)

dk.to_parquet('/tmp/test.parquet')
dg = pd.read_parquet('/tmp/test.parquet')
assert dg.equals(dk)  # ==> assert error

The saved dataframe is totally corrupted with most values now being False.

Problem description

This issue only appears when using boolean.
The issue is not there when reindexing only columns or only index or when doing both in two steps ie df.reindex(columns=columns).reindex(index=index). df.loc[index, columns] is also fine.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.3.final.0
python-bits : 64
OS : Linux
OS-release : 4.14.128-112.105.amzn2.x86_64
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.0
numpy : 1.16.4
pytz : 2019.1
dateutil : 2.8.0
pip : 19.2.1
setuptools : 41.0.1
Cython : 0.29.13
pytest : 5.0.1
hypothesis : 4.32.2
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.4.0
html5lib : 1.0.1
pymysql : None
psycopg2 : 2.8.3 (dt dec pq3 ext lo64)
jinja2 : 2.10.1
IPython : 7.7.0
pandas_datareader: None
bs4 : 4.8.0
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : 4.4.0
matplotlib : 3.1.1
numexpr : 2.6.9
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.14.1
pytables : None
s3fs : 0.3.1
scipy : 1.3.0
sqlalchemy : 1.3.6
tables : 3.5.2
xarray : 0.12.3
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : None

[TomAugspurger]

Is this likely an issue in pandas or pyarrow?

[0x0L]

Hi @TomAugspurger
Since the bug requires the call to reindex I felt it was more relevant to post here.

[TomAugspurger]

Possibly. Pandas doesn't do a whole lot with parquet IO though. The bulk of the work is done in the engine library.

…

On Thu, Aug 22, 2019 at 10:23 AM nullptr ***@***.***> wrote: Hi @TomAugspurger <https://github.com/TomAugspurger> Since the bug requires the call to reindex I felt it was more relevant to post here. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28090?email_source=notifications&email_token=AAKAOISJCUMKWWCKPEQK7VDQF2VOJA5CNFSM4IOVPSKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD45OINQ#issuecomment-523953206>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIQBCGKVUYNEI6TI55LQF2VOJANCNFSM4IOVPSKA> .

[jorisvandenbossche]

This seems a serious bug in the pandas->arrow conversion of Arrow. It is not related to parquet, since just converting to an Arrow table already gives the wrong values:

In [13]: table = pa.Table.from_pandas(dk)

In [14]: table.column(0)
Out[14]: 
<pyarrow.lib.ChunkedArray object at 0x7fd09ce6c120>
[
  [
    true,
    false,
    false,
    false,
    false,
    false,
    false,
    false,
    true,
    false
  ]
]

I will open an issue on the Arrow side.

[jorisvandenbossche]

I opened https://issues.apache.org/jira/browse/ARROW-6325

@0x0L Thanks a lot for the report!

Closing this issue, as it is something to fix on the Arrow side.