Allow RLE for bools in v1 pages

[martindurant]

Fixes #884

@jorisvandenbossche , could you please try with this?

Interestingly, there seems to have been no issue with V2 pages. I wonder why the push to change the defaults (and indeed the allowed encodings) for a version that is long superceded?

[martindurant]

I don't think fastparquet will follow suit and write bools as RLE...

[jorisvandenbossche]

It might be worth adding a test for this .. as it doesn't seem to be fixing it for me (well, almost, but values are still inverted). Using the files from the issue:

In [1]: pd.read_parquet("test_bool_pa13_plain.parquet", engine="fastparquet")
Out[1]: 
     col
0   True
1  False
2   True

In [2]: pd.read_parquet("test_bool_pa13_rle.parquet", engine="fastparquet")
Out[2]: 
     col
0  False
1   True
2  False

In [3]: pd.read_parquet("test_bool_pa13_rle.parquet", engine="pyarrow")
Out[3]: 
     col
0   True
1  False
2   True

In [4]: import fastparquet; fastparquet.__version__
Out[4]: '2023.8.1.dev2'

[martindurant]

Are you certain?

In [17]: pd.read_parquet("test_bool_pa13_rle.parquet", engine="fastparquet").equals(pd.read_parquet("test_bool_pa13_rle.parquet", engine="pyarrow"))
Out[17]: True

In [18]: pd.read_parquet("test_bool_pa13_rle.parquet", engine="fastparquet").equals(pd.read_parquet("test_bool_pa13_plain.parquet", engine="pyarrow"))
Out[18]: True

In [19]: pd.read_parquet("test_bool_pa13_plain.parquet", engine="fastparquet").equals(pd.read_parquet("test_bool_pa13_plain.parquet", engine="pyarrow"))
Out[19]: True

[jorisvandenbossche]

Well, you can see what I printed above. That was after doing a pip install git+https://github.com/dask/fastparquet, and didn't further investigate except from what I printed above.

You can check if the files you created are actually using RLE encoding with pq.read_metadata("test_bool_pa13_rle.parquet").row_group(0).column(0).encodings.

The files I was using: test_bool_pa13.zip

[emkornfield]

Interestingly, there seems to have been no issue with V2 pages. I wonder why the push to change the defaults (and indeed the allowed encodings) for a version that is long superceded?

V2 Data pages are actually in a weird state, where most of the benefits. Last I checked, I think Arrow still had some bugs for them. There is also a somewhat stalled effort to try to standardize expectations around the format: apache/parquet-format#164

[wgtmac]

Interestingly, there seems to have been no issue with V2 pages. I wonder why the push to change the defaults (and indeed the allowed encodings) for a version that is long superceded?

V2 Data pages are actually in a weird state, where most of the benefits. Last I checked, I think Arrow still had some bugs for them. There is also a somewhat stalled effort to try to standardize expectations around the format: apache/parquet-format#164

Did you remember any outstanding bug related to data page v2 in the parquet-cpp?

[emkornfield]

Interestingly, there seems to have been no issue with V2 pages. I wonder why the push to change the defaults (and indeed the allowed encodings) for a version that is long superceded?

V2 Data pages are actually in a weird state, where most of the benefits. Last I checked, I think Arrow still had some bugs for them. There is also a somewhat stalled effort to try to standardize expectations around the format: apache/parquet-format#164

Did you remember any outstanding bug related to data page v2 in the parquet-cpp?

Sorry for the late reply, doing a quick audit appears that they might all be fixed (I think row count might have been the last one).