to_parquet when column with only null values

[odovad]

I did create a file named 'system_data.txt' with the following content :

SYSTEM_DATE,col1 
Test1,Test1
Test2,Test2
,Test3

This code produces an error :

import dask.dataframe as dd
df = dd.read_csv('system_data.txt',  dtype={"SYSTEM_DATE":'str', 'col1':'str'})
df = df.repartition(npartitions=3)
df.to_parquet('test')

What happened:
RuntimeError: AppendRowGroups requires equal schemas.
I tried with fastparquet, and there is no errors.

What you expected to happen:
The parquet file should be created without errors.

Environment:

• Dask version: 2022.2.1
• Python version: 3.8
• Pyarrow : 7.0.0
• Operating System: Linux
• Install method : pip

[jsignell]

Thanks for opening this issue. I suspect this has to do with one of the partitions (the last one) having an entirely nan column. If you look at the error message, you'll see that it recommends using to_parquet(..., schema="infer"). That worked for me to solve the issue. Does that also work for you?

RuntimeError: Schemas are inconsistent, try using `to_parquet(..., schema="infer")`, or pass an explicit pyarrow schema. Such as `to_parquet(..., schema={"column1": pa.string()})`

ping @rjzamora just to make sure I got that right.

[odovad]

Hi @jsignell, I did try to use schema="infer"but the computation was taking too long (many additional tasks on the real example). Possible it would have worked but I didn't wait until the end.

[jsignell]

Ah in that case maybe passing an explicit schema (also mentioned in the error message) would be preferable.

[odovad]

@jsignell thanks, yes I'll try passing explicit schema. I found it was weird in a way to have dtype issues since in the real example I passed all dtypes when calling read_csv (so before calling to_parquet)

[scharlottej13]

@odovad, just checking in, did passing an explicit schema work for you?

[odovad]

@scharlottej13 yes, it worked when passing explicit schema. I thought that the dtypes that we see on the dask dataframe should be the one used for all partitions.

[bryanwweber]

This seems to be resolved by setting the explicit schema. @odovad Please feel free to reopen if there's more that needs to be done!