Unable to write from csv to parquet + register glue after 0.0.2

[josecw]

i am having issue when writing data consists of null to parquet.
Following error message is returned:

f"Unsupported Pyarrow type for column {name}: {dtype}")
awswrangler.exceptions.UnsupportedType: Unsupported Pyarrow type for column xxxxxxx: null

The same code work fine in version 0.0.2.

[igorborgest]

@BellyTheMagnificent

That was deliberated and the reason is because Pandas has a too generic "data type" named object. Pandas object columns can be string, dates, etc, etc.
We can handle this object column fine inferring the types of theses objects inside the values, Pyarrow does that like a charm.
So the real problem starts when we have a completely empty object column, because we don't have anything to infer.
Before 0.0.3 we were simply considering all object columns as string on Athena and this isn't the best approach. From 0.0.3 and beyond we are inferring from the columns values and throwing a exception in case of a completely null column.

To work with empty object columns you can explicitly set the expected Athena data type for the target table doing:

import awswrangler
import pandas as pd

dataframe = pd.DataFrame({
    "col": [1, 2],
    "col_string_null": [None, None],
    "col_date_null": [None, None],
})
session = awswrangler.Session()
session.pandas.to_parquet(
    dataframe=dataframe,
    database="DATABASE",
    path=f"s3://...",
    cast_columns={
      "col_string_null": "string",
      "col_date_null": "date"
    })

Btw, I will also improve the documentation and the exception message to help others.