Value Error when categorical column is of dtype "Categorical"

[manujosephv]

Describe the bug
When the categorical columns are of dtype Categorical in pandas, it throws this error.
ValueError: Cannot setitem on a Categorical with a new category, set the categories first

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[DominikFilipiak]

Workaround: make sure that in your data frame all categorical columns are casted to string.

for c in cols_category:
    df[c] = df[c].astype("string") 
    df[c].fillna("na", inplace=True)

[snehilchatterjee]

Hi, I managed to fix the issue by modifying src/pytorch_tabular/tabular_datamodule.py and src/pytorch_tabular/categorical_encoders.py by adding the following:

category_cols = data[cat_cols].select_dtypes(include='category').columns
data[category_cols] = data[category_cols].astype('object')

Can I open a PR for this, or are further adjustments needed in other files that I might have missed?

[snehilchatterjee]

Hey @manujosephv is this still open ? should I make a PR for this ?

[manujosephv]

Yes.. This is still open... Feel free to raise a PR

[snehilchatterjee]

@manujosephv I have raised the PR. Let me know if any changes are needed before merging. Also while you are at it I'd appreciate it if you could also review my previous PR.