TabularDataModule: cannot convert series to <class 'int'> when trying to get nuniquevalues

[charitarthchugh]

Describe the bug
https://github.com/manujosephv/pytorch_tabular/blob/da20ed3b26c5464bffdec50ffc8bc2005ae41d65/src/pytorch_tabular/tabular_datamodule.py#L297
It seems that when getting the nuniquevalues, pandas is returning a series instead of an int
To reproduce:
I think this should suffice? I can send the whole notebook as well.

data_config = DataConfig(
    target=["AttackEncoded"],
    continuous_cols=continous_cols,
    categorical_cols=categorical_cols,
)
optimizer_config = OptimizerConfig()

model_config = CategoryEmbeddingModelConfig(
    task="classification",
    layers="42-36-20",  # Number of nodes in each layer
    activation="LeakyReLU",  # Activation between each layers
    learning_rate=1e-3,
)

tabular_model = TabularModel(
    data_config=data_config,
    model_config=model_config,
    optimizer_config=optimizer_config,
    trainer_config=trainer_config,
    verbose=True
)
tabular_model.fit(train=df_train, validation=df_test) #error occurs here

Please let me know if you need anything more to reproduce this.

[charitarthchugh]

I messed around with the code a little bit, got it fixed. Instead of calling nunique() for each of the categorical cols, we can do something along the lines of:

nunique = self.train_dataset.data[config.categorical_cols].nunique().values
categorical_cardinality = [int(x)+1 for x in list(nunique)]

let me know if you want me to create a pr for this.

[manujosephv]

Strange. Because self.train_dataset.data[col] should be a series if col is a string. And if self.train_dataset.data[col] is a series, self.train_dataset.data[col].nunique() should be an int.

And pandas documentation confirms that the return type is int for series.nunique.

Whereas the return type for dataframe.nunique is pd.Series. So, that would happen if col is a list. By any chance did you have a nested list in categorical_cols?

[charitarthchugh]

My categorical_cols are not nested at all. My categorical columns look like this (trying to predict network attacks):

catagorical_cols = [
    "TCP_FLAGS",
    "PROTOCOL",
    "L7_PROTO",
    "CLIENT_TCP_FLAGS",
    "SERVER_TCP_FLAGS",
    "DNS_QUERY_TYPE",
    "ICMP_TYPE",
    "ICMP_IPV4_TYPE",
    "FTP_COMMAND_RET_CODE",
]

In my testing, I found that some of these columns were indeed returning an int, but some were returning a series which makes it more strange.

[manujosephv]

Weird indeed.. but your way seems to be better and handles this quirk.. Please go ahead and raise a PR for this

[charitarthchugh]

Should I change https://github.com/manujosephv/pytorch_tabular/blob/da20ed3b26c5464bffdec50ffc8bc2005ae41d65/src/pytorch_tabular/tabular_datamodule.py#L293 too? If so how should I test it?

[manujosephv]

yes! good catch. Although I dont think that if is used anymore, lets make tha change in both places. In a future version or PR, I'll remove the unnecessary if condition