Integer categoricals being sampled as strings instead of integer values

[LihuaXiong2020]

• SDV version: 0.4.0
• Python version: 3.6.8
• Operating System: Windows

Description & What I did

• Specified some columns of dataframe to be Categorical in metadata;
• after SDV is trained and the sampled data is generated, run evaluate(metadata, orig_data, sample_data)
• Type Error is raised complaining "ufunc 'isnan' not supported for input types..."
• Debugged and found that the sampled categorical columns are stored as "object", which needs to be transformed into a numerical type manually before calling evaluate()

[csala]

Hi @LihuaXiong2020 ! Thanks for reporting this.

If I understand this correctly, the problem is not really about sdmetrics but rather about "Integer categoricals being sampled as strings instead of integer values". Would you mind updating the title accordingly?

I can actually reproduce it like this:

import pandas as pd

from sdv import Metadata, SDV

df = pd.DataFrame({
    'int': [1, 2, 1, 3],
})
orig_data = {'a': df}
fields_metadata = {
    'int': {
        'type': 'categorical'
    },
}
metadata = Metadata()
metadata.add_table('a', df, fields_metadata=fields_metadata)

sdv = SDV()
sdv.fit(metadata, orig_data)
print(sdv.sample('a', sample_children=False).dtypes)

Which outputs:

int    object
dtype: object

As a technical note, the changes to fix this will need to be made in two places:

• RDT: Make the CategoricalEncoder learn the column dtype and restore it back when reverse transforming.
• SDV: Remove the dropna().astype(dtype) (the entire loop) inside the Metadata.reverse_transform which is actually unnecessary and incorrectly transforms all the categorical columns to object.

[LihuaXiong2020]

Hi @csala, thanks for looking into it. Yes, I agree that the problem should be solved by your updated code to learn to original dtype.

[JagdishKolhe]

I am too getting error " Type Error is raised complaining "ufunc 'isnan' not supported for input types...""

Not sure if it is related to this issue. I am using SDV 0.4.5

[csala]

I am too getting error " Type Error is raised complaining "ufunc 'isnan' not supported for input types...""

Not sure if it is related to this issue. I am using SDV 0.4.5

Hi @JagdishKolhe this sounds like it could be a separated topic.

Would you mind opening a new issue with all the details about what you executed and the output that you obtained?