Use own dataset

[Arrmlet]

• SDV version: 0.4.1
• Python version: 3.8
• Operating System: Linux

Description

Hello everybody,
I am enjoy SDV package, I have tried to generate synthetic data using demo models.
Owing to this, I have tried to load my own dataset( csv file), but I cannot find a way how to do this, I am ensured that I could do this.
Can somebody help me with that, or share some documentation where I could find this.

Regards,
arrmlet

What I Did

I cannot imagine what to do.

[csala]

Hello @Arrmlet

If you have your data stored in a CSV file you should be able to load it using pandas.

In most cases you would only need these two lines:

import pandas as pd

your_dataframe = pd.read_csv('path/to/your.csv')

But in some cases you might need to tweak the read_csv call with different arguments to fully adapt it to your data format.

Here you should be able to find all the details and tutorials about loading the data with pandas: https://pandas.pydata.org/docs/getting_started/intro_tutorials/02_read_write.html

[Arrmlet]

Hello @csala,
Thanks for response, I have tried this code example:

from sdv import SDV
from sdv import Metadata
import pandas as pd


# Load csv file into pandas dataframe
df = pd.read_csv('filepath')

sdv = SDV()

# Generate a metadata object
metadata = Metadata()
metadata.add_table('test_df')
tables = {'test_df': df}
# Train the model
sdv.fit(metadata, tables)

But when I run this code I got an error:

MetadataError: Unexpected column in table test_df: column_name

I think I do not creating a metadata object in right way.

Thank you for your job,
arrmlet

[csala]

# Generate a metadata object
metadata = Metadata()
metadata.add_table('test_df')

The problem is that you are only passing the name of the table to the metadata instance, so it does not have the chance to learn about which columns it should expect your dataframe to have. To make it work you will have to pass both the name and the data:

metadata.add_table(name='test_df', data=df)

If you change this line, the rest should work as expected.

In any case, if you are using only one table, you might want to can consider using the Tabular models instead of the SDV class, as these do not require you to create a metadata beforehand and have additional options. You can read more about them here: https://sdv.dev/SDV/user_guides/single_table/gaussian_copula.html

[Arrmlet]

@csala Thanks a lot:)

[csala]

Closing this, as the question is already answered.