Transparent and privacy-friendly synthetic data generation

Generate synthetic tabular data in a transparent, understandable, and privacy-friendly way. Metasyn makes it possible for owners of sensitive data to create test data, do open science, improve code reproducibility, encourage data reuse, and enhance accessibility of their datasets, without worrying about leaking private information.

With metasyn you can fit a model to an existing dataframe, export it to a transparent and auditable .json file, and synthesize a dataframe that looks a lot like the real one. In contrast to most other synthetic data software, we make the explicit choice to strictly limit the statistical information in our model in order to adhere to the highest privacy standards.

Highlights

• 👋 Accessible. Metasyn is designed to be easy to use and understand, and we do our best to be welcoming to newcomers and novice users. Let us know if we can improve!
• ✨ Fully featured. Out of the box, metasyn natively handles a wide range of data types, missing values, categorical data, key columns with unique values, and structured strings such as postcodes or identifiers.
• 🔎 Transparent. With metasyn you share not only synthetic data, but also the model and settings used to create it through a traceable, auditable metadata format. Everyone can read and understand what the model does; it is crystal clear which information becomes public.
• 🔐 Private. By default, metasyn does not incorporate multivariate information, meaning less risk of privacy issues such as identity, attribute, or group disclosure. On top of this, we support privacy plugins such as our own disclosure control plugin to further enhance privacy in critically sensitive situations.
• 🔗 Integrated. We integrate closely with popular, modern tools in the python ecosystem, building on the wonderful polars dataframe library (pandas is supported too), as well as faker to generate localized data for names, emails, and phone numbers, and more.
• 📦 Extensible. Are you missing features? Do you have a different definition of privacy? Our plugin system allows you (or your organisation) to create their own extension to adjust metasyn to what you need. Or you can contribute directly to the project.

Installation

Metasyn can be installed directly from PyPI using the following command in the terminal:

pip install metasyn

The latest (possibly unstable) development version can be installed directly from GitHub like so:

pip install git+https://github.com/sodascience/metasyn

Usage

To generate synthetic data, metasyn first needs to fit a MetaFrame to the data which can then be used to produce new synthetic rows:

In Python code this happens as follows:

import polars as pl
from metasyn import MetaFrame, demo_file

# Get the csv file path for built-in demo dataset
csv_path = demo_file("fruit")

# Create a polars dataframe from the csv file.
# It is important to ensure the data types are correct  
# when creating your dataframe, especially categorical data!
df = pl.read_csv(csv_path, dtypes={
  "fruits": pl.Categorical, 
  "cars": pl.Categorical
})

# Create a MetaFrame from the DataFrame.
mf = MetaFrame.fit_dataframe(df)

# Generate a new DataFrame with 5 rows from the MetaFrame.
df_synth = mf.synthesize(5)

# This DataFrame can be exported to csv, parquet, excel and more.
df_synth.write_csv("output.csv")

To explore more options and try this out online, take a look at our interactive tutorial:

For more information on how to create dataframes with polars, refer to the Polars documentation. Metasyn also works with pandas dataframes!

Where to go next

• As a next step to learn more about generating synthetic data with metasyn we recommend to check out the user guide.
• For even more privacy, have a look at our disclosure control plugin.
• To learn more about how metasyn works, go to detailed overview in our documentation.
• Want to create programs that build on metasyn? Take a look at our versioned Docker containers and our CLI.

Contributing

Metasyn is an open-source project, and we welcome contributions from the community, from bug reports & feature requests to code contributions. Read our contributing guidelines for more information and to get started!

Contact

Metasyn is a project by the ODISSEI Social Data Science (SoDa) team. Do you have questions, suggestions, or remarks on the technical implementation? Create an issue in the issue tracker or feel free to contact Erik-Jan van Kesteren or Raoul Schram.