crc_data_and_metric_bundle_1.2/notebooks/nb3_race_distribution.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ee4989fc-5510-4f9f-88c1-7a1972bf38c6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# %load_ext autoreload\n",
    "# %autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f402d1a",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Authors:\n",
    "- [Karan Bhagat](mailto:kb.in.triangulum@gmail.com) - Developer\n",
    "- [Christine Task](mailto:christine.task@knexusresearch.com) - Project technical lead\n",
    "- [Dhruv Kapur](mailto:dhruvka@umich.edu) - Contributor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "528f8301",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## NIST Collaborative Research Cycle (CRC)\n",
    "### Example Notebook 3: Race Distribution (difference in count of records of each race in deid dataset and target dataset).\n",
    "\n",
    "One concern with data deidentification is the difficulty of maintaining accurate data distributions on data with diverse subpopulations. Privacy algorithms tend to reduce the visibility of outliers-- either intentionally, through Statistical Disclosure Limitation suppression schemes such as k-anonymity, or indirectly, through additive noise that has a relatively bigger impact on small counts (ie, a count of 2 +/- 2 retains less information than a count of 200 +/- 2).  When a diverse subpopulation has a smaller number of individuals, and those individuals have feature combinations that differ from the majority, that group can be dispersed more thinly across the feature space and then become significantly altered or erased during deidentification.\n",
    "\n",
    "In this notebook we show how to check in on this potential problem for our deidentified data samples, using a simple metric that compares the edit distance between the race distribution in the target data and deidentified data (using the csv data files for each). At the end of the notebook we print a data frame containing this metric and other algorithm properties that may be worthy of further exploration.\n",
    "\n",
    "#### List of topics covered in this notebook:\n",
    "0. [Setup notebook.](#nb3-section-0)\n",
    "1. [Load deid datasets index file (index.csv).](#nb3-section-1)  \n",
    "<!--    1.1 [List properties (columns) of each deid. dataset (each row) in the index dataframe (created from index.csv).](#section-1.1)  -->\n",
    "2. [CCompute difference in densities of each race between deid and target datasets.](#nb3-section-2)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa496fac",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<a id=\"nb3-section-0\"></a>\n",
    "## Setup notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b81fac",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Import required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9d7bf66a",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from pathlib import Path  # for creating paths for the filesystem operations\n",
    "import os  # for filesystem operations\n",
    "import pandas as pd  # for tabular data operations\n",
    "import numpy as np  # for tabular data operations\n",
    "import matplotlib.pyplot as plt  # for plotting data as charts\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e84ff29",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Import local libraries present in libs package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cfb0904f",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Import helper constants and functions\n",
    "# libs.util has some small utility functions\n",
    "# such has read_json, list_report etc.\n",
    "import libs.util as u\n",
    "# libs.plot module contains some helper\n",
    "# funtions to create matplotlib charts\n",
    "import libs.plot as plot\n",
    "# libs.strs contains some constant strings\n",
    "# represeting columns/feature names in index.csv file,\n",
    "# report.json file, etc.\n",
    "# Some examples of such constants: \n",
    "# LIBRARY_NAME, ALOGIRTHM_NAME, TARGET_DATASETetc.\n",
    "from libs.strs import *\n",
    "# lib.strs provides get_colors function that can be\n",
    "# use to generate N random colors.\n",
    "import libs.color as color"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46ccca8b",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Change Current Working Directory (CWD) to: crc_data_and_metric_bundle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "eca144bf",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CWD Before:  /Users/karan/Documents/Knexus/NIST/SDNist/crc_acceleration_bundle_1.1/crc_data_and_metric_bundle_1.1/notebooks\n",
      "CWD After:  /Users/karan/Documents/Knexus/NIST/SDNist/crc_acceleration_bundle_1.1/crc_data_and_metric_bundle_1.1\n"
     ]
    }
   ],
   "source": [
    "print('CWD Before: ', Path.cwd())\n",
    "# get current working directory\n",
    "cwd = Path.cwd()\n",
    "# get path to the directory where is notebook is located\n",
    "notebook_dir = u.notebook_path()\n",
    "# Name of the parent directory of this notebook is crc_data_and_metric_bundle_1.0\n",
    "new_cwd = notebook_dir.parent\n",
    "# Set new current working directory path which ends in crc_data_and_metric_bundle_1.0\n",
    "os.chdir(new_cwd)\n",
    "print('CWD After: ', Path.cwd())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939d13da",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5f83359",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76fc51b1",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<a id=\"nb3-section-1\"></a>\n",
    "## 1. Load deid datasets index file (index.csv)\n",
    "index.csv file has tabular information about the deidentified (deid) datasets created using different data-deidentification libraries and algorithms. \n",
    "#### Location of index.csv: *crc_data_and_metric_bundle_1.0/index.csv*\n",
    "#### Location of this notebook: *crc_data_and_metric_bundle_1.0/notebooks/intro.ipynb*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "98ff01a1",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "CWD = Path.cwd()  # get current working directory path\n",
    "# Create path to the index.csv file\n",
    "INDEX_FILE_PATH = Path(CWD, 'index.csv')\n",
    "# read index csv into idx_df variable\n",
    "# idx stands for index\n",
    "idx_df = pd.read_csv(INDEX_FILE_PATH, index_col=0).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63215f2e",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Index data has a total of 298 different deid. datasets and each dataset has 23 different properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "09090b9e",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(458, 22)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See the shape of the index dataframe.\n",
    "idx_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "952651a6",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>library name</th>\n",
       "      <th>algorithm name</th>\n",
       "      <th>algorithm type</th>\n",
       "      <th>target dataset</th>\n",
       "      <th>feature set name</th>\n",
       "      <th>feature space size</th>\n",
       "      <th>features list</th>\n",
       "      <th>privacy category</th>\n",
       "      <th>privacy label detail</th>\n",
       "      <th>epsilon</th>\n",
       "      <th>...</th>\n",
       "      <th>variant label detail</th>\n",
       "      <th>research papers</th>\n",
       "      <th>data path</th>\n",
       "      <th>labels path</th>\n",
       "      <th>report path</th>\n",
       "      <th>team</th>\n",
       "      <th>submission number</th>\n",
       "      <th>submission timestamp</th>\n",
       "      <th>quasi identifiers subset</th>\n",
       "      <th>deid data id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>420</th>\n",
       "      <td>tumult</td>\n",
       "      <td>DPHist</td>\n",
       "      <td>histogram</td>\n",
       "      <td>national2019</td>\n",
       "      <td>demographic-focused-except-AGEP-DEYE</td>\n",
       "      <td>1135134</td>\n",
       "      <td>SEX, MSP, RAC1P, HOUSING_TYPE, OWN_RENT, EDU, ...</td>\n",
       "      <td>dp</td>\n",
       "      <td>Differentially private, histogram-based synthe...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://web.cs.ucdavis.edu/~franklin/ecs289/20...</td>\n",
       "      <td>deid_data/tumult_dphist/national/dphist_e_1_cf...</td>\n",
       "      <td>deid_data/tumult_dphist/national/dphist_e_1_cf...</td>\n",
       "      <td>deid_data/tumult_dphist/national/report_dphist...</td>\n",
       "      <td>CRC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5/20/2023 00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4a5ef378898955b24257566d7bcc365b00701bb3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>311</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>32432400</td>\n",
       "      <td>PUMA, SEX, MSP, HISP, RAC1P, OWN_RENT, INDP_CA...</td>\n",
       "      <td>dp</td>\n",
       "      <td>Conditional tabular GAN using Private Aggregat...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://openreview.net/forum?id=S1zk9iRqF7</td>\n",
       "      <td>deid_data/smartnoise_patectgan/tx/patectgan_e_...</td>\n",
       "      <td>deid_data/smartnoise_patectgan/tx/patectgan_e_...</td>\n",
       "      <td>deid_data/smartnoise_patectgan/tx/report_patec...</td>\n",
       "      <td>CRC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5/20/2023 00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>95f827acc71472a99b9a733f6241c0a5ce0a69d4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>196151155200000000000</td>\n",
       "      <td>PUMA, AGEP, SEX, MSP, HISP, RAC1P, NOC, NPF, H...</td>\n",
       "      <td>dp</td>\n",
       "      <td>Conditional tabular GAN using Private Aggregat...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://openreview.net/forum?id=S1zk9iRqF7</td>\n",
       "      <td>deid_data/smartnoise_patectgan/national/patect...</td>\n",
       "      <td>deid_data/smartnoise_patectgan/national/patect...</td>\n",
       "      <td>deid_data/smartnoise_patectgan/national/report...</td>\n",
       "      <td>CRC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5/20/2023 00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>d2bae376f0c8d58bb3109007ec52d84b2a9b4a55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>410</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>tvae</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>27027000</td>\n",
       "      <td>PUMA, SEX, MSP, HISP, RAC1P, OWN_RENT, INDP_CA...</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>A conditional VAE network which can handle tab...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>added a column target to the original dataset ...</td>\n",
       "      <td>https://doi.org/10.48550/arXiv.1907.00503</td>\n",
       "      <td>deid_data/sythcity_tvae/ma/tvae_industry.csv</td>\n",
       "      <td>deid_data/sythcity_tvae/ma/tvae_industry.json</td>\n",
       "      <td>deid_data/sythcity_tvae/ma/report_tvae_industr...</td>\n",
       "      <td>CRC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5/20/2023 00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>040ed2f17bccfc27c53bff0586b3fc6f5fdd6971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>cart</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>27027000</td>\n",
       "      <td>PUMA, SEX, MSP, HISP, RAC1P, OWN_RENT, INDP_CA...</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>Non-differentially private synthetic data.  Pr...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://doi.org/10.18637/jss.v074.i11</td>\n",
       "      <td>deid_data/rsynthpop_cart/ma/cart_industry_ma20...</td>\n",
       "      <td>deid_data/rsynthpop_cart/ma/cart_industry_ma20...</td>\n",
       "      <td>deid_data/rsynthpop_cart/ma/report_cart_indust...</td>\n",
       "      <td>CRC</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5/20/2023 00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>dde7ecbf0a1e39a35314e638f2fe6f55cfc68fe4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         library name algorithm name algorithm type target dataset   \n",
       "420            tumult         DPHist      histogram   national2019  \\\n",
       "311  smartnoise-synth      patectgan     neural net         tx2019   \n",
       "299  smartnoise-synth      patectgan     neural net   national2019   \n",
       "410         synthcity           tvae     neural net         ma2019   \n",
       "46          rsynthpop           cart     stat model         ma2019   \n",
       "\n",
       "                         feature set name     feature space size   \n",
       "420  demographic-focused-except-AGEP-DEYE                1135134  \\\n",
       "311                      industry-focused               32432400   \n",
       "299                       simple-features  196151155200000000000   \n",
       "410                      industry-focused               27027000   \n",
       "46                       industry-focused               27027000   \n",
       "\n",
       "                                         features list privacy category   \n",
       "420  SEX, MSP, RAC1P, HOUSING_TYPE, OWN_RENT, EDU, ...               dp  \\\n",
       "311  PUMA, SEX, MSP, HISP, RAC1P, OWN_RENT, INDP_CA...               dp   \n",
       "299  PUMA, AGEP, SEX, MSP, HISP, RAC1P, NOC, NPF, H...               dp   \n",
       "410  PUMA, SEX, MSP, HISP, RAC1P, OWN_RENT, INDP_CA...           non_dp   \n",
       "46   PUMA, SEX, MSP, HISP, RAC1P, OWN_RENT, INDP_CA...           non_dp   \n",
       "\n",
       "                                  privacy label detail  epsilon  ...   \n",
       "420  Differentially private, histogram-based synthe...      1.0  ...  \\\n",
       "311  Conditional tabular GAN using Private Aggregat...      5.0  ...   \n",
       "299  Conditional tabular GAN using Private Aggregat...      1.0  ...   \n",
       "410  A conditional VAE network which can handle tab...      NaN  ...   \n",
       "46   Non-differentially private synthetic data.  Pr...      NaN  ...   \n",
       "\n",
       "                                  variant label detail   \n",
       "420                                                NaN  \\\n",
       "311                                                NaN   \n",
       "299                                                NaN   \n",
       "410  added a column target to the original dataset ...   \n",
       "46                                                 NaN   \n",
       "\n",
       "                                       research papers   \n",
       "420  https://web.cs.ucdavis.edu/~franklin/ecs289/20...  \\\n",
       "311         https://openreview.net/forum?id=S1zk9iRqF7   \n",
       "299         https://openreview.net/forum?id=S1zk9iRqF7   \n",
       "410          https://doi.org/10.48550/arXiv.1907.00503   \n",
       "46               https://doi.org/10.18637/jss.v074.i11   \n",
       "\n",
       "                                             data path   \n",
       "420  deid_data/tumult_dphist/national/dphist_e_1_cf...  \\\n",
       "311  deid_data/smartnoise_patectgan/tx/patectgan_e_...   \n",
       "299  deid_data/smartnoise_patectgan/national/patect...   \n",
       "410       deid_data/sythcity_tvae/ma/tvae_industry.csv   \n",
       "46   deid_data/rsynthpop_cart/ma/cart_industry_ma20...   \n",
       "\n",
       "                                           labels path   \n",
       "420  deid_data/tumult_dphist/national/dphist_e_1_cf...  \\\n",
       "311  deid_data/smartnoise_patectgan/tx/patectgan_e_...   \n",
       "299  deid_data/smartnoise_patectgan/national/patect...   \n",
       "410      deid_data/sythcity_tvae/ma/tvae_industry.json   \n",
       "46   deid_data/rsynthpop_cart/ma/cart_industry_ma20...   \n",
       "\n",
       "                                           report path team submission number   \n",
       "420  deid_data/tumult_dphist/national/report_dphist...  CRC               NaN  \\\n",
       "311  deid_data/smartnoise_patectgan/tx/report_patec...  CRC               NaN   \n",
       "299  deid_data/smartnoise_patectgan/national/report...  CRC               NaN   \n",
       "410  deid_data/sythcity_tvae/ma/report_tvae_industr...  CRC               NaN   \n",
       "46   deid_data/rsynthpop_cart/ma/report_cart_indust...  CRC               NaN   \n",
       "\n",
       "    submission timestamp  quasi identifiers subset   \n",
       "420   5/20/2023 00:00:00                       NaN  \\\n",
       "311   5/20/2023 00:00:00                       NaN   \n",
       "299   5/20/2023 00:00:00                       NaN   \n",
       "410   5/20/2023 00:00:00                       NaN   \n",
       "46    5/20/2023 00:00:00                       NaN   \n",
       "\n",
       "                                 deid data id  \n",
       "420  4a5ef378898955b24257566d7bcc365b00701bb3  \n",
       "311  95f827acc71472a99b9a733f6241c0a5ce0a69d4  \n",
       "299  d2bae376f0c8d58bb3109007ec52d84b2a9b4a55  \n",
       "410  040ed2f17bccfc27c53bff0586b3fc6f5fdd6971  \n",
       "46   dde7ecbf0a1e39a35314e638f2fe6f55cfc68fe4  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See five random rows of the index dataframe\n",
    "idx_df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "848594e1",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 1.1 Veiw available features in the index dataframe.\n",
    "List features(columns) of each deid. dataset (each row) in the index dataframe (created from index.csv). \n",
    "\n",
    "Some features inform about the method of data synthesis (features such as 'library', 'algorithm name', 'epsilon', 'variant label' etc.) and others about the location of resources (such as 'data path' path to deid dataset csv file, \n",
    "'report path' path to sdnist evaluation report of the deid dataset file etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e22f81a1",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "library name\n",
      "algorithm name\n",
      "algorithm type\n",
      "target dataset\n",
      "feature set name\n",
      "feature space size\n",
      "features list\n",
      "privacy category\n",
      "privacy label detail\n",
      "epsilon\n",
      "delta\n",
      "variant label\n",
      "variant label detail\n",
      "research papers\n",
      "data path\n",
      "labels path\n",
      "report path\n",
      "team\n",
      "submission number\n",
      "submission timestamp\n",
      "quasi identifiers subset\n",
      "deid data id\n"
     ]
    }
   ],
   "source": [
    "# Iterate over the columns of index dataframe\n",
    "# and print each column name\n",
    "for col in idx_df.columns:\n",
    "    print(col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2bdbe044",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Index Column: \n",
      "-------------\n",
      "library name\n",
      "\n",
      "\n",
      "Description:\n",
      "-----------\n",
      "Software library used for this submission, or team name if a private codebase was used\n"
     ]
    }
   ],
   "source": [
    "# Use index_feature_description function from the libs.util module.\n",
    "# index_feature_description function takes just one argument which \n",
    "# should be one of the column name available in the index file. \n",
    "# Column names are printed in the last code cell for your reference.\n",
    "u.index_feature_description(LIBRARY_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eecaf15",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<a id=\"nb3-section-2\"></a>\n",
    "## 2. Compute difference in densities percentage of each race between deid and target datasets.\n",
    "Race density is computed by finding the fraction of people of a race in a dataset. We find race density percentage for both a deid dataset and its target dataset, and then take the absolute difference of these density percentages.\n",
    "We then use the absolute difference of density percentages to create a new score called race_edit_distance_percent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fdeff834",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Store path to the target dataset base directory which is \n",
    "# diverse_communities_data_excerpts.\n",
    "TARGET_DATA_DIR = Path(CWD, 'diverse_communities_data_excerpts')\n",
    "# Store path to all the three datasets: ma2019, tx2019, and national2019\n",
    "MA_PATH = Path(TARGET_DATA_DIR, 'massachusetts', 'ma2019.csv')\n",
    "TX_PATH = Path(TARGET_DATA_DIR, 'texas', 'tx2019.csv')\n",
    "NAT_PATH = Path(TARGET_DATA_DIR, 'national', 'national2019.csv')\n",
    "\n",
    "\n",
    "# Declare constants that we use as\n",
    "# column names\n",
    "DEID_COUNT = 'deid_count'\n",
    "DEID_DENSITY_PERCENT = 'deid_density_percent'\n",
    "TARGET_COUNT = 'target_count'\n",
    "TARGET_DENSITY_PERCENT = 'target_density_percent'\n",
    "DIFFERENCE = 'difference'\n",
    "RACE_EDIT_DISTANCE_PERCENT = 'race_edit_distance_percent'\n",
    "\n",
    "\n",
    "# Create a new zero filled column\n",
    "idx_df[RACE_EDIT_DISTANCE_PERCENT] = np.zeros(idx_df.shape[0]).astype(int)\n",
    "\n",
    "# Iterate over each row (deid dataset) in index dataframe.\n",
    "for i, row in idx_df.iterrows():\n",
    "    # Get path of deid. data csv file\n",
    "    data_path = row[DATA_PATH]\n",
    "    # Read the csv file into a pandas dataframe using data path.\n",
    "    data_deid_df = pd.read_csv(data_path)\n",
    "    # Cleaning step (removes extra leading column from csv).\n",
    "    data_deid_df = data_deid_df.loc[:, ~data_deid_df.columns.str.startswith('Unnamed')]\n",
    "    data_deid_df = data_deid_df.reset_index(drop=True)\n",
    "    \n",
    "    # Get path to the target dataset from which the\n",
    "    # current deid dataset is created\n",
    "    target_dataset_name = row[TARGET_DATASET]\n",
    "    target_path = ''\n",
    "    \n",
    "    if target_dataset_name == MA2019:\n",
    "        target_path = MA_PATH\n",
    "    elif target_dataset_name == TX2019:\n",
    "        target_path = TX_PATH\n",
    "    elif target_dataset_name == NATIONAL2019:\n",
    "        target_path = NAT_PATH\n",
    "    \n",
    "    data_target_df = pd.read_csv(target_path)\n",
    "    \n",
    "    if RAC1P not in data_deid_df.columns.tolist():\n",
    "        continue\n",
    "\n",
    "    # Get count of each race in deid dataset and save it as new dataframe\n",
    "    deid_race_df = data_deid_df[RAC1P].value_counts().to_frame(DEID_COUNT)\n",
    "    # Compute density percent of each race in the deid dataset\n",
    "    deid_race_df[DEID_DENSITY_PERCENT] = deid_race_df[DEID_COUNT] / data_deid_df.shape[0] * 100\n",
    "    \n",
    "    # Get count of each race in target dataset and save it as new dataframe\n",
    "    target_race_df = data_target_df[RAC1P].value_counts().to_frame(TARGET_COUNT)\n",
    "    # Compute density percent of each race in the target dataset\n",
    "    target_race_df[TARGET_DENSITY_PERCENT] = target_race_df[TARGET_COUNT] / data_target_df.shape[0] * 100\n",
    "    \n",
    "    # Move unique values of RAC1P to a new column named RAC1P and set type as\n",
    "    # 'int'. We change type to 'int' because some deid dataset save RAC1P\n",
    "    # value as floats.\n",
    "    deid_race_df[RAC1P] = deid_race_df.index.astype(int)\n",
    "    target_race_df[RAC1P] = target_race_df.index.astype(int)\n",
    "\n",
    "    # Reset index after get race values\n",
    "    deid_race_df = deid_race_df.reset_index(drop=True)\n",
    "    target_race_df = target_race_df.reset_index(drop=True)\n",
    "    \n",
    "    # Sort deid and target dataset by race\n",
    "    deid_race_df = deid_race_df.sort_values(by=RAC1P)\n",
    "\n",
    "    target_race_df = target_race_df.sort_values(by=RAC1P)\n",
    "\n",
    "    # Merge both dataset on RAC1P and do outer merge to take RAC1P values from\n",
    "    # both deid and target dataset\n",
    "    merged_df = target_race_df.merge(deid_race_df, on=RAC1P, how='outer')\n",
    "    \n",
    "    # Since few races can be missing in the deid dataset, those\n",
    "    # will have nan values for deid_counts and deid_density_percent\n",
    "    # columns after the merge operation\n",
    "    merged_df = merged_df.fillna(0)\n",
    "    # Get absolute difference of density percentages between target and deid dataset\n",
    "    merged_df[DIFFERENCE] = abs(merged_df[DEID_DENSITY_PERCENT] - merged_df[TARGET_DENSITY_PERCENT])\n",
    "        \n",
    "    # Calculate edit distance between density percentages of races in \n",
    "    # target data and count of races in deid data\n",
    "    edit_distance = sum(merged_df[DIFFERENCE].values)\n",
    "\n",
    "    # Save value race values difference in\n",
    "    # index dataframe.\n",
    "    idx_df.loc[i, RACE_EDIT_DISTANCE_PERCENT] = round(edit_distance, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25de0247-e2df-4a67-bcd5-389b7ccf2e9c",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 2.1 Worst performing 75 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9c4d4108",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>library name</th>\n",
       "      <th>algorithm name</th>\n",
       "      <th>algorithm type</th>\n",
       "      <th>target dataset</th>\n",
       "      <th>privacy category</th>\n",
       "      <th>epsilon</th>\n",
       "      <th>feature set name</th>\n",
       "      <th>race_edit_distance_percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>pram</td>\n",
       "      <td>sdc</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>sdv</td>\n",
       "      <td>ctgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>16.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>ipf</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>2.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>2.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>205</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>2.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>aindo-synth</td>\n",
       "      <td>aindo-synth</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>1.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Anonos Data Embassy SDK</td>\n",
       "      <td>Anonos Data Embassy SDK</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>409</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>tvae</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>8.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused-except-DEYE</td>\n",
       "      <td>0.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>pategan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>13.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>ipf</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>3.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>MostlyAI SD</td>\n",
       "      <td>MostlyAI SD</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>333</th>\n",
       "      <td>subsample</td>\n",
       "      <td>subsample_40pcnt</td>\n",
       "      <td>sdc</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>1.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>cart</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-21</td>\n",
       "      <td>0.57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>139.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>289</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>17.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>399</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>privbayes</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>119.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>3.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused-with-DEYE</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Sarus SDG</td>\n",
       "      <td>Sarus SDG</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>2.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>ipf</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>5.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>457</th>\n",
       "      <td>ydata-synthetic</td>\n",
       "      <td>ctgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>simple-features-except-PINCP-POVPIP</td>\n",
       "      <td>10.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>438</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>7.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>335</th>\n",
       "      <td>subsample</td>\n",
       "      <td>subsample_5pcnt</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>3.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>407</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>tvae</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>10.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused-except-DEYE</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>subsample</td>\n",
       "      <td>subsample_1pcnt</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>8.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused-with-DEYE</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>287</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>5.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>subsample</td>\n",
       "      <td>subsample_1pcnt</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>7.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mwem</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>custom-features-12</td>\n",
       "      <td>1.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>364</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>pategan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>all-features</td>\n",
       "      <td>11.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>453</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused-except-PINCP_DECILE-EDU</td>\n",
       "      <td>186.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>pram</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>346</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>adsgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>62.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>sdv</td>\n",
       "      <td>tvae</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>5.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>4.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>431</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused-except-PINCP_DECILE-EDU</td>\n",
       "      <td>195.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>307</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>10.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>MostlyAI SD</td>\n",
       "      <td>MostlyAI SD</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>1.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>sdv</td>\n",
       "      <td>ctgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>8.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>sdv</td>\n",
       "      <td>gaussian-copula</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>20.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>398</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>privbayes</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>sdv</td>\n",
       "      <td>tvae</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>14.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>6.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>MostlyAI SD</td>\n",
       "      <td>MostlyAI SD</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>6.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>1.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>pategan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>all-features</td>\n",
       "      <td>30.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>kanonymity</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>32.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>1.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>aindo-synth</td>\n",
       "      <td>aindo-synth</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>1.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>sdv</td>\n",
       "      <td>ctgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>7.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>262</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>pacsynth</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>18.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>312</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>128.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>kanonymity</td>\n",
       "      <td>sdc</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>13.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>166</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused-except-AGEP-DEYE</td>\n",
       "      <td>0.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>family-focused-with-DEYE</td>\n",
       "      <td>1.57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>pacsynth</td>\n",
       "      <td>query matching</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>13.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>295</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>114.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>all-features</td>\n",
       "      <td>1.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>429</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-5</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>sdv</td>\n",
       "      <td>tvae</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>30.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>patectgan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>5.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>query matching</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>all-features</td>\n",
       "      <td>2.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>444</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused-except-PINCP_DECILE-EDU</td>\n",
       "      <td>183.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>1.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>pategan</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>18.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>337</th>\n",
       "      <td>subsample</td>\n",
       "      <td>subsample_5pcnt</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>2.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>405</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>privbayes</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>40.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>sdv</td>\n",
       "      <td>tvae</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>6.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                library name             algorithm name  algorithm type   \n",
       "98                  sdcmicro                       pram             sdc  \\\n",
       "114                      sdv                      ctgan      neural net   \n",
       "72                 rsynthpop                        ipf  query matching   \n",
       "205         smartnoise-synth                        mst      stat model   \n",
       "155         smartnoise-synth                        aim      stat model   \n",
       "40               aindo-synth                aindo-synth      neural net   \n",
       "0    Anonos Data Embassy SDK    Anonos Data Embassy SDK      neural net   \n",
       "409                synthcity                       tvae      neural net   \n",
       "229         smartnoise-synth                        mst      stat model   \n",
       "384                synthcity                    pategan      neural net   \n",
       "70                 rsynthpop                        ipf  query matching   \n",
       "28               MostlyAI SD                MostlyAI SD      neural net   \n",
       "333                subsample           subsample_40pcnt             sdc   \n",
       "47                 rsynthpop                       cart      stat model   \n",
       "285         smartnoise-synth                  patectgan      neural net   \n",
       "289         smartnoise-synth                  patectgan      neural net   \n",
       "399                synthcity                  privbayes      stat model   \n",
       "150         smartnoise-synth                        aim      stat model   \n",
       "284         smartnoise-synth                  patectgan      neural net   \n",
       "233         smartnoise-synth                        mst      stat model   \n",
       "29                 Sarus SDG                  Sarus SDG      neural net   \n",
       "71                 rsynthpop                        ipf  query matching   \n",
       "457          ydata-synthetic                      ctgan      neural net   \n",
       "438                ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "335                subsample            subsample_5pcnt             sdc   \n",
       "407                synthcity                       tvae      neural net   \n",
       "195         smartnoise-synth                        mst      stat model   \n",
       "315                subsample            subsample_1pcnt             sdc   \n",
       "201         smartnoise-synth                        mst      stat model   \n",
       "287         smartnoise-synth                  patectgan      neural net   \n",
       "318                subsample            subsample_1pcnt             sdc   \n",
       "241         smartnoise-synth                       mwem  query matching   \n",
       "364                synthcity                    pategan      neural net   \n",
       "453                ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "106                 sdcmicro                       pram             sdc   \n",
       "346                synthcity                     adsgan      neural net   \n",
       "132                      sdv                       tvae      neural net   \n",
       "174         smartnoise-synth                        aim      stat model   \n",
       "431                ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "12                Genetic SD                 Genetic SD  query matching   \n",
       "307         smartnoise-synth                  patectgan      neural net   \n",
       "25               MostlyAI SD                MostlyAI SD      neural net   \n",
       "116                      sdv                      ctgan      neural net   \n",
       "127                      sdv            gaussian-copula      stat model   \n",
       "398                synthcity                  privbayes      stat model   \n",
       "142                      sdv                       tvae      neural net   \n",
       "15                Genetic SD                 Genetic SD  query matching   \n",
       "24               MostlyAI SD                MostlyAI SD      neural net   \n",
       "149         smartnoise-synth                        aim      stat model   \n",
       "382                synthcity                    pategan      neural net   \n",
       "91                  sdcmicro                 kanonymity             sdc   \n",
       "194         smartnoise-synth                        mst      stat model   \n",
       "41               aindo-synth                aindo-synth      neural net   \n",
       "119                      sdv                      ctgan      neural net   \n",
       "262         smartnoise-synth                   pacsynth  query matching   \n",
       "312         smartnoise-synth                  patectgan      neural net   \n",
       "78                  sdcmicro                 kanonymity             sdc   \n",
       "166         smartnoise-synth                        aim      stat model   \n",
       "235         smartnoise-synth                        mst      stat model   \n",
       "254         smartnoise-synth                   pacsynth  query matching   \n",
       "295         smartnoise-synth                  patectgan      neural net   \n",
       "145         smartnoise-synth                        aim      stat model   \n",
       "180         smartnoise-synth                        aim      stat model   \n",
       "429                ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "134                      sdv                       tvae      neural net   \n",
       "283         smartnoise-synth                  patectgan      neural net   \n",
       "4                 Genetic SD                 Genetic SD  query matching   \n",
       "444                ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "163         smartnoise-synth                        aim      stat model   \n",
       "377                synthcity                    pategan      neural net   \n",
       "337                subsample            subsample_5pcnt             sdc   \n",
       "188         smartnoise-synth                        aim      stat model   \n",
       "405                synthcity                  privbayes      stat model   \n",
       "131                      sdv                       tvae      neural net   \n",
       "220         smartnoise-synth                        mst      stat model   \n",
       "\n",
       "    target dataset privacy category  epsilon   \n",
       "98          ma2019              sdc      NaN  \\\n",
       "114   national2019           non_dp      NaN   \n",
       "72    national2019               dp      2.0   \n",
       "205         ma2019               dp      1.0   \n",
       "155         ma2019               dp     10.0   \n",
       "40    national2019           non_dp      NaN   \n",
       "0           ma2019           non_dp      NaN   \n",
       "409         ma2019           non_dp      NaN   \n",
       "229         tx2019               dp     10.0   \n",
       "384         tx2019               dp      1.0   \n",
       "70    national2019               dp     10.0   \n",
       "28          tx2019           non_dp      NaN   \n",
       "333         tx2019              sdc      NaN   \n",
       "47    national2019           non_dp      NaN   \n",
       "285         ma2019               dp      1.0   \n",
       "289   national2019               dp     10.0   \n",
       "399   national2019               dp     10.0   \n",
       "150         ma2019               dp     10.0   \n",
       "284         ma2019               dp      5.0   \n",
       "233         tx2019               dp     10.0   \n",
       "29    national2019               dp     10.0   \n",
       "71    national2019               dp      1.0   \n",
       "457         ma2019           non_dp      NaN   \n",
       "438   national2019           non_dp      NaN   \n",
       "335   national2019              sdc      NaN   \n",
       "407         ma2019           non_dp      NaN   \n",
       "195         ma2019               dp      5.0   \n",
       "315   national2019              sdc      NaN   \n",
       "201         ma2019               dp     10.0   \n",
       "287         ma2019               dp     10.0   \n",
       "318   national2019              sdc      NaN   \n",
       "241   national2019               dp     10.0   \n",
       "364         ma2019               dp     10.0   \n",
       "453         tx2019           non_dp      NaN   \n",
       "106   national2019              sdc      NaN   \n",
       "346   national2019           non_dp      NaN   \n",
       "132         ma2019           non_dp      NaN   \n",
       "174   national2019               dp      1.0   \n",
       "431         ma2019           non_dp      NaN   \n",
       "12    national2019               dp     10.0   \n",
       "307         tx2019               dp      5.0   \n",
       "25    national2019           non_dp      NaN   \n",
       "116   national2019           non_dp      NaN   \n",
       "127         ma2019           non_dp      NaN   \n",
       "398   national2019               dp     10.0   \n",
       "142         tx2019           non_dp      NaN   \n",
       "15    national2019               dp      1.0   \n",
       "24          ma2019           non_dp      NaN   \n",
       "149         ma2019               dp      5.0   \n",
       "382         tx2019               dp      1.0   \n",
       "91    national2019              sdc      NaN   \n",
       "194         ma2019               dp      1.0   \n",
       "41          tx2019           non_dp      NaN   \n",
       "119   national2019           non_dp      NaN   \n",
       "262   national2019               dp      5.0   \n",
       "312         tx2019               dp      1.0   \n",
       "78          ma2019              sdc      NaN   \n",
       "166   national2019               dp      5.0   \n",
       "235         tx2019               dp      1.0   \n",
       "254         ma2019               dp     10.0   \n",
       "295   national2019               dp      1.0   \n",
       "145         ma2019               dp      5.0   \n",
       "180         tx2019               dp      5.0   \n",
       "429         ma2019           non_dp      NaN   \n",
       "134   national2019           non_dp      NaN   \n",
       "283         ma2019               dp     10.0   \n",
       "4           ma2019               dp     10.0   \n",
       "444   national2019           non_dp      NaN   \n",
       "163   national2019               dp      1.0   \n",
       "377   national2019               dp     10.0   \n",
       "337   national2019              sdc      NaN   \n",
       "188         tx2019               dp     10.0   \n",
       "405         tx2019               dp     10.0   \n",
       "131         ma2019           non_dp      NaN   \n",
       "220   national2019               dp     10.0   \n",
       "\n",
       "                                feature set name  race_edit_distance_percent  \n",
       "98                           demographic-focused                        0.76  \n",
       "114                                 all-features                       16.84  \n",
       "72                           demographic-focused                        2.67  \n",
       "205                             industry-focused                        2.44  \n",
       "155                              simple-features                        0.81  \n",
       "40                                  all-features                        1.66  \n",
       "0                                   all-features                        0.79  \n",
       "409                               family-focused                        8.07  \n",
       "229              demographic-focused-except-DEYE                        0.30  \n",
       "384                          demographic-focused                       13.95  \n",
       "70                           demographic-focused                        3.40  \n",
       "28                                  all-features                        0.78  \n",
       "333                              simple-features                        1.05  \n",
       "47                            custom-features-21                        0.57  \n",
       "285                             industry-focused                      139.01  \n",
       "289                          demographic-focused                       17.50  \n",
       "399                             industry-focused                      119.55  \n",
       "150                               family-focused                        0.58  \n",
       "284                             industry-focused                        3.35  \n",
       "233                     family-focused-with-DEYE                        0.52  \n",
       "29                           demographic-focused                        2.79  \n",
       "71                           demographic-focused                        5.78  \n",
       "457          simple-features-except-PINCP-POVPIP                       10.24  \n",
       "438                                 all-features                        7.00  \n",
       "335                          demographic-focused                        3.14  \n",
       "407                                 all-features                       10.03  \n",
       "195              demographic-focused-except-DEYE                        0.52  \n",
       "315                                 all-features                        8.70  \n",
       "201                     family-focused-with-DEYE                        0.47  \n",
       "287                              simple-features                        5.34  \n",
       "318                             industry-focused                        7.64  \n",
       "241                           custom-features-12                        1.09  \n",
       "364                                 all-features                       11.03  \n",
       "453  demographic-focused-except-PINCP_DECILE-EDU                      186.48  \n",
       "106                             industry-focused                        0.52  \n",
       "346                             industry-focused                       62.61  \n",
       "132                               family-focused                        5.89  \n",
       "174                              simple-features                        4.22  \n",
       "431  demographic-focused-except-PINCP_DECILE-EDU                      195.28  \n",
       "12                               simple-features                        0.61  \n",
       "307                               family-focused                       10.82  \n",
       "25                                  all-features                        1.87  \n",
       "116                                 all-features                        8.75  \n",
       "127                          demographic-focused                       20.28  \n",
       "398                               family-focused                        0.97  \n",
       "142                             industry-focused                       14.81  \n",
       "15                               simple-features                        6.25  \n",
       "24                                  all-features                        6.37  \n",
       "149                               family-focused                        1.23  \n",
       "382                                 all-features                       30.21  \n",
       "91                              industry-focused                       32.01  \n",
       "194                          demographic-focused                        1.91  \n",
       "41                                  all-features                        1.10  \n",
       "119                                 all-features                        7.24  \n",
       "262                             industry-focused                       18.22  \n",
       "312                             industry-focused                      128.87  \n",
       "78                           demographic-focused                       13.43  \n",
       "166         demographic-focused-except-AGEP-DEYE                        0.93  \n",
       "235                     family-focused-with-DEYE                        1.57  \n",
       "254                             industry-focused                       13.65  \n",
       "295                               family-focused                      114.86  \n",
       "145                                 all-features                        1.26  \n",
       "180                          demographic-focused                        0.63  \n",
       "429                            custom-features-5                        0.00  \n",
       "134                                 all-features                       30.55  \n",
       "283                             industry-focused                        5.76  \n",
       "4                                   all-features                        2.44  \n",
       "444  demographic-focused-except-PINCP_DECILE-EDU                      183.77  \n",
       "163                          demographic-focused                        1.30  \n",
       "377                               family-focused                       18.19  \n",
       "337                             industry-focused                        2.61  \n",
       "188                              simple-features                        0.47  \n",
       "405                             industry-focused                       40.58  \n",
       "131                          demographic-focused                        6.52  \n",
       "220                             industry-focused                        0.08  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See the random 100 records from index dataframe and\n",
    "# the new column race_edit_distance which we just added to the\n",
    "# index dataframe\n",
    "\n",
    "# Set pandas display max rows option to display more\n",
    "# dataframe rows in the cell output\n",
    "pd.set_option('display.max_rows', 151)\n",
    "# Name of the columns to display in the selected data samples\n",
    "columns_to_show = [LIBRARY_NAME, ALGORITHM_NAME, ALGORITHM_TYPE, TARGET_DATASET,\n",
    "                   PRIVACY_CATEGORY, EPSILON, FEATURE_SET_NAME, RACE_EDIT_DISTANCE_PERCENT]\n",
    "\n",
    "# Sort the samples by race_edit_distance\n",
    "idx_df = idx_df.sort_values(by=RACE_EDIT_DISTANCE_PERCENT, ascending=False)\n",
    "# Show 75  worst performing samples in the index dataframe\n",
    "idx_sample_df = idx_df[columns_to_show].sample(75)\n",
    "# Show samples dataframe\n",
    "idx_sample_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bcbcfb96",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "### 2.1 Best performing 75 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "536d00e2-52c9-42e2-a3ab-433d68a8b9d5",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>library name</th>\n",
       "      <th>algorithm name</th>\n",
       "      <th>algorithm type</th>\n",
       "      <th>target dataset</th>\n",
       "      <th>privacy category</th>\n",
       "      <th>epsilon</th>\n",
       "      <th>feature set name</th>\n",
       "      <th>race_edit_distance_percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>449</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-3</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>428</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-3</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>429</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-5</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>440</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-5</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>catall</td>\n",
       "      <td>histogram</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>100.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>450</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-5</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>439</th>\n",
       "      <td>ydata-sdk</td>\n",
       "      <td>YData Fabric Synthesizers</td>\n",
       "      <td>neural net</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-3</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>UTDallas-AIFairness</td>\n",
       "      <td>smote</td>\n",
       "      <td>geometric</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>UTDallas-AIFairness</td>\n",
       "      <td>smote</td>\n",
       "      <td>geometric</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>UTDallas-AIFairness</td>\n",
       "      <td>smote</td>\n",
       "      <td>geometric</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>219</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>213</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused-except-DEYE</td>\n",
       "      <td>0.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>217</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>family-focused-with-DEYE</td>\n",
       "      <td>0.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>216</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>214</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>419</th>\n",
       "      <td>tumult</td>\n",
       "      <td>DPHist</td>\n",
       "      <td>histogram</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused-except-AGEP-DEYE</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>206</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused-except-DEYE</td>\n",
       "      <td>0.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused-except-AGEP-DEYE</td>\n",
       "      <td>0.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>237</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused-except-DEYE</td>\n",
       "      <td>0.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>322</th>\n",
       "      <td>subsample</td>\n",
       "      <td>subsample_40pcnt</td>\n",
       "      <td>sdc</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>cart</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>pram</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>pram</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>family-focused-with-DEYE</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>pram</td>\n",
       "      <td>sdc</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused-with-DEYE</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>218</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused-except-DEYE</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>pram</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused-with-DEYE</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>pram</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>ipf_NonDP</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>cart</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-21</td>\n",
       "      <td>0.57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused-except-DEYE</td>\n",
       "      <td>0.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>404</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>privbayes</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>242</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mwem</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>demographic-focused-except-DEYE</td>\n",
       "      <td>0.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>324</th>\n",
       "      <td>subsample</td>\n",
       "      <td>subsample_40pcnt</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>all-features</td>\n",
       "      <td>0.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>325</th>\n",
       "      <td>subsample</td>\n",
       "      <td>subsample_40pcnt</td>\n",
       "      <td>sdc</td>\n",
       "      <td>national2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>225</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>privbayes</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>392</th>\n",
       "      <td>synthcity</td>\n",
       "      <td>privbayes</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>family-focused</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>cart</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>non_dp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>custom-features-21</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>tx2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>rsynthpop</td>\n",
       "      <td>catall</td>\n",
       "      <td>histogram</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>100.0</td>\n",
       "      <td>demographic-focused</td>\n",
       "      <td>0.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>208</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>mst</td>\n",
       "      <td>stat model</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>smartnoise-synth</td>\n",
       "      <td>aim</td>\n",
       "      <td>stat model</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>5.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>Genetic SD</td>\n",
       "      <td>query matching</td>\n",
       "      <td>national2019</td>\n",
       "      <td>dp</td>\n",
       "      <td>10.0</td>\n",
       "      <td>simple-features</td>\n",
       "      <td>0.68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>sdcmicro</td>\n",
       "      <td>pram</td>\n",
       "      <td>sdc</td>\n",
       "      <td>ma2019</td>\n",
       "      <td>sdc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>industry-focused</td>\n",
       "      <td>0.68</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            library name             algorithm name  algorithm type   \n",
       "449            ydata-sdk  YData Fabric Synthesizers      neural net  \\\n",
       "428            ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "429            ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "440            ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "58             rsynthpop                     catall       histogram   \n",
       "450            ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "439            ydata-sdk  YData Fabric Synthesizers      neural net   \n",
       "35   UTDallas-AIFairness                      smote       geometric   \n",
       "38   UTDallas-AIFairness                      smote       geometric   \n",
       "210     smartnoise-synth                        mst      stat model   \n",
       "32   UTDallas-AIFairness                      smote       geometric   \n",
       "219     smartnoise-synth                        mst      stat model   \n",
       "220     smartnoise-synth                        mst      stat model   \n",
       "213     smartnoise-synth                        mst      stat model   \n",
       "217     smartnoise-synth                        mst      stat model   \n",
       "216     smartnoise-synth                        mst      stat model   \n",
       "212     smartnoise-synth                        mst      stat model   \n",
       "214     smartnoise-synth                        mst      stat model   \n",
       "419               tumult                     DPHist       histogram   \n",
       "236     smartnoise-synth                        mst      stat model   \n",
       "206     smartnoise-synth                        mst      stat model   \n",
       "221     smartnoise-synth                        mst      stat model   \n",
       "199     smartnoise-synth                        mst      stat model   \n",
       "224     smartnoise-synth                        mst      stat model   \n",
       "229     smartnoise-synth                        mst      stat model   \n",
       "204     smartnoise-synth                        mst      stat model   \n",
       "164     smartnoise-synth                        aim      stat model   \n",
       "161     smartnoise-synth                        aim      stat model   \n",
       "237     smartnoise-synth                        mst      stat model   \n",
       "197     smartnoise-synth                        mst      stat model   \n",
       "322            subsample           subsample_40pcnt             sdc   \n",
       "192     smartnoise-synth                        mst      stat model   \n",
       "172     smartnoise-synth                        aim      stat model   \n",
       "209     smartnoise-synth                        mst      stat model   \n",
       "43             rsynthpop                       cart      stat model   \n",
       "102             sdcmicro                       pram             sdc   \n",
       "105             sdcmicro                       pram             sdc   \n",
       "234     smartnoise-synth                        mst      stat model   \n",
       "188     smartnoise-synth                        aim      stat model   \n",
       "99              sdcmicro                       pram             sdc   \n",
       "146     smartnoise-synth                        aim      stat model   \n",
       "201     smartnoise-synth                        mst      stat model   \n",
       "231     smartnoise-synth                        mst      stat model   \n",
       "218     smartnoise-synth                        mst      stat model   \n",
       "183     smartnoise-synth                        aim      stat model   \n",
       "230     smartnoise-synth                        mst      stat model   \n",
       "195     smartnoise-synth                        mst      stat model   \n",
       "106             sdcmicro                       pram             sdc   \n",
       "233     smartnoise-synth                        mst      stat model   \n",
       "101             sdcmicro                       pram             sdc   \n",
       "76             rsynthpop                  ipf_NonDP  query matching   \n",
       "171     smartnoise-synth                        aim      stat model   \n",
       "193     smartnoise-synth                        mst      stat model   \n",
       "198     smartnoise-synth                        mst      stat model   \n",
       "160     smartnoise-synth                        aim      stat model   \n",
       "47             rsynthpop                       cart      stat model   \n",
       "150     smartnoise-synth                        aim      stat model   \n",
       "227     smartnoise-synth                        mst      stat model   \n",
       "404            synthcity                  privbayes      stat model   \n",
       "173     smartnoise-synth                        aim      stat model   \n",
       "242     smartnoise-synth                       mwem  query matching   \n",
       "324            subsample           subsample_40pcnt             sdc   \n",
       "12            Genetic SD                 Genetic SD  query matching   \n",
       "11            Genetic SD                 Genetic SD  query matching   \n",
       "325            subsample           subsample_40pcnt             sdc   \n",
       "225     smartnoise-synth                        mst      stat model   \n",
       "394            synthcity                  privbayes      stat model   \n",
       "392            synthcity                  privbayes      stat model   \n",
       "42             rsynthpop                       cart      stat model   \n",
       "180     smartnoise-synth                        aim      stat model   \n",
       "59             rsynthpop                     catall       histogram   \n",
       "208     smartnoise-synth                        mst      stat model   \n",
       "175     smartnoise-synth                        aim      stat model   \n",
       "14            Genetic SD                 Genetic SD  query matching   \n",
       "100             sdcmicro                       pram             sdc   \n",
       "\n",
       "    target dataset privacy category  epsilon   \n",
       "449         tx2019           non_dp      NaN  \\\n",
       "428         ma2019           non_dp      NaN   \n",
       "429         ma2019           non_dp      NaN   \n",
       "440   national2019           non_dp      NaN   \n",
       "58    national2019               dp    100.0   \n",
       "450         tx2019           non_dp      NaN   \n",
       "439   national2019           non_dp      NaN   \n",
       "35    national2019           non_dp      NaN   \n",
       "38          tx2019           non_dp      NaN   \n",
       "210   national2019               dp     10.0   \n",
       "32          ma2019           non_dp      NaN   \n",
       "219   national2019               dp      5.0   \n",
       "220   national2019               dp     10.0   \n",
       "213   national2019               dp      5.0   \n",
       "217   national2019               dp      5.0   \n",
       "216   national2019               dp     10.0   \n",
       "212   national2019               dp      5.0   \n",
       "214   national2019               dp      5.0   \n",
       "419   national2019               dp     10.0   \n",
       "236         tx2019               dp      5.0   \n",
       "206         ma2019               dp     10.0   \n",
       "221   national2019               dp     10.0   \n",
       "199         ma2019               dp     10.0   \n",
       "224         tx2019               dp     10.0   \n",
       "229         tx2019               dp     10.0   \n",
       "204         ma2019               dp      5.0   \n",
       "164   national2019               dp     10.0   \n",
       "161   national2019               dp     10.0   \n",
       "237         tx2019               dp     10.0   \n",
       "197         ma2019               dp     10.0   \n",
       "322         ma2019              sdc      NaN   \n",
       "192         ma2019               dp     10.0   \n",
       "172   national2019               dp     10.0   \n",
       "209   national2019               dp      5.0   \n",
       "43          ma2019           non_dp      NaN   \n",
       "102   national2019              sdc      NaN   \n",
       "105   national2019              sdc      NaN   \n",
       "234         tx2019               dp      5.0   \n",
       "188         tx2019               dp     10.0   \n",
       "99          ma2019              sdc      NaN   \n",
       "146         ma2019               dp     10.0   \n",
       "201         ma2019               dp     10.0   \n",
       "231         tx2019               dp     10.0   \n",
       "218   national2019               dp      1.0   \n",
       "183         tx2019               dp     10.0   \n",
       "230         tx2019               dp      5.0   \n",
       "195         ma2019               dp      5.0   \n",
       "106   national2019              sdc      NaN   \n",
       "233         tx2019               dp     10.0   \n",
       "101   national2019              sdc      NaN   \n",
       "76    national2019           non_dp      NaN   \n",
       "171   national2019               dp      5.0   \n",
       "193         ma2019               dp      5.0   \n",
       "198         ma2019               dp      5.0   \n",
       "160   national2019               dp      5.0   \n",
       "47    national2019           non_dp      NaN   \n",
       "150         ma2019               dp     10.0   \n",
       "227         tx2019               dp      5.0   \n",
       "404         tx2019               dp     10.0   \n",
       "173   national2019               dp     10.0   \n",
       "242   national2019               dp     10.0   \n",
       "324   national2019              sdc      NaN   \n",
       "12    national2019               dp     10.0   \n",
       "11    national2019               dp     10.0   \n",
       "325   national2019              sdc      NaN   \n",
       "225         tx2019               dp      5.0   \n",
       "394         ma2019               dp      1.0   \n",
       "392         ma2019               dp     10.0   \n",
       "42          ma2019           non_dp      NaN   \n",
       "180         tx2019               dp      5.0   \n",
       "59    national2019               dp    100.0   \n",
       "208         ma2019               dp     10.0   \n",
       "175   national2019               dp      5.0   \n",
       "14    national2019               dp     10.0   \n",
       "100         ma2019              sdc      NaN   \n",
       "\n",
       "                         feature set name  race_edit_distance_percent  \n",
       "449                     custom-features-3                        0.00  \n",
       "428                     custom-features-3                        0.00  \n",
       "429                     custom-features-5                        0.00  \n",
       "440                     custom-features-5                        0.00  \n",
       "58                    demographic-focused                        0.00  \n",
       "450                     custom-features-5                        0.00  \n",
       "439                     custom-features-3                        0.00  \n",
       "35                           all-features                        0.00  \n",
       "38                           all-features                        0.00  \n",
       "210                   demographic-focused                        0.05  \n",
       "32                           all-features                        0.05  \n",
       "219                      industry-focused                        0.07  \n",
       "220                      industry-focused                        0.08  \n",
       "213       demographic-focused-except-DEYE                        0.13  \n",
       "217              family-focused-with-DEYE                        0.15  \n",
       "216                        family-focused                        0.15  \n",
       "212                   demographic-focused                        0.18  \n",
       "214                        family-focused                        0.22  \n",
       "419  demographic-focused-except-AGEP-DEYE                        0.24  \n",
       "236                      industry-focused                        0.26  \n",
       "206                      industry-focused                        0.26  \n",
       "221                       simple-features                        0.29  \n",
       "199                        family-focused                        0.29  \n",
       "224                   demographic-focused                        0.30  \n",
       "229       demographic-focused-except-DEYE                        0.30  \n",
       "204                      industry-focused                        0.31  \n",
       "164  demographic-focused-except-AGEP-DEYE                        0.34  \n",
       "161                   demographic-focused                        0.34  \n",
       "237                      industry-focused                        0.34  \n",
       "197       demographic-focused-except-DEYE                        0.37  \n",
       "322                      industry-focused                        0.40  \n",
       "192                   demographic-focused                        0.42  \n",
       "172                      industry-focused                        0.43  \n",
       "209                          all-features                        0.43  \n",
       "43                    demographic-focused                        0.45  \n",
       "102                   demographic-focused                        0.46  \n",
       "105                      industry-focused                        0.46  \n",
       "234              family-focused-with-DEYE                        0.47  \n",
       "188                       simple-features                        0.47  \n",
       "99                         family-focused                        0.47  \n",
       "146                   demographic-focused                        0.47  \n",
       "201              family-focused-with-DEYE                        0.47  \n",
       "231                        family-focused                        0.50  \n",
       "218                      industry-focused                        0.50  \n",
       "183                        family-focused                        0.50  \n",
       "230                        family-focused                        0.50  \n",
       "195       demographic-focused-except-DEYE                        0.52  \n",
       "106                      industry-focused                        0.52  \n",
       "233              family-focused-with-DEYE                        0.52  \n",
       "101                   demographic-focused                        0.52  \n",
       "76                    demographic-focused                        0.54  \n",
       "171                      industry-focused                        0.55  \n",
       "193                   demographic-focused                        0.55  \n",
       "198                        family-focused                        0.55  \n",
       "160                          all-features                        0.57  \n",
       "47                     custom-features-21                        0.57  \n",
       "150                        family-focused                        0.58  \n",
       "227       demographic-focused-except-DEYE                        0.58  \n",
       "404                        family-focused                        0.58  \n",
       "173                       simple-features                        0.59  \n",
       "242       demographic-focused-except-DEYE                        0.61  \n",
       "324                          all-features                        0.61  \n",
       "12                        simple-features                        0.61  \n",
       "11                           all-features                        0.61  \n",
       "325                   demographic-focused                        0.62  \n",
       "225                   demographic-focused                        0.63  \n",
       "394                      industry-focused                        0.63  \n",
       "392                        family-focused                        0.63  \n",
       "42                     custom-features-21                        0.63  \n",
       "180                   demographic-focused                        0.63  \n",
       "59                    demographic-focused                        0.65  \n",
       "208                       simple-features                        0.65  \n",
       "175                       simple-features                        0.67  \n",
       "14                        simple-features                        0.68  \n",
       "100                      industry-focused                        0.68  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See the random 100 records from index dataframe and\n",
    "# the new column race_edit_distance which we just added to the\n",
    "# index dataframe\n",
    "\n",
    "# Set pandas display max rows option to display more\n",
    "# dataframe rows in the cell output\n",
    "pd.set_option('display.max_rows', 151)\n",
    "# Name of the columns to display in the selected data samples\n",
    "columns_to_show = [LIBRARY_NAME, ALGORITHM_NAME, ALGORITHM_TYPE, TARGET_DATASET,\n",
    "                   PRIVACY_CATEGORY, EPSILON, FEATURE_SET_NAME, RACE_EDIT_DISTANCE_PERCENT]\n",
    "\n",
    "# Sort the samples by race_edit_distance\n",
    "idx_df = idx_df.sort_values(by=RACE_EDIT_DISTANCE_PERCENT, ascending=True)\n",
    "# Show 75 best performing samples in the index dataframe\n",
    "idx_sample_df = idx_df[columns_to_show].head(75)\n",
    "# Show samples dataframe\n",
    "idx_sample_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b39bf8ec-573b-4b60-8f8f-5ff56ea7eb1c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b20f16f4-1408-47b8-80f8-cc82c375fab3",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e71b64ac-0dc8-4604-a6d4-4c308c7bd72e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "crc_new",
   "language": "python",
   "name": "crc_new"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}