docs/notebooks/reproduce_results.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reproducing results\n",
    "\n",
    "\n",
    "The following code reproduces results for Slab dataset, Rotated MNIST and Fashion-MNIST dataset, and PACS dataset corresponding to Tables 1, 2, 3, 4, 5, 6 in the main paper.\n",
    "\n",
    "## Note regarding hardware requirements\n",
    "\n",
    "The code requires a GPU device, also the batch size for MatchDG Phase 1 training might need to be adjusted according to the memory limits of the GPU device. In case of CUDA of out of memory issues, try with a smaller batch size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Libraries\n",
    "\n",
    "List of all the required packages are mentioned in the file 'requirements.txt'\n",
    "\n",
    "You may install them as follows: `pip install -r requirements.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 1: Slab Dataset\n",
    "\n",
    "## Prepare Slab Dataset\n",
    "\n",
    "Run the following command:\n",
    "\n",
    "`python3 data_gen_syn.py`\n",
    "\n",
    "## Table 1\n",
    "\n",
    "Run the following command:\n",
    "\n",
    "`python3 reproduce_scripts/reproduce_slab.py train`\n",
    "\n",
    "The results would be stored in the `results/slab/logs/` directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 2, 3: RotMNIST & Fashion MNIST\n",
    "\n",
    "## Prepare Data for Rot MNIST & Fashion MNIST\n",
    "\n",
    "Run the following command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python data/data_gen_mnist.py --dataset rot_mnist --model resnet18 --img_h 224 --img_w 224 --subset_size 2000` \n",
    "\n",
    "`python data/data_gen_mnist.py --dataset fashion_mnist --model resnet18 --img_h 224 --img_w 224 --subset_size 10000`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2\n",
    "\n",
    "Rotated MNIST dataset with training domains set to [15, 30, 45, 60, 75] and the test domains set to [0, 90]. \n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_all --metric train`\n",
    "\n",
    "The results would be present in the `results/rot_mnist/train_logs/` directory\n",
    "\n",
    "To obtain results for the FashionMNIST dataset, change the dataset parameter `--dataset` from `rot_mnist` to `fashion_mnist`.\n",
    "\n",
    "To obtain results for the different set of training domains in the paper, change the input to the parameter `--train_case` to `train_abl_3` for training with domains [30, 45, 60], and `train_abl_2` for training with domains [30, 45] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 3\n",
    "\n",
    "Run the following commands:\n",
    "\n",
    "`python test.py --dataset rot_mnist --method_name erm_match --match_case 0.0 --penalty_ws 0.0 --test_metric match_score`\n",
    "\n",
    "`python test.py --dataset rot_mnist --method_name matchdg_ctr --match_case 0.0 --match_flag 1 --pos_metric cos --test_metric match_score`\n",
    "\n",
    "For MDG Perf, run the folllowing command to first train the model:\n",
    "\n",
    "`python3 reproduce_scripts/mnist_mdg_ctr_run.py --dataset rot_mnist --perf_init 1`\n",
    "\n",
    "Then run the following commands to evalute match function metrics:\n",
    "\n",
    "`python test.py --dataset rot_mnist --method_name matchdg_ctr --match_case 1.0 --match_flag 1 --pos_metric cos --test_metric match_score`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 4, 5: PACS\n",
    "\n",
    "## Prepare Data for PACS\n",
    "\n",
    "Download the PACS dataset (https://drive.google.com/drive/folders/0B6x7gtvErXgfUU1WcGY5SzdwZVk?resourcekey=0-2fvpQY_QSyJf2uIECzqPuQ&usp=sharing) and place it in the directory '/data/datasets/pacs/' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 4\n",
    "\n",
    "* RandMatch: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python3 reproduce_scripts/pacs_run.py --method rand --model resnet18`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* MatchDG:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For contrastive phase, we train with the resnet50 model despite the model architecture in Phase 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python3 reproduce_scripts/pacs_run.py --method matchdg_ctr --model resnet50`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python3 reproduce_scripts/pacs_run.py --method matchdg_erm --model resnet18`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* MDGHybrid:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No need to train the contrastive phase again if already done while training MatchDG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python3 reproduce_scripts/pacs_run.py --method hybrid --model resnet18`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results would be stored in the `results/pacs/logs/` directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 5\n",
    "\n",
    "Repeat the above commands and replace the argument to flag --model with resnet50 with resnet18 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 6: Chest X-Ray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Data for Chest X-Ray\n",
    "\n",
    "    -Follow the steps in the Preprocess.ipynb notebook to donwload and process the Chest X-Ray datasets\n",
    "    -Then follow the steps in the ChestXRay_Translate.ipynb notebook to perform image translations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* NIH: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python3 reproduce_scripts/cxray_run.py --test_domain nih --metric train`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Chexpert: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python3 reproduce_scripts/cxray_run.py --test_domain chex --metric train`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* RSNA: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python3 reproduce_scripts/cxray_run.py --test_domain kaggle --metric train`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results would be stored in the `results/chestxray/train_logs` directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 11\n",
    "\n",
    "Run the following command for data generation:\n",
    "\n",
    "`python data/data_gen_mnist.py --dataset rot_mnist --model lenet --img_h 32 --img_w 32 --subset_size 1000`\n",
    "\n",
    "Run the following commands for training models:\n",
    "\n",
    "`python3 reproduce_rmnist_lenet.py`\n",
    "\n",
    "The results will be stored in the directory: `results/rmnist_lenet/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 12\n",
    "\n",
    "Run the following command for data generation:\n",
    "\n",
    "`python3 data/data_gen_domainbed.py`\n",
    "\n",
    "Run the following command for training models:\n",
    "\n",
    "`python3 reproduce_rmnist_domainbed.py`\n",
    "\n",
    "The results will be stored in the directory: `results/rmnist_domain_bed/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 13\n",
    "\n",
    "To obtain results for the FashionMNIST dataset, change the dataset parameter `--dataset` from `rot_mnist` to `fashion_mnist`.\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_all --metric train --methods approx_25 approx_50 approx_75`\n",
    "\n",
    "The results will be stored in the directory: `results/rot_mnist/train_logs/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 14\n",
    "\n",
    "To obtain results for the FashionMNIST dataset, change the dataset parameter `--dataset` from `rot_mnist` to `fashion_mnist`.\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_all --metric match_score --data_case train --methods rand perf matchdg`\n",
    "\n",
    "The results would be stored in the directory: `results/rot_mnist/match_score_train/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 15\n",
    "\n",
    "Generate data again for the Fashion MNIST 2k sample case by running the following command:\n",
    "\n",
    "`python data/data_gen_mnist.py --dataset fashion_mnist --model resnet18 --img_h 224 --img_w 224 --subset_size 2000`\n",
    "\n",
    "Then follow the same commands as mentioned in the Table 2 section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 16\n",
    "\n",
    "To obtain results for the FashionMNIST dataset, change the dataset parameter `--dataset` from `rot_mnist` to `fashion_mnist`.\n",
    "\n",
    "MatchDG Iterative corresponds to the default MatchDG algorithm, with the same results as in Table 3\n",
    "\n",
    "For MatchDG Non Iterative, run the folllowing command to first the model\n",
    "\n",
    "`python3 reproduce_scripts/mnist_mdg_ctr_run.py --dataset rot_mnist --iterative 0`\n",
    "\n",
    "Then run the following command to evaluate match function metrics:\n",
    "\n",
    "`python test.py --dataset rot_mnist --method_name matchdg_ctr --match_case 0.0 --match_flag 0 --pos_metric cos --test_metric match_score`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table 18\n",
    "\n",
    "Repeat the commands mentioned for PACS ResNet-18 (Table 4) and replace the argument to flag --model with alexnet with resnet18 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Privacy & OOD: ICLR\n",
    "\n",
    "# Slab Dataset\n",
    "\n",
    "### Preparing Data\n",
    "\n",
    "`python3 data_gen_syn.py`\n",
    "\n",
    "### Training Models\n",
    "\n",
    "Run the following command to train models with no noise in the prediction mechanism based on slab features\n",
    "\n",
    "`python3 reproduce_scripts/slab-run.py --slab_noise 0.0`\n",
    "\n",
    "Run the following command to train models with noise in the prediction mechanism based on slab features\n",
    "\n",
    "`python3 reproduce_scripts/slab-run.py --slab_noise 0.10`\n",
    "\n",
    "\n",
    "### Evaluating OOD Accuracy, Randomised-AUC, & Privacy Loss Attack\n",
    "\n",
    "Run the following command for the case of no noise in the prediction mechanism based on slab features\n",
    "\n",
    "`python3 reproduce_scripts/slab-run.py --slab_noise 0.0 --case test`\n",
    "\n",
    "Run the following command for the case of noise in the prediction mechanism based on slab features\n",
    "\n",
    "`python3 reproduce_scripts/slab-run.py --slab_noise 0.10 --case test`\n",
    "\n",
    "### Plotting Results\n",
    "\n",
    "`python3 reproduce_scripts/slab-plot.py 0.0`\n",
    "\n",
    "`python3 reproduce_scripts/slab-plot.py 0.1`\n",
    "\n",
    "The plots would be stored in the directory: `results/slab/plots/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rotated & Fashion MNIST \n",
    "\n",
    "For convenience we provide the commands for the Rotated MNIST dataset. To obtain results for the FashionMNIST dataset, change the dataset parameter `--dataset` from `rot_mnist` to `fashion_mnist`.\n",
    "\n",
    "### Preparing Data\n",
    "\n",
    "`python data/data_gen_mnist.py --dataset rot_mnist --model resnet18 --img_h 224 --img_w 224 --subset_size 2000` \n",
    "\n",
    "\n",
    "### Training Models\n",
    "\n",
    "Training Domains: [15, 30, 45, 60, 75]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_all --metric train --data_aug 0`\n",
    "\n",
    "Training Domains: [30, 45, 60]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_3 --metric train --data_aug 0`\n",
    "\n",
    "Training Domains: [30, 45]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_2 --metric train --data_aug 0`\n",
    "\n",
    "The results would be present in the results/rot_mnist/train_logs/ directory\n",
    "\n",
    "\n",
    "### Evaluating OOD Accuracy\n",
    "\n",
    "Training Domains: [15, 30, 45, 60, 75]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_all --metric acc --data_case train --data_aug 0 `\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_all --metric acc --data_case test --data_aug 0 `\n",
    "\n",
    "Training Domains: [30, 45, 60]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_3 --metric acc --data_case train --data_aug 0`\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_3 --metric acc --data_case test --data_aug 0`\n",
    "\n",
    "Training Domains: [30, 45]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_2 --metric acc --data_case train --data_aug 0`\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_2 --metric acc --data_case test --data_aug 0`\n",
    "\n",
    "\n",
    "\n",
    "### Evaluating MI Attack Accuracy\n",
    "\n",
    "Training Domains: [15, 30, 45, 60, 75]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_all --metric privacy_loss_attack --data_aug 0 `\n",
    "\n",
    "Training Domains: [30, 45, 60]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_3 --metric privacy_loss_attack --data_aug 0`\n",
    "\n",
    "Training Domains: [30, 45]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_2 --metric privacy_loss_attack --data_aug 0`\n",
    "\n",
    "\n",
    "### Evaluating Mean Rank\n",
    "\n",
    "Training Domains: [15, 30, 45, 60, 75]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_all --metric match_score --data_case test --data_aug 0 `\n",
    "\n",
    "Training Domains: [30, 45, 60]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_3 --metric match_score --data_case test --data_aug 0`\n",
    "\n",
    "Training Domains: [30, 45]\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --train_case train_abl_2 --metric match_score --data_case test --data_aug 0`\n",
    "\n",
    "### Plotting Results\n",
    "\n",
    "`python3 reproduce_scripts/mnist_plot.py rot_mnist`\n",
    "\n",
    "The plots would be stored in the directory: `results/rot_mnist/plots/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Differentially Private Noise\n",
    "\n",
    "For convenience we provide the commands for the Rotated MNIST dataset. To obtain results for the FashionMNIST dataset, change the dataset parameter `--dataset` from `rot_mnist` to `fashion_mnist`.\n",
    "\n",
    "The command below produces results for the case of epsilon 1.0; repeat the same command by changing the input to the paramter `--dp_epsilon` to the other values from the list: [1, 2, 5, 10]. \n",
    "\n",
    "\n",
    "### Training Models\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --dp_noise 1 --dp_epsilon 1.0 --data_aug 0 --methods erm perf`\n",
    "\n",
    "### Evaluating OOD Accuracy\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --dp_noise 1 --dp_epsilon 1.0 --data_aug 0 --methods erm perf --metric acc --data_case test `\n",
    "\n",
    "\n",
    "### Evaluating MI Attack Accuracy\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist --dp_noise 1 --dp_epsilon 1.0 --data_aug 0 --methods erm perf --metric privacy_loss_attack`\n",
    "\n",
    "### Infinite Epsilon Case\n",
    "\n",
    "Append this extra parameter ` --dp_attach_opt 0 ` to all the commands above. This does not attach the differential privacy engine with the optimizer. Also, change the epsilon value to the parameter ` --dp_epsilon ` to any random value as it does not matter since the privacy engine is not attached to the optimizer\n",
    "\n",
    "### Plotting Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ChestXRay Dataset\n",
    "\n",
    "### Prepare Data for Chest X-Ray\n",
    "\n",
    "    -Follow the steps in the Preprocess.ipynb notebook to donwload and process the Chest X-Ray datasets\n",
    "    -Then follow the steps in the ChestXRay_Translate.ipynb notebook to perform image translations\n",
    "\n",
    "### Training Models\n",
    "\n",
    "Test Domain NIH\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain nih --metric train`\n",
    "\n",
    "Test Domain Chexpert\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain chex --metric train`\n",
    "\n",
    "Test Domain RSNA\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain kaggle --metric train`\n",
    "\n",
    "### Evaluating OOD Accuracy\n",
    "\n",
    "Test Domain NIH\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain nih --metric acc --data_case train`\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain nih --metric acc --data_case test`\n",
    "\n",
    "Test Domain Chexpert\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain chex --metric acc --data_case train`\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain chex --metric acc --data_case test`\n",
    "\n",
    "Test Domain RSNA\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain kaggle --metric acc --data_case train`\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain kaggle --metric acc --data_case test`\n",
    "\n",
    "\n",
    "### Evaluating MI Attack Accuracy\n",
    "\n",
    "Test Domain NIH\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain nih --metric privacy_loss_attack`\n",
    "\n",
    "Test Domain Chexpert\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain chex --metric privacy_loss_attack`\n",
    "\n",
    "Test Domain RSNA\n",
    "\n",
    "`python3 reproduce_scripts/cxray_run.py --test_domain kaggle --metric privacy_loss_attack`\n",
    "\n",
    "\n",
    "### Plotting Results\n",
    "\n",
    "`python3 reproduce_scripts/cxray_plot.py`\n",
    "\n",
    "The plots would be stored in the directory: `results/chestxray/plots/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attribute Attack\n",
    "\n",
    "### Preparing Data\n",
    "\n",
    "`python data/data_gen_mnist.py --dataset rot_mnist_spur --model resnet18 --img_h 224 --img_w 224 --subset_size 2000`\n",
    "\n",
    "### Training Models\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist_spur --train_case train_all --metric train --data_aug 0`\n",
    "\n",
    "### Evaluating OOD Accuracy\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist_spur --train_case train_all --metric acc --data_case train --data_aug 0`\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist_spur --train_case train_all --metric acc --data_case test --data_aug 0`\n",
    "\n",
    "### Evaluating AI Attack Accuracy\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist_spur --train_case train_all --metric attribute_attack --data_case 0 --data_aug 0`\n",
    "\n",
    "### Evaluating OOD Accuracy on the permuted test domain\n",
    "\n",
    "Generate data with permuted test domain\n",
    "\n",
    "`python data/data_gen_mnist.py --dataset rot_mnist_spur --model resnet18 --img_h 224 --img_w 224 --subset_size 2000 --cmnist_permute 1`\n",
    "\n",
    "Run the following command to obtain OOD accuracy on permuted test domain\n",
    "\n",
    "`python3 reproduce_scripts/mnist_run.py --dataset rot_mnist_spur --train_case train_all --metric acc --data_case test --data_aug 0`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}