00_Data_Prep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation\n",
    "In this notebook, we use a subset of [Stack Exchange network](https://archive.org/details/stackexchange) question data which includes original questions tagged as 'JavaScript', their duplicate questions and their answers. Here, we provide the steps to prepare the data to use for training, tuning, and testing a model that will match a new question with an existing original question. The data files produced are stored in a `data` directory for ease of reference and also to keep them separate from the training script.\n",
    "\n",
    "The data preparation steps are\n",
    "- [import libraries and define parameters](#import),\n",
    "- [ingest the data](#ingest),\n",
    "- [cleanse the data](#cleanse),\n",
    "- [prepare the train, tune, and test datasets](#prepare), and\n",
    "- [save the datasets.](#save)\n",
    "\n",
    "## Imports and parameters <a id='import'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from text_utilities import read_csv_gz, clean_text, round_sample_strat, random_merge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we define some parameters that will be used in the data cleaning as well as train and test set preparation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tune_size   = 0.10 # The proportion of duplicate questions in the tune set.\n",
    "test_size   = 0.10 # The proportion of duplicate questions in the test set.\n",
    "min_text    = 150  # The minimum length of clean text.\n",
    "min_dupes   = 12   # The minimum number of duplicates per question.\n",
    "match       = 40   # The number of duplicate matches.\n",
    "output_path = os.path.join('.', 'data')  # The location of data files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data ingestion <a id='ingest'></a>\n",
    "Next, we download the questions, duplicate questions and answers and load the datasets into pandas dataframes using the helper functions.\n",
    "\n",
    "Create URLs to original questions, duplicate questions, and answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_url = 'https://bostondata.blob.core.windows.net/stackoverflow/{}'\n",
    "questions_url = data_url.format('orig-q.tsv.gz')\n",
    "dupes_url = data_url.format('dup-q.tsv.gz')\n",
    "answers_url = data_url.format('ans.tsv.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = read_csv_gz(questions_url, names=('Id', 'AnswerId', 'Text0', 'CreationDate'))\n",
    "dupes = read_csv_gz(dupes_url, names=('Id', 'AnswerId', 'Text0', 'CreationDate'))\n",
    "answers = read_csv_gz(answers_url, names=('Id', 'Text0'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now check the dataframes. Notice that both questions and duplicates have an \"AnswerID\" column used to match them with the index of the answers. Here are some of the original questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The duplicate questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dupes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the answers to the questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answers.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the first original question's text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions.Text0.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now check the duplicates of that question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dupes[dupes.AnswerId == questions.AnswerId.iloc[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the answer to the original question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answers.Text0[questions.AnswerId.iloc[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data cleaning <a id='cleanse'></a>\n",
    "Next, we use a helper function to clean questions, duplicates, and answers of unwanted text such as code, html tags, and links. These clean texts in lowercase are added in new columns 'Text' in each dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for df in (questions, dupes, answers):\n",
    "    df['Text'] = df.Text0.apply(clean_text).str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep only rows with some clean text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = questions[questions.Text.str.len() > 0]\n",
    "answers = answers[answers.Text.str.len() > 0]\n",
    "dupes = dupes[dupes.Text.str.len() > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the first original question and cleaned version as an example. First, here's an original question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions.Text0.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the question after cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions.iloc[0,3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It turns out that some duplicate questions were also in the original questions, and also some original questions and some duplicate questions had duplicates in their respective datasets. In the following, we remove them from the dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dupes = dupes[~dupes.index.isin(questions.index)] # Remove dupes that are questions.\n",
    "questions = questions[~questions.index.duplicated(keep='first')] # Then remove duplicates from the questions and duplicates.\n",
    "dupes = dupes[~dupes.index.duplicated(keep='first')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also make sure we keep only questions with answers and duplicates, and answers and duplicates that have questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = questions[questions.AnswerId.isin(answers.index) & questions.AnswerId.isin(dupes.AnswerId)]\n",
    "answers = answers[answers.index.isin(questions.AnswerId)]\n",
    "dupes = dupes[dupes.AnswerId.isin(questions.AnswerId)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify the integrity of the remaining data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not questions.AnswerId.isin(answers.index).all():\n",
    "    raise Exception('Not all original questions have answers')\n",
    "if not answers.index.isin(questions.AnswerId).all():\n",
    "    raise Exception('Not all answers have original questions.')\n",
    "if not questions.AnswerId.isin(dupes.AnswerId).all():\n",
    "    raise Exception('Not all original questions have duplicates.')\n",
    "if not dupes.AnswerId.isin(questions.AnswerId).all():\n",
    "    raise Exception('Not all duplicates have original questions.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are counts of unique clean texts in each dataframe, and statistics on the lengths of those texts. There are also statistics on the number of duplicates available for each question. Notice that some questions have only a few duplicates while others have a large number. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Text statistics:')\n",
    "print(pd.DataFrame([questions.Text.str.len().describe()\n",
    "                    .rename('questions'),\n",
    "                    answers.Text.str.len().describe()\n",
    "                    .rename('answers'),\n",
    "                    dupes.Text.str.len().describe()\n",
    "                    .rename('dupes')]))\n",
    "print('\\nDuplication statistics:')\n",
    "print(pd.DataFrame([dupes.AnswerId.value_counts().describe()\n",
    "                    .rename('duplications')]))\n",
    "print('\\nLargest class: {:.2%}'\n",
    "      .format(dupes.AnswerId.value_counts().max()\n",
    "              / dupes.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we reset all dataframe indexes to use them as columns in the remaining steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions.reset_index(inplace=True)\n",
    "answers.reset_index(inplace=True)\n",
    "dupes.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We filter the questions and duplicates to have at least min_text number of characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = questions[questions.Text.str.len() >= min_text]\n",
    "dupes = dupes[dupes.Text.str.len() >= min_text]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, keep only questions with dupes and dupes of questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = questions[questions.AnswerId.isin(dupes.AnswerId)]\n",
    "dupes = dupes[dupes.AnswerId.isin(questions.AnswerId)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we keep questions that have at least min_dupes duplicates, and then keep only those duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answerid_count = dupes.groupby('AnswerId').AnswerId.count()      # Count the number of duplicates by AnswerId.\n",
    "answerid_min = answerid_count.index[answerid_count >= min_dupes] # Find the AnswerIds with at least min_dupes duplicates.\n",
    "questions = questions[questions.AnswerId.isin(answerid_min)]     # Keep only questions with those AnswerIds.\n",
    "dupes = dupes[dupes.AnswerId.isin(answerid_min)]                 # Keep only dupes with those AnswerIds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, verify data integrity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not questions.AnswerId.isin(dupes.AnswerId).all():\n",
    "    raise Exception('Not all original questions have duplicates.')\n",
    "if not dupes.AnswerId.isin(questions.AnswerId).all():\n",
    "    raise Exception('Not all duplicates have original questions.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some statistics on the resulting dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Restrictions: min_text={}, min_dupes={}'\n",
    "      .format(min_text, min_dupes))\n",
    "print('Restricted text statistics:')\n",
    "print(pd.DataFrame([questions.Text.str.len().describe()\n",
    "                    .rename('questions'),\n",
    "                    dupes.Text.str.len().describe()\n",
    "                    .rename('dupes')]))\n",
    "print('\\nRestricted duplication statistics:')\n",
    "print(pd.DataFrame([dupes.AnswerId.value_counts().describe()\n",
    "                    .rename('duplications')]))\n",
    "print('\\nRestricted largest class: {:.2%}'\n",
    "      .format(dupes.AnswerId.value_counts().max()\n",
    "              / dupes.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare train, tune, and test sets <a id='prepare'></a>\n",
    "\n",
    "In this part, we prepare train, tune, and test sets. For training a binary classification model, we will need to construct match and non-match pairs from duplicates and their questions. Finding matching pairs can be accomplished by joining each duplicate with its question. However, non-match examples need to be constructed randomly.\n",
    "\n",
    "As a first step, to make sure we train and test the performance of the model on each question, we will need to have examples of match and non-match pairs for each question both in train and test sets. In order to achieve that, we split the duplicates in a stratified manner into train, tune, and test sets making sure at least 1 or more duplicates per question is in both the tune and tests set depending on the tune_size and test_size parameters and number of duplicates per each question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dupes_test = round_sample_strat(dupes, dupes.AnswerId, frac=test_size)\n",
    "dupes_train = dupes[~dupes.Id.isin(dupes_test.Id)]\n",
    "if not (dupes_test.AnswerId.unique().shape[0] == dupes.AnswerId.unique().shape[0]):\n",
    "    raise Exception('The number of unique questions in dupes_test is not equal to those in dupes.')\n",
    "\n",
    "dupes_tune = round_sample_strat(dupes_train, dupes_train.AnswerId, frac=tune_size)\n",
    "dupes_train = dupes_train[~dupes_train.Id.isin(dupes_tune.Id)]\n",
    "if not (dupes_tune.AnswerId.unique().shape[0] == dupes_train.AnswerId.unique().shape[0]):\n",
    "    raise Exception('The number of unique questions in dupes_tune is not equal to those in dupes_train.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Report on the number of duplicate questions in each dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"dupes_train has {:,} questions\".format(dupes_train.shape[0]))\n",
    "print(\"dupes_tune has {:,} questions\".format(dupes_tune.shape[0]))\n",
    "print(\"dupes_test has {:,} questions\".format(dupes_test.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The names we will use for the relevant columns for text pairs data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_pairs_columns = ['Id_x', 'AnswerId_x', 'Text_x', 'Id_y', 'Text_y', 'AnswerId_y', 'Label', 'n']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we use AnswerId to pair each training duplicate in train set with its matching question and N-1 random questions using the helper function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time balanced_pairs_train = random_merge(dupes_train, questions, N=match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a label for each row such that matching pairs are labeled as 1 and non-matching pairs are labeled as 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_pairs_train['Label'] = (balanced_pairs_train.AnswerId_x == balanced_pairs_train.AnswerId_y).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep only the relevant columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_pairs_train = balanced_pairs_train[balanced_pairs_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sort the data by the dupe ID and the Label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_pairs_train.sort_values(by=['Id_x', 'Label'], ascending=[True, False], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the first few rows of the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "balanced_pairs_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the tune and test sets, we match each duplicate with _all_ the original questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time balanced_pairs_tune = random_merge(dupes_tune, questions, N=questions.shape[0])\n",
    "%time balanced_pairs_test = random_merge(dupes_test, questions, N=questions.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label the rows in same way as was done for the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_pairs_tune['Label'] = (balanced_pairs_tune.AnswerId_x == balanced_pairs_tune.AnswerId_y).astype(int)\n",
    "balanced_pairs_test['Label'] = (balanced_pairs_test.AnswerId_x == balanced_pairs_test.AnswerId_y).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep only the relevant columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_pairs_tune = balanced_pairs_tune[balanced_pairs_columns]\n",
    "balanced_pairs_test = balanced_pairs_test[balanced_pairs_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sort the data by dupe ID and Label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_pairs_tune.sort_values(by=['Id_x', 'Label'], ascending=[True, False], inplace=True)\n",
    "balanced_pairs_test.sort_values(by=['Id_x', 'Label'], ascending=[True, False], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the first few rows of the test data. The tune data are similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_pairs_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We report statistics the train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('balanced_pairs_train: {:,} rows with {:.2%} matches'\n",
    "      .format(balanced_pairs_train.shape[0], \n",
    "              balanced_pairs_train.Label.mean()))\n",
    "print('balanced_pairs_tune: {:,} rows with {:.2%} matches'\n",
    "      .format(balanced_pairs_tune.shape[0], \n",
    "              balanced_pairs_tune.Label.mean()))\n",
    "print('balanced_pairs_test: {:,} rows with {:.2%} matches'\n",
    "      .format(balanced_pairs_test.shape[0], \n",
    "              balanced_pairs_test.Label.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the datasets <a id='save'></a>\n",
    "Finally, we save as text files the questions, the train and test duplicates, and the train and test dataframes of duplicate-question pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(output_path, exist_ok=True)\n",
    "\n",
    "questions_path = os.path.join(output_path, 'questions.tsv')\n",
    "print('Writing {:,} rows to {}'.format(questions.shape[0], questions_path))\n",
    "questions.to_csv(questions_path, sep='\\t',header=True, index=False)\n",
    "\n",
    "answers_path = os.path.join(output_path, 'answers.tsv')\n",
    "print('Writing {:,} rows to {}'.format(answers.shape[0], answers_path))\n",
    "answers.to_csv(answers_path, sep='\\t',header=True, index=False)\n",
    "\n",
    "dupes_train_path = os.path.join(output_path, 'dupes_train.tsv')\n",
    "print('Writing {:,} rows to {}'.format(dupes_train.shape[0], dupes_train_path))\n",
    "dupes_train.to_csv(dupes_train_path, sep='\\t',header=True, index=False)\n",
    "\n",
    "dupes_tune_path = os.path.join(output_path, 'dupes_tune.tsv')\n",
    "print('Writing {:,} rows to {}'.format(dupes_tune.shape[0], dupes_tune_path))\n",
    "dupes_tune.to_csv(dupes_tune_path, sep='\\t',header=True, index=False)\n",
    "\n",
    "dupes_test_path = os.path.join(output_path, 'dupes_test.tsv')\n",
    "print('Writing {:,} rows to {}'.format(dupes_test.shape[0], dupes_test_path))\n",
    "dupes_test.to_csv(dupes_test_path, sep='\\t',header=True, index=False)\n",
    "\n",
    "balanced_pairs_train_path = os.path.join(output_path, 'balanced_pairs_train.tsv')\n",
    "print('Writing {:,} rows to {}'.format(balanced_pairs_train.shape[0], balanced_pairs_train_path))\n",
    "balanced_pairs_train.to_csv(balanced_pairs_train_path, sep='\\t',header=True, index=False)\n",
    "\n",
    "balanced_pairs_tune_path = os.path.join(output_path, 'balanced_pairs_tune.tsv')\n",
    "print('Writing {:,} rows to {}'.format(balanced_pairs_tune.shape[0], balanced_pairs_tune_path))\n",
    "balanced_pairs_tune.to_csv(balanced_pairs_tune_path, sep='\\t', header=True, index=False)\n",
    "\n",
    "balanced_pairs_test_path = os.path.join(output_path, 'balanced_pairs_test.tsv')\n",
    "print('Writing {:,} rows to {}'.format(balanced_pairs_test.shape[0], balanced_pairs_test_path))\n",
    "balanced_pairs_test.to_csv(balanced_pairs_test_path, sep='\\t', header=True, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now move on to [defining the model training script](01_Training_Script.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}