370_Mid-term-student.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SI 370 FA 2018: Mid-term Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Please turn in HTML and notebook file on canvas by 10:30am."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "import statsmodels.api as sm\n",
    "import statsmodels.formula.api as smf\n",
    "from statsmodels.graphics.mosaicplot import mosaic\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn import metrics\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This test has 7 questions and is worth 150 points.  You may use any online or paper-based resources *except* real-time, direct communication with other people (i.e. instant messaging)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1 (Basic pandas, 15 points)):\n",
    "### 1a: Write code to create the following DataFrame:\n",
    "![](resources/midterm-dataframe1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1b: Write code to use a universal function to transform the DataFrame you created in 1a to the following DataFrame:\n",
    "\n",
    "![](resources/midterm-dataframe2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2 (Pandas and code flow, 15 points):\n",
    "Put the following 5 code blocks in the correct order so the output is \n",
    "```\n",
    "2     1075000.0\n",
    "6     1762000.0\n",
    "13     774294.0\n",
    "Name: current_salary_cleaned, dtype: float64\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players_salaries = nfl_football_players.dropna(subset=['current_salary_cleaned'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players['current_salary_cleaned'] = nfl_football_players['current_salary_nocommas'].astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players = pd.read_json('data/nfl_football_profiles.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players_salaries['current_salary_cleaned'].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players['current_salary_nocommas'] = nfl_football_players['current_salary'].str.replace(',', '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3 (Visualization and Analysis, 20 points): "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall the wine quality data dataset that we used in this class.  In the code below, we load the data and then\n",
    "separate it into \"good\" (i.e. quality > 6) and \"bad\" (i.e. quality <= 5) categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine = pd.read_csv('data/winequality-red.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine['good_or_bad'] = np.where(wine['quality'] > 6, 'good','bad')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3a. Create a boxplot that shows the values of pH for good and bad wines.  \n",
    "Based on your visualization, does it look like there is a significant difference of the pH between good and bad wines?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the results of a t-test on pH for good and bad wines using the following code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import ttest_ind\n",
    "goodwine = wine[wine['good_or_bad'] == 'good']\n",
    "badwine = wine[wine['good_or_bad'] == 'bad']\n",
    "results = ttest_ind(goodwine['pH'],badwine['pH'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.pvalue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3b.  Based on your t-test, is the difference between good and bad wines significant?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Insert your interpretation here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4 (Correlation & Regression, 20 points):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This question also uses the wine quality dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4a. Fill in the correct formula to create two regressions of pH vs. \"total sulfur dioxide\", one for \"good\" wines and one for \"bad\" wines (code for the regression is given to you below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "formula = \"INSERT YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model0 = smf.ols(formula, data=goodwine).fit()\n",
    "model0.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model0 = smf.ols(formula, data=badwine).fit()\n",
    "model0.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4b. How do the results of the two regressions differ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5 (Categorical variables, 20 points):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains details about a list of comic book characters from DC and Marvel, such as:  sex and sexuality, hair color, eye color, whether the characters were good guys or bad guys or neutral, whether they had secret identities, and whether they were alive or dead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comic_characters = pd.read_csv(\"data/comic_characters.csv\", index_col=\"id\")\n",
    "comic_characters.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comic_characters = comic_characters[(comic_characters['Identity']== 'Secret Identity') | (comic_characters['Identity'] == 'Public Identity')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5a. Create a crosstab of Identity vs. alive status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5b. Create a mosaic plot of identity vs alive status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 6 (Machine Learning: Clustering, 20 points):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For clustering, it's sometimes helpful to create dummy variables.  For example, the \"Identity\" property can be replaced by Identity_Public Identity and Indentity_Secret Identity.  The following code does that for several of the categorical variables in the comic characters dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comic_dummies = pd.get_dummies(comic_characters,columns=['Identity','align','eye','hair','sex','alive'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comic_dummies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code creates a new DataFrame with only the resulting dummy variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comic_dummies_vars = comic_dummies.iloc[:,7:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 6: Find the optimal value of \"k\" for k-means clustering of the comic character dataset with dummy variables (i.e. comic_dummies_vars)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 7 (Natural Language Processing, 20 points):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data/vox_transcript.csv file that we distributed as part of this test contains a transcript of the first 2016 Presidential Debate between Clinton and Trump.  \n",
    "\n",
    "Run the following code block to load and peek at the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "debate = pd.read_csv('data/vox_transcript.csv')\n",
    "debate.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.lang.en.stop_words import STOP_WORDS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the data so we have one DataFrame for Trump and one DataFrame for Clinton"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trump = debate[debate['speaker'] == 'Trump']\n",
    "clinton = debate[debate['speaker'] == 'Clinton']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code concatenates all the text from each of those dataframes into strings (one for each of Trump and Clinton):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trump_debate_text = trump['text'].str.cat(sep = ' ')\n",
    "clinton_debate_text = clinton['text'].str.cat(sep = ' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(trump_debate_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = [x.pos_ for x in doc if x.text not in STOP_WORDS]\n",
    "#Counter(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(45,10))\n",
    "sns.countplot(labels, order=pd.Series(labels).value_counts().index)\n",
    "# sns.countplot(words_nostop, order=[counted[0] for counted in Counter(words_nostop).most_common()])\n",
    "plt.xticks(rotation=90)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7a. Repeat the above steps on the Clinton text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7b. Describe how these distributions differ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Insert your interpretation here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Please turn in HTML and notebook file on canvas by 10:30am."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}