B-Cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 3: 'AskFeminists' vs. 'MensRights'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part B. Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I kept the AskFeminists and MensRights dataframes separate during the cleaning process. For submissions, I combined 'title' and 'selftext' content into one 'text' column. I created columns to track posts/comments that were removed (by a moderator) or deleted (by a poster). AskFeminists had 1,847 removed posts/comments, about twice as many as MensRights. I removed links, numbers, characters, and references to '[removed]' or '[deleted]' from the text and lemmatized it (keeping stopwords in.) After lemmatizing, I removed contraction leftovers (ex. 'nt', 've') so they wouldn't skew the vectorizer counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libaries\n",
    "import pandas as pd\n",
    "import regex as re\n",
    "import nltk\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "pd.options.display.max_columns = 999"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in the two csvs\n",
    "fem = pd.read_csv('./data/askfeminists_all')\n",
    "men = pd.read_csv('./data/mensrights_all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop unnamed column for both\n",
    "fem.drop(columns='Unnamed: 0', inplace = True)\n",
    "men.drop(columns='Unnamed: 0', inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# making text column string\n",
    "fem['text'] = fem['text'].astype('str')\n",
    "men['text'] = men['text'].astype('str')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Making Columns for Removed/Deleted posts/comments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# making column for 'removed'\n",
    "fem['removed'] = [1 if '[removed]' in i else 0 for i in fem['text']]\n",
    "# making column for 'deleted'\n",
    "fem['deleted'] = [1 if '[deleted]' in i else 0 for i in fem['text']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# making column for 'removed'\n",
    "men['removed'] = [1 if '[removed]' in i else 0 for i in men['text']]\n",
    "# making column for 'deleted'\n",
    "men['deleted'] = [1 if '[deleted]' in i else 0 for i in men['text']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Removing [Removed]/[Deleted]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for fem\n",
    "fem['text'] = fem['text'].str.replace(r'[[][r][e][m][o][v][e][d][]]', '')\n",
    "fem['text'] = fem['text'].str.replace(r'[[][d][e][l][e][t][e][d][]]', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for men\n",
    "men['text'] = men['text'].str.replace(r'[[][r][e][m][o][v][e][d][]]', '')\n",
    "men['text'] = men['text'].str.replace(r'[[][d][e][l][e][t][e][d][]]', '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating column for cleaned data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loosely based on Matt's code to clean text\n",
    "def review_to_words(raw_review):\n",
    "    no_links = re.sub(\"(https.\\S+)\", \" \", raw_review) # from Maurie, remove https links\n",
    "    no_links_2 = re.sub(\"(http.\\S+)\", \" \", no_links) # from Maurie, remove http links       \n",
    "    letters_only = re.sub(\"[^a-zA-Z]\", \" \", no_links_2) # remove non-letters\n",
    "    lower = letters_only.lower() # convert to lowercase\n",
    "    return lower"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create column for clean text with stop words\n",
    "fem['clean_text_stop'] = fem['text'].apply(lambda x: review_to_words(str(x)))\n",
    "men['clean_text_stop'] = men['text'].apply(lambda x: review_to_words(str(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking Removed/Deleted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking how many posts/comments deleted or removed for each subreddit. AskFeminists has more removed content, suggesting more posters who violated the rules of the sub (either more trolling, or stricter moderators?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    27458\n",
       "1    1847 \n",
       "Name: removed, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking how many removed/deleted posts in each\n",
    "fem['removed'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    30504\n",
       "1    942  \n",
       "Name: removed, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "men['removed'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    29193\n",
       "1    112  \n",
       "Name: deleted, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fem['deleted'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    30599\n",
       "1    847  \n",
       "Name: deleted, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "men['deleted'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatizing Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>type</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>removed</th>\n",
       "      <th>deleted</th>\n",
       "      <th>clean_text_stop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Should students still be shown Neil DeGrasse Tyson's videos?Hi, feminists. I'm a middle school science teacher. Astronomy is part of my curriculum. \\n\\nOver the past 10 years, Neil DeGrasse Tyson seems to have devoted his life to appearing in every damn astronomy video made. Like, he really is in all of them! And, for the most part, the videos are good and can be valuable to kids' education.\\n\\nHe was accused this past week, by several different women, of various forms of sexual misconduct. The accusations range from creepiness to rape.\\n\\nI've tried my best to be supportive of the metoo movement and all that's come with it. \\n\\nSo there's my dilemma. Should I continue to show videos to my students that are hosted by Neil DeGrasse Tyson?</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>should students still be shown neil degrasse tyson s videos hi  feminists  i m a middle school science teacher  astronomy is part of my curriculum    over the past    years  neil degrasse tyson seems to have devoted his life to appearing in every damn astronomy video made  like  he really is in all of them  and  for the most part  the videos are good and can be valuable to kids  education   he was accused this past week  by several different women  of various forms of sexual misconduct  the accusations range from creepiness to rape   i ve tried my best to be supportive of the metoo movement and all that s come with it    so there s my dilemma  should i continue to show videos to my students that are hosted by neil degrasse tyson</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Does games like BFV and Call of Duty where there are female avatars promote/normalise violence towards women ?in the game women who get shot also cry in pain</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>does games like bfv and call of duty where there are female avatars promote normalise violence towards women  in the game women who get shot also cry in pain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Feminist political stance?</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>what is feminist political stance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What makes some victim blaming okay, and some not okay?For example if someone leaved their wallet on a bench, you'd victim blame them.</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>what makes some victim blaming okay  and some not okay for example if someone leaved their wallet on a bench  you d victim blame them</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>how do you feel that you are sandwich makers and dishwashers?</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>how do you feel that you are sandwich makers and dishwashers</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          text  \\\n",
       "0  Should students still be shown Neil DeGrasse Tyson's videos?Hi, feminists. I'm a middle school science teacher. Astronomy is part of my curriculum. \\n\\nOver the past 10 years, Neil DeGrasse Tyson seems to have devoted his life to appearing in every damn astronomy video made. Like, he really is in all of them! And, for the most part, the videos are good and can be valuable to kids' education.\\n\\nHe was accused this past week, by several different women, of various forms of sexual misconduct. The accusations range from creepiness to rape.\\n\\nI've tried my best to be supportive of the metoo movement and all that's come with it. \\n\\nSo there's my dilemma. Should I continue to show videos to my students that are hosted by Neil DeGrasse Tyson?   \n",
       "1  Does games like BFV and Call of Duty where there are female avatars promote/normalise violence towards women ?in the game women who get shot also cry in pain                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "2  What is Feminist political stance?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n",
       "3  What makes some victim blaming okay, and some not okay?For example if someone leaved their wallet on a bench, you'd victim blame them.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
       "4  how do you feel that you are sandwich makers and dishwashers?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "\n",
       "   type     subreddit  removed  deleted  \\\n",
       "0  post  AskFeminists  0        0         \n",
       "1  post  AskFeminists  0        0         \n",
       "2  post  AskFeminists  1        0         \n",
       "3  post  AskFeminists  0        0         \n",
       "4  post  AskFeminists  1        0         \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       clean_text_stop  \n",
       "0  should students still be shown neil degrasse tyson s videos hi  feminists  i m a middle school science teacher  astronomy is part of my curriculum    over the past    years  neil degrasse tyson seems to have devoted his life to appearing in every damn astronomy video made  like  he really is in all of them  and  for the most part  the videos are good and can be valuable to kids  education   he was accused this past week  by several different women  of various forms of sexual misconduct  the accusations range from creepiness to rape   i ve tried my best to be supportive of the metoo movement and all that s come with it    so there s my dilemma  should i continue to show videos to my students that are hosted by neil degrasse tyson   \n",
       "1  does games like bfv and call of duty where there are female avatars promote normalise violence towards women  in the game women who get shot also cry in pain                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
       "2  what is feminist political stance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \n",
       "3  what makes some victim blaming okay  and some not okay for example if someone leaved their wallet on a bench  you d victim blame them                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
       "4  how do you feel that you are sandwich makers and dishwashers                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fem.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting up tokenizer and lemmatizer\n",
    "tokenizer = RegexpTokenizer(r'\\w+')\n",
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to lemmatize\n",
    "def lemma(text):\n",
    "    tokens = tokenizer.tokenize(str(text))\n",
    "    lems = [lemmatizer.lemmatize(i) for i in tokens]\n",
    "    \n",
    "    return(\" \".join(lems))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add column for lemmatized words for fems\n",
    "fem['lems'] = fem['clean_text_stop'].apply(lambda x: lemma(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add column for lemmatized words for men\n",
    "men['lems'] = men['clean_text_stop'].apply(lambda x: lemma(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Cleaning up contraction leftovers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to remove hanging contraction leftovers\n",
    "def nocontract(x):\n",
    "    x = re.sub(\"([ ][r][e][ ])\", \" \", x)\n",
    "    x = re.sub(\"([ ][v][e][ ])\", \" \", x)\n",
    "    x = re.sub(\"([ ][l][l][ ])\", \" \", x)\n",
    "    x = re.sub(\"([ ][d][ ])\", \" \", x)\n",
    "    x = re.sub(\"([ ][t][ ])\", \" \", x)\n",
    "    x = re.sub(\"([ ][m][ ])\", \" \", x)\n",
    "    x = re.sub(\"([ ][s][ ])\", \" \", x)\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# applying no contractions function to both dfs\n",
    "fem['lems'] = fem['lems'].apply(lambda x: nocontract(x))\n",
    "men['lems'] = men['lems'].apply(lambda x: nocontract(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>type</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>removed</th>\n",
       "      <th>deleted</th>\n",
       "      <th>clean_text_stop</th>\n",
       "      <th>lems</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Should students still be shown Neil DeGrasse Tyson's videos?Hi, feminists. I'm a middle school science teacher. Astronomy is part of my curriculum. \\n\\nOver the past 10 years, Neil DeGrasse Tyson seems to have devoted his life to appearing in every damn astronomy video made. Like, he really is in all of them! And, for the most part, the videos are good and can be valuable to kids' education.\\n\\nHe was accused this past week, by several different women, of various forms of sexual misconduct. The accusations range from creepiness to rape.\\n\\nI've tried my best to be supportive of the metoo movement and all that's come with it. \\n\\nSo there's my dilemma. Should I continue to show videos to my students that are hosted by Neil DeGrasse Tyson?</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>should students still be shown neil degrasse tyson s videos hi  feminists  i m a middle school science teacher  astronomy is part of my curriculum    over the past    years  neil degrasse tyson seems to have devoted his life to appearing in every damn astronomy video made  like  he really is in all of them  and  for the most part  the videos are good and can be valuable to kids  education   he was accused this past week  by several different women  of various forms of sexual misconduct  the accusations range from creepiness to rape   i ve tried my best to be supportive of the metoo movement and all that s come with it    so there s my dilemma  should i continue to show videos to my students that are hosted by neil degrasse tyson</td>\n",
       "      <td>should student still be shown neil degrasse tyson video hi feminist i a middle school science teacher astronomy is part of my curriculum over the past year neil degrasse tyson seems to have devoted his life to appearing in every damn astronomy video made like he really is in all of them and for the most part the video are good and can be valuable to kid education he wa accused this past week by several different woman of various form of sexual misconduct the accusation range from creepiness to rape i tried my best to be supportive of the metoo movement and all that come with it so there my dilemma should i continue to show video to my student that are hosted by neil degrasse tyson</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Does games like BFV and Call of Duty where there are female avatars promote/normalise violence towards women ?in the game women who get shot also cry in pain</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>does games like bfv and call of duty where there are female avatars promote normalise violence towards women  in the game women who get shot also cry in pain</td>\n",
       "      <td>doe game like bfv and call of duty where there are female avatar promote normalise violence towards woman in the game woman who get shot also cry in pain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Feminist political stance?</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>what is feminist political stance</td>\n",
       "      <td>what is feminist political stance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What makes some victim blaming okay, and some not okay?For example if someone leaved their wallet on a bench, you'd victim blame them.</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>what makes some victim blaming okay  and some not okay for example if someone leaved their wallet on a bench  you d victim blame them</td>\n",
       "      <td>what make some victim blaming okay and some not okay for example if someone leaved their wallet on a bench you victim blame them</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>how do you feel that you are sandwich makers and dishwashers?</td>\n",
       "      <td>post</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>how do you feel that you are sandwich makers and dishwashers</td>\n",
       "      <td>how do you feel that you are sandwich maker and dishwasher</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          text  \\\n",
       "0  Should students still be shown Neil DeGrasse Tyson's videos?Hi, feminists. I'm a middle school science teacher. Astronomy is part of my curriculum. \\n\\nOver the past 10 years, Neil DeGrasse Tyson seems to have devoted his life to appearing in every damn astronomy video made. Like, he really is in all of them! And, for the most part, the videos are good and can be valuable to kids' education.\\n\\nHe was accused this past week, by several different women, of various forms of sexual misconduct. The accusations range from creepiness to rape.\\n\\nI've tried my best to be supportive of the metoo movement and all that's come with it. \\n\\nSo there's my dilemma. Should I continue to show videos to my students that are hosted by Neil DeGrasse Tyson?   \n",
       "1  Does games like BFV and Call of Duty where there are female avatars promote/normalise violence towards women ?in the game women who get shot also cry in pain                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "2  What is Feminist political stance?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n",
       "3  What makes some victim blaming okay, and some not okay?For example if someone leaved their wallet on a bench, you'd victim blame them.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
       "4  how do you feel that you are sandwich makers and dishwashers?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "\n",
       "   type     subreddit  removed  deleted  \\\n",
       "0  post  AskFeminists  0        0         \n",
       "1  post  AskFeminists  0        0         \n",
       "2  post  AskFeminists  1        0         \n",
       "3  post  AskFeminists  0        0         \n",
       "4  post  AskFeminists  1        0         \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       clean_text_stop  \\\n",
       "0  should students still be shown neil degrasse tyson s videos hi  feminists  i m a middle school science teacher  astronomy is part of my curriculum    over the past    years  neil degrasse tyson seems to have devoted his life to appearing in every damn astronomy video made  like  he really is in all of them  and  for the most part  the videos are good and can be valuable to kids  education   he was accused this past week  by several different women  of various forms of sexual misconduct  the accusations range from creepiness to rape   i ve tried my best to be supportive of the metoo movement and all that s come with it    so there s my dilemma  should i continue to show videos to my students that are hosted by neil degrasse tyson    \n",
       "1  does games like bfv and call of duty where there are female avatars promote normalise violence towards women  in the game women who get shot also cry in pain                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
       "2  what is feminist political stance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \n",
       "3  what makes some victim blaming okay  and some not okay for example if someone leaved their wallet on a bench  you d victim blame them                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "4  how do you feel that you are sandwich makers and dishwashers                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                lems  \n",
       "0  should student still be shown neil degrasse tyson video hi feminist i a middle school science teacher astronomy is part of my curriculum over the past year neil degrasse tyson seems to have devoted his life to appearing in every damn astronomy video made like he really is in all of them and for the most part the video are good and can be valuable to kid education he wa accused this past week by several different woman of various form of sexual misconduct the accusation range from creepiness to rape i tried my best to be supportive of the metoo movement and all that come with it so there my dilemma should i continue to show video to my student that are hosted by neil degrasse tyson  \n",
       "1  doe game like bfv and call of duty where there are female avatar promote normalise violence towards woman in the game woman who get shot also cry in pain                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
       "2  what is feminist political stance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
       "3  what make some victim blaming okay and some not okay for example if someone leaved their wallet on a bench you victim blame them                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \n",
       "4  how do you feel that you are sandwich maker and dishwasher                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fem.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>type</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>removed</th>\n",
       "      <th>deleted</th>\n",
       "      <th>clean_text_stop</th>\n",
       "      <th>lems</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>E-Trade Implies Support for Female SupremacyFrom [https://us.etrade.com/knowledge/thematic-investing/gender-diversity](https://us.etrade.com/knowledge/thematic-investing/gender-diversity).   You may need an etrade account to see that page.  \\n\\nFrom that page, emphasis mine:\\n\\n&amp;gt;Looking to invest in ways that support the values of gender diversity and efforts to achieve it? The fund below invests in companies that are leaders in areas such as equitable hiring, equal pay, and the ***advancement of women into managerial, executive, and director positions***.\\n\\nScreenshot of the page:\\n\\n[https://imgur.com/a/etn0B3g](https://imgur.com/a/etn0B3g)\\n\\nThey are in my opinion making an argument for female supremacy by only emphasizing \"powerful\" jobs.  If they were advocating for equal opportunity in all jobs then that would be OK.  They aren't though, they are just saying, \"advance women into managerial, executive, and director positions.\"</td>\n",
       "      <td>post</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>e trade implies support for female supremacyfrom      you may need an etrade account to see that page     from that page  emphasis mine    gt looking to invest in ways that support the values of gender diversity and efforts to achieve it  the fund below invests in companies that are leaders in areas such as equitable hiring  equal pay  and the    advancement of women into managerial  executive  and director positions      screenshot of the page       they are in my opinion making an argument for female supremacy by only emphasizing  powerful  jobs   if they were advocating for equal opportunity in all jobs then that would be ok   they aren t though  they are just saying   advance women into managerial  executive  and director positions</td>\n",
       "      <td>e trade implies support for female supremacyfrom you may need an etrade account to see that page from that page emphasis mine gt looking to invest in way that support the value of gender diversity and effort to achieve it the fund below invests in company that are leader in area such a equitable hiring equal pay and the advancement of woman into managerial executive and director position screenshot of the page they are in my opinion making an argument for female supremacy by only emphasizing powerful job if they were advocating for equal opportunity in all job then that would be ok they aren though they are just saying advance woman into managerial executive and director position</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Public comment period now open for proposed Title IX regulations - PLEASE send in comments on the .gov website!</td>\n",
       "      <td>post</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>public comment period now open for proposed title ix regulations   please send in comments on the  gov website</td>\n",
       "      <td>public comment period now open for proposed title ix regulation please send in comment on the gov website</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Question regarding your opinions on the punishment for false rape accusationsWhat do you guys think would be a fair punishment for a woman falsely accusing a man of rape?</td>\n",
       "      <td>post</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>question regarding your opinions on the punishment for false rape accusationswhat do you guys think would be a fair punishment for a woman falsely accusing a man of rape</td>\n",
       "      <td>question regarding your opinion on the punishment for false rape accusationswhat do you guy think would be a fair punishment for a woman falsely accusing a man of rape</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>what really annoys me.I'm a female abuse victim and I've had woman get mad when I tell them that not all men are trash. I have met one of the trashiest men in the world (aka my abuser) and not for one second did I believe all men are trash. \\nby Brain can't even begin to think how you can find all men trash. \\n\\n</td>\n",
       "      <td>post</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>what really annoys me i m a female abuse victim and i ve had woman get mad when i tell them that not all men are trash  i have met one of the trashiest men in the world  aka my abuser  and not for one second did i believe all men are trash   by brain can t even begin to think how you can find all men trash</td>\n",
       "      <td>what really annoys me i a female abuse victim and i had woman get mad when i tell them that not all men are trash i have met one of the trashiest men in the world aka my abuser and not for one second did i believe all men are trash by brain can even begin to think how you can find all men trash</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>New Proposed Title IX Guidelines - Comment period now open</td>\n",
       "      <td>post</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>new proposed title ix guidelines   comment period now open</td>\n",
       "      <td>new proposed title ix guideline comment period now open</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     text  \\\n",
       "0  E-Trade Implies Support for Female SupremacyFrom [https://us.etrade.com/knowledge/thematic-investing/gender-diversity](https://us.etrade.com/knowledge/thematic-investing/gender-diversity).   You may need an etrade account to see that page.  \\n\\nFrom that page, emphasis mine:\\n\\n&gt;Looking to invest in ways that support the values of gender diversity and efforts to achieve it? The fund below invests in companies that are leaders in areas such as equitable hiring, equal pay, and the ***advancement of women into managerial, executive, and director positions***.\\n\\nScreenshot of the page:\\n\\n[https://imgur.com/a/etn0B3g](https://imgur.com/a/etn0B3g)\\n\\nThey are in my opinion making an argument for female supremacy by only emphasizing \"powerful\" jobs.  If they were advocating for equal opportunity in all jobs then that would be OK.  They aren't though, they are just saying, \"advance women into managerial, executive, and director positions.\"   \n",
       "1  Public comment period now open for proposed Title IX regulations - PLEASE send in comments on the .gov website!                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
       "2  Question regarding your opinions on the punishment for false rape accusationsWhat do you guys think would be a fair punishment for a woman falsely accusing a man of rape?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
       "3  what really annoys me.I'm a female abuse victim and I've had woman get mad when I tell them that not all men are trash. I have met one of the trashiest men in the world (aka my abuser) and not for one second did I believe all men are trash. \\nby Brain can't even begin to think how you can find all men trash. \\n\\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
       "4  New Proposed Title IX Guidelines - Comment period now open                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
       "\n",
       "   type   subreddit  removed  deleted  \\\n",
       "0  post  MensRights  0        0         \n",
       "1  post  MensRights  1        0         \n",
       "2  post  MensRights  0        0         \n",
       "3  post  MensRights  0        0         \n",
       "4  post  MensRights  1        0         \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               clean_text_stop  \\\n",
       "0  e trade implies support for female supremacyfrom      you may need an etrade account to see that page     from that page  emphasis mine    gt looking to invest in ways that support the values of gender diversity and efforts to achieve it  the fund below invests in companies that are leaders in areas such as equitable hiring  equal pay  and the    advancement of women into managerial  executive  and director positions      screenshot of the page       they are in my opinion making an argument for female supremacy by only emphasizing  powerful  jobs   if they were advocating for equal opportunity in all jobs then that would be ok   they aren t though  they are just saying   advance women into managerial  executive  and director positions     \n",
       "1  public comment period now open for proposed title ix regulations   please send in comments on the  gov website                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
       "2  question regarding your opinions on the punishment for false rape accusationswhat do you guys think would be a fair punishment for a woman falsely accusing a man of rape                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \n",
       "3  what really annoys me i m a female abuse victim and i ve had woman get mad when i tell them that not all men are trash  i have met one of the trashiest men in the world  aka my abuser  and not for one second did i believe all men are trash   by brain can t even begin to think how you can find all men trash                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "4  new proposed title ix guidelines   comment period now open                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               lems  \n",
       "0  e trade implies support for female supremacyfrom you may need an etrade account to see that page from that page emphasis mine gt looking to invest in way that support the value of gender diversity and effort to achieve it the fund below invests in company that are leader in area such a equitable hiring equal pay and the advancement of woman into managerial executive and director position screenshot of the page they are in my opinion making an argument for female supremacy by only emphasizing powerful job if they were advocating for equal opportunity in all job then that would be ok they aren though they are just saying advance woman into managerial executive and director position  \n",
       "1  public comment period now open for proposed title ix regulation please send in comment on the gov website                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
       "2  question regarding your opinion on the punishment for false rape accusationswhat do you guy think would be a fair punishment for a woman falsely accusing a man of rape                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "3  what really annoys me i a female abuse victim and i had woman get mad when i tell them that not all men are trash i have met one of the trashiest men in the world aka my abuser and not for one second did i believe all men are trash by brain can even begin to think how you can find all men trash                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "4  new proposed title ix guideline comment period now open                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "men.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>type</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>removed</th>\n",
       "      <th>deleted</th>\n",
       "      <th>clean_text_stop</th>\n",
       "      <th>lems</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10470</th>\n",
       "      <td></td>\n",
       "      <td>comment</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23597</th>\n",
       "      <td>😂😂</td>\n",
       "      <td>comment</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29057</th>\n",
       "      <td></td>\n",
       "      <td>comment</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16355</th>\n",
       "      <td></td>\n",
       "      <td>comment</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20359</th>\n",
       "      <td>https://www.health.harvard.edu/newsletter_article/marriage-and-mens-health</td>\n",
       "      <td>comment</td>\n",
       "      <td>MensRights</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                             text  \\\n",
       "10470                                                                               \n",
       "23597  😂😂                                                                           \n",
       "29057                                                                               \n",
       "16355                                                                               \n",
       "20359  https://www.health.harvard.edu/newsletter_article/marriage-and-mens-health   \n",
       "\n",
       "          type   subreddit  removed  deleted clean_text_stop lems  \n",
       "10470  comment  MensRights  1        0                             \n",
       "23597  comment  MensRights  0        0                             \n",
       "29057  comment  MensRights  0        1                             \n",
       "16355  comment  MensRights  0        1                             \n",
       "20359  comment  MensRights  0        0                             "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "men.sort_values(by='lems', ascending=True).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>type</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>removed</th>\n",
       "      <th>deleted</th>\n",
       "      <th>clean_text_stop</th>\n",
       "      <th>lems</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5098</th>\n",
       "      <td></td>\n",
       "      <td>comment</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29065</th>\n",
       "      <td></td>\n",
       "      <td>comment</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5366</th>\n",
       "      <td></td>\n",
       "      <td>comment</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26686</th>\n",
       "      <td></td>\n",
       "      <td>comment</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23315</th>\n",
       "      <td></td>\n",
       "      <td>comment</td>\n",
       "      <td>AskFeminists</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      text     type     subreddit  removed  deleted clean_text_stop lems\n",
       "5098        comment  AskFeminists  1        0                           \n",
       "29065       comment  AskFeminists  1        0                           \n",
       "5366        comment  AskFeminists  1        0                           \n",
       "26686       comment  AskFeminists  1        0                           \n",
       "23315       comment  AskFeminists  1        0                           "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fem.sort_values(by='lems', ascending=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Saving to csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "men.to_csv('./data/men_clean_lem')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "fem.to_csv('./data/fem_clean_lem')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}