hw1-ijh216.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://github.com/isaachaberman/dsga1011_hw1.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Readin \n",
    "\n",
    "Training data was shuffled and split into training and validation data-sets of 20,000 and 5,000 respectively.  Those functions can be seen in `hw_functions.py` and `tokenization.py`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ablation Study Study\n",
    "\n",
    "#### Tokenization Scheme\n",
    "<img src=\"pics/output_5_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_5_3.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_5_5.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "| Tokenization Scheme | Accuracy   |\n",
    "|------|------|\n",
    "|   Original  | 87.56|\n",
    "|   Stop Word Removal  | 87.86|\n",
    "|   Entities  | 87.06|\n",
    "\n",
    "I tried three tokenization schemes: \n",
    "1. A variant of the original scheme used in class.  I added an additional removal of html breaks.  \n",
    "2. Stop Word Removal.  Following the previous scheme, I added a removal of stop words and apostrophes.\n",
    "3. Entity Keeping.  Using Spacy's entities, I kept the tokens that had proper entities and removed the others.\n",
    "\n",
    "Of the three, the second tokenization scheme performed the best, with a maximum accuracy of 87.86%, marginally better than the other maximum accuracies of 87.56% and 87.06%.  I also note, that the first and third tokenization schemes have worse overfitting than the second scheme, with greater dims in accuracy towards the end of training.  I suspect that the the stop word removal performed the best, as the stop words cloud the bag of words by being among the most common words and removing them allows for easier sentiment analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### N-grams\n",
    "\n",
    "<img src=\"pics/output_9_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_9_3.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "<img src=\"pics/output_9_5.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_9_7.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "| N-grams | Accuracy   |\n",
    "|------|------|\n",
    "|   Unigrams  | 87.98|\n",
    "|   Bigrams  | 50.06|\n",
    "|   Trigrams  | 50.06|\n",
    "|Tetragrams | 50.06|\n",
    "\n",
    "I tried four n-grams sizes, unigrams, bigrams, trigrams and tetragrams.  For each n-gram scheme, I tested each set of n-grams by themselves.  Of those unigrams performed remarkably better than any of the other schemes with a validation accuracy of 87.98% compared to other accuracies of 50%.  I ssupect much of the results may lie in a faulty implementation or the need for the larger n-grams to be used in conjunction with unigrams, a configuration I did not test.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Vocabulary Size\n",
    "\n",
    "<img src=\"pics/output_12_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_12_3.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_12_5.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "<img src=\"pics/output_12_7.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_12_9.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_12_11.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "<img src=\"pics/output_12_13.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "| Vocabulary Size | Accuracy   |\n",
    "|------|------|\n",
    "|   10k  | 88.16|\n",
    "|   20k  | 88.62|\n",
    "|   30k  | 88.5|\n",
    "|   40k  | 88.44|\n",
    "|   50k  | 88.04|\n",
    "|   100k | 88.94|\n",
    "|   200k | 88.34|\n",
    "\n",
    "I varied vocabulary size from 10,000 to 200,000k, with all vocabulary sizes having similar maximum validation accuracies and similar levels of overfitting.  That being said, I chose a vocabulary size of 100,000 as it was the best performing.  With the removal of stop words and the use of unigrams, it may be that the vocabulary size is no longer an important hyperparameter and any nmber of words can be used in the Bag of Words model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Embedding Size\n",
    "\n",
    "<img src=\"pics/output_16_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_16_3.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "<img src=\"pics/output_16_5.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_16_7.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "| Embedding Size | Accuracy   |\n",
    "|------|------|\n",
    "|   50  | 88.2|\n",
    "|   100  | 88.3|\n",
    "|   150  | 88.9|\n",
    "|   200  | 88.92|\n",
    "\n",
    "Using the aforementioned hyperparameters, I tested embedding sizes from 50 - 200.  All four embeddings performed similarly, with maximum validation accuracies around 88% and similar levels of overfitting.  That being said, the best performing embedding size was 200 with a maximum validation accuracy of 88.92%.  Similarly to the vocabulary size, its possible the importance of embedding size is lessened by the prior hyperparameter choices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Learning Rate\n",
    "\n",
    "<img src=\"pics/output_20_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_20_3.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "<img src=\"pics/output_20_5.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_20_7.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "<img src=\"pics/output_20_9.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "| Learning Rate | Accuracy   |\n",
    "|------|------|\n",
    "|   0.0001  | 80.42|\n",
    "|   0.001  | 89.2|\n",
    "|   0.01  | 88.62|\n",
    "|   0.1  | 86.7|\n",
    "|   1  | 86.14|\n",
    "\n",
    "Unlike the previous two hyperparameters, there were vast differences between the maximum validation accuracies among the learning rates.  The best learning rate occured at 0.001, one of the smaller learning rates I tested.  From the results it is clear that choosing too small a learning rate does not produce satisfactory results and the data appears underfit.  And for learning rates that are too large, the model moves to quickly, and there appear to be poor moves as there is a large dip in accuracy for both the training and validation accuracy for the biggest learning rates.  Since learning rates can be altered depending on the epoch, I also tested linear annealing of learning rates.  The results can be seen below.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Annealing Learning Rate\n",
    "\n",
    "<img src=\"pics/output_23_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_23_3.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "<img src=\"pics/output_23_5.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_23_7.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "<img src=\"pics/output_23_9.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "| Annealing Learning Rate | Accuracy   |\n",
    "|------|------|\n",
    "|   0.0001  | 52.66|\n",
    "|   0.001  | 80.62|\n",
    "|   0.01  | 88.26|\n",
    "|   0.1  | 88.04|\n",
    "|   1  | 86.98|\n",
    "\n",
    "For the linear annealed learning rate, the best results were quite similar to the best results of the non-annealed learning rate, with small initial values not improving and the largest initial values slightly overfitting.  Keeping gamma constant, the best performing annealing learning rate was 0.01 with a maximum validation accuracy of 88.26%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gamma\n",
    "<img src=\"pics/output_27_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_27_3.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "<img src=\"pics/output_27_5.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_27_7.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "| Learning Rate-Gamma | Accuracy   |\n",
    "|------|------|\n",
    "|   0.001  | 88.74|\n",
    "|   0.01  | 88.18|\n",
    "|   0.1  | 88.9|\n",
    "|   1  | 88.24|\n",
    "\n",
    "For the other annealing learning rate hyperparameter, the best validation accuracy produced was very similar to the non-annealed learning rates.  For a gamma of 0.1, the validation accuracy was 88.9%.  Since the optimized annealed learning rate had similar validation accuracy to the non-annealed learning rate, I chose to use the annealed learning rate to help prevent overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimizer\n",
    "\n",
    "<img src=\"pics/output_32_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_32_3.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "\n",
    "| Optimizer | Accuracy   |\n",
    "|------|------|\n",
    "|   Adam  | 88.48|\n",
    "|   SGD  | 61.18|\n",
    "\n",
    "The Adam optimizer produced remarkably better results than the SGD optimizer.  Therefore, I used the Adam optimizer for the final model.  Since the earlier hyperparameter choices had been optimized to Adam, I suspect that the poor results of SGD may stem from the earlier tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weight Decay\n",
    "\n",
    "<img src=\"pics/output_36_1.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_36_3.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "<img src=\"pics/output_36_5.png\" alt=\"drawing\" width=\"200\"/> <img src=\"pics/output_36_7.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "<img src=\"pics/output_36_9.png\" alt=\"drawing\" width=\"200\"/> \n",
    "\n",
    "\n",
    "| Weight Decay | Accuracy   |\n",
    "|------|------|\n",
    "|   0.0001  | 87.3|\n",
    "|   0.001  | 83.34|\n",
    "|   0.01  | 54.52|\n",
    "|   0.1  | 49.94|\n",
    "|   1  | 50.06|\n",
    "\n",
    "Lastly, I did a hyperparameter search over the weight decay to help prevent overfitting in the final model.  Of the weight decay options, 0.0001 produced the best validation accuracy with a value of 87.3%.  Given too large a weight decay, the model performs rather poorly as the constraint of the weight decay destroys performance gains, while the smaller weight decays produce better results since they restrict the model less.  Since there does not appear to be much overfitting in the final results, the validation accuracy not dipping, the smallest weight decay was an appropriate choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Final Model\n",
    "Using the best performing hyperparameters, the model produces a final test accuracy of 86.7% compared to a test accuracy of 82.61% using the non-ablated hyperparameters.  In other words, by using a doing an abalation study, we were able to successfully classify an additional 1000 reviews.\n",
    "\n",
    "I suspect with some additional tuning we may be able to improve the test accuracy.  I did not test the rate in which the linear annealing occurs as a function of the step size, nor did I test the use of unigrams and bigrams together, two other hyperparameter choices that may have improves results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Misclassified Examples\n",
    "\n",
    "'two things changed first kid anymore second new seagal movies terrible this opinion worst movie derailed starred jean claude van damme there plot movie plot excuse shoot terrible action scenes painful watch i love action movies action movie this nt movie group irritating scenes connected annoying way kill viewers love action movies i sorry nt respect liked movie he intelligence problem i hope seagal new good movies future good luck' - While the tokenization removes some of the readability, the reviews can still be analyzed.  For instance, this first misclassification might have been caused by the repeated use of \"love\" a term associated with positive sentiment but here is used as background.  The author loves action films, yet does not like this film.\n",
    "\n",
    "'after watch movie i surprised like i think worst comedy i seen if think seen worst comedy wait watch crap not funny ok laugh maybe high acting terrible story nt exist even like i hate movie ok ok nice babes nice tricks beautiful mountains s watch porno set mountains instead acting story line better sometimes imdb i read bad review movie end awful believe pure trash do nt waste time money' - This review refers to \"favourite movies\" (it might be worth standardizing spellings of words in a different tokenization scheme), \"positive\", \"brightest\" and other positive words, which in conjunction with other words would produce negative sentiment, but as seen by themselves produce positive sentiment.\n",
    "\n",
    "'after watch movie i surprised like i think worst comedy i seen if think seen worst comedy wait watch crap not funny ok laugh maybe high acting terrible story nt exist even like i hate movie ok ok nice babes nice tricks beautiful mountains s watch porno set mountains instead acting story line better sometimes imdb i read bad review movie end awful believe pure trash do nt waste time money' - The reviewer spends much of the review trashing the movie, yet starts the review with \"i surprised like,\" i.e. the reviewer enjoyed the film even though there were negative elements to it.  Therefore, the review was classified negatively even though it was a positive review.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classified Examples\n",
    "\n",
    "'after astronaut dies space brought military base inside man discovered alien embryos host terrible alien invasion this film comes director bernard l. kowalski directed attack giant leeches separate review better known work columbo executive producer roger corman known creator better films particularly 1960s this movie cheesy poorly constructed what comes interesting poor effects actual film one scene shows close alien embryos embarrassing cartoon representation even 1958 and grown alien appears ll wonder wearing shoes or perceptive ll wonder ve seen alien suit movies by means worst science fiction film ll and credit alien host overtaking military base idea predates the thing alien number years i nt know films inspired way i doubt ahead time beyond film flops great heckling drinking i nt seen mystery science theater version sure film worthy insults' - There is clear negative sentiment in this review with words like \"flops\", \"embarassing\", \"worst\", and \"insults.\"\n",
    "\n",
    "'minor spoilersin chicago grace beasley kathy bates housewife having years marriage lawyer max beasley dan aykroyd hysterical psychotic dwarf daughter law meredith eaton grace worships singer victor fox jonathan price present tv chicago spots row tv promotion kate calls wins ticket max simultaneously asks divorce claiming lives monotonous grace depressed goes audience informed chicago serial killer uses crossbow killed victor fox with broken heart decides fly england victor foxs funeral there realizes gay friend mate dirk simpson rupert everett they fly chicago trying find killer this movie delightful original weird dramatic comedy having bizarre characters it huge potential cult movie presences julie andrews barry joke nicolas roegs masterpiece do nt look now wears red raincoat chicagos underground city the beginning movie jonathan price singing hitchcock railway wonderful i repeated consecutive times the cast magnificent performance highlighting kathy bates jonathan price rupert everett unknown meredith eaton indeed movie excellent entertainment my vote title brazil amor love proof' - The reviewer refers to this film as \"delightful\" a positively charged word.  Since much of the review is a summary of the film, there is littler other sentiment.\n",
    "\n",
    "'this movie potential good movie eyes nicholas sparks great romance author movie chance great the notebook s sets apart notebook dream team leads mcadams gosling balance thrown miserably inept acting channing felt lot scenes uneven purely performance lot emotion scenes lost nt act leaving awkward uneven situation amanda given great performance tatum drop ball mood lost scene nt recover this story deserved cast right got pretty boy nt act tatum stick s good movies physical ability albeit horrible like gi joe step fighting the talks better try think jaded hater channing tatum i went movie open mind i ve surprised time likes adam sandler reign me i gave chance tatum i nt view sum past roles purely performance movie sadly letdown' - While the reviewer has qualms with the film, they enjoy the lead actressses performance and Nicholas Sparks novels and therefore the positive sentiment outweighs the negative."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}