hw2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2018 term\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This homework covers material from the sentiment classification unit. The primary value of doing the work is that it provides more hands-on experience with the dataset and the models we explored. All the code you write has potential value in the bake-off for this unit as well.\n",
    "\n",
    "Submission URL: https://canvas.stanford.edu/courses/83399/quizzes/50657"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions 1–4: Reproducing a Socher et al's NaiveBayes baseline [4 points]\n",
    "\n",
    "[Socher et al. 2013](http://www.aclweb.org/anthology/D/D13/D13-1170.pdf) compare against a Naive Bayes baseline with bigram features. See how close you can come to reproducing the performance of this model on the binary, root-only problem (values in the rightmost column of their Table 1, rows 1 and 3).\n",
    "\n",
    "Specific tasks:\n",
    "\n",
    "1. Write a bigrams feature function, on the model of `unigrams_phi`. Call this `bigrams_phi`. In writing this function, ensure that each example is padded with start and end symbols (say, `<S>` and `</S>`), so that these contexts are properly reflected in the feature space.\n",
    "\n",
    "1. Write a function `fit_nb_classifier_with_crossvalidation` that serves as a wrapper for `sklearn.naive_bayes.MultinomialNB` and searches over these values for the smoothing parameter `alpha`: `[0.1, 0.5, 1.0, 2.0]`, using 3-fold cross-validation.\n",
    "\n",
    "1. Use `sst.experiment` to run the experiments, assessing against `dev_reader`.\n",
    "\n",
    "__To submit:__\n",
    "\n",
    "1. Your `bigrams_phi`\n",
    "1. Your `fit_nb_classifier`\n",
    "1. Your call to `sst.experiment` \n",
    "1. The average F1 score that `sst.experiment` reported.\n",
    "\n",
    "__A note on performance__: in our experience, the bigrams Naive Bayes model gets around 0.75. It's fine to submit answers with comparable numbers; the Socher et al. baselines are very strong. We're not evaluating how good your model is; we want to see your code, and we're interested to see what the range of F1 scores is across the whole class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5–6: A more powerful vector-summing baseline [4 points]\n",
    "\n",
    "In the section [Distributed representations as features](sst_03_neural_networks.ipynb#Distributed-representations-as-features), we looked at a baseline for the binary SST problem in which each example is modeled as the sum of its 50-dimensional GloVe representations. A `LogisticRegression` model was used for prediction. A neural network might do better here, since there might be complex relationships between the input feature dimensions that a linear classifier can't learn. \n",
    "\n",
    "To address this question, rerun the experiment with `tf_shallow_neural_classifier.TfShallowNeuralClassifier` as the classifier. Specs:\n",
    "* Use `sst.experiment` to conduct the experiment. \n",
    "* Using 3-fold cross-validation, exhaustively explore this set of hyperparameter combinations:\n",
    "  * The hidden dimensionality at 50, 100, and 200.\n",
    "  * The hidden activation function as `tf.nn.tanh` or `tf.nn.relu`.\n",
    "* (For all other parameters to `TfShallowNeuralClassifier`, use the defaults.)\n",
    "\n",
    "__To submit:__\n",
    "\n",
    "* Your average F1 score according to `sst.experiment`. \n",
    "* The optimal hyperparameters chosen in your experiment. (You can just paste in the dict that `sst._experiment` prints.)\n",
    "\n",
    "No need to include your supporting code. \n",
    "\n",
    "We're not evaluating the quality of your model. (We've specified the protocols completely, but there will still be a  lot of variation in the results.) However, the primary goal of this question is to get you thinking more about this strikingly good baseline feature representation scheme for SST, so we're sort of hoping you feel compelled to try out variations on your own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions 7–8: Bidirectional RNN [2 points]\n",
    "\n",
    "The auxiliary notebook `tensorflow_models.ipynb` [subclasses TfRNNClassifier with a bidirectional RNN](tensorflow_models.ipynb#A-bidirectional-RNN-Classifier). In this model, the RNN is run in both directions and the concatenation of the two final states is used as the basis for the classification decision. Evaluate this model against the SST dev set. You can set up the model however you wish for this.\n",
    "\n",
    "__To submit:__\n",
    "\n",
    "* Your call to `TfBidirectionalRNNClassifier` (so that we can see the hyperparmeters you chose).\n",
    "* Your average F1 score according to a `classification_report` on the `dev` set.\n",
    "\n",
    "As above, we will not evaluate you based on how good your F1 score is. You just need to submit one. __There is even value in seeing what really doesn't work__, so low scores have interest!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}