examples/anchor_text_movie.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anchor explanations for movie sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we will explain why a certain sentence is classified by a logistic regression as having negative or positive sentiment. The logistic regression is trained on negative and positive movie reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "import spacy\n",
    "from alibi.explainers import AnchorText\n",
    "from alibi.datasets import movie_sentiment\n",
    "from alibi.utils.download import spacy_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load movie review dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, labels = movie_sentiment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define shuffled training, validation and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test, train_labels, test_labels = train_test_split(data, labels, test_size=.2, random_state=42)\n",
    "train, val, train_labels, val_labels = train_test_split(train, train_labels, test_size=.1, random_state=42)\n",
    "train_labels = np.array(train_labels)\n",
    "test_labels = np.array(test_labels)\n",
    "val_labels = np.array(val_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply CountVectorizer to training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(min_df=1)\n",
    "vectorizer.fit(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='warn',\n",
       "          n_jobs=None, penalty='l2', random_state=None, solver='liblinear',\n",
       "          tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "clf = LogisticRegression(solver='liblinear')\n",
    "clf.fit(vectorizer.transform(train), train_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define prediction function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_fn = lambda x: clf.predict(vectorizer.transform(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make predictions on train and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.9801624284382905\n",
      "Validation accuracy 0.7544910179640718\n",
      "Test accuracy 0.7589841878294202\n"
     ]
    }
   ],
   "source": [
    "preds_train = predict_fn(train)\n",
    "preds_val = predict_fn(val)\n",
    "preds_test = predict_fn(test)\n",
    "print('Train accuracy', accuracy_score(train_labels, preds_train))\n",
    "print('Validation accuracy', accuracy_score(val_labels, preds_val))\n",
    "print('Test accuracy', accuracy_score(test_labels, preds_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load spaCy model\n",
    "\n",
    "English multi-task CNN trained on OntoNotes, with GloVe vectors trained on Common Crawl. Assigns word vectors, context-specific token vectors, POS tags, dependency parse and named entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = 'en_core_web_md'\n",
    "spacy_model(model=model)\n",
    "nlp = spacy.load(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize anchor text explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "explainer = AnchorText(nlp, predict_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explain a prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_names = ['negative', 'positive']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction: positive\n"
     ]
    }
   ],
   "source": [
    "text = 'This is a good book .'\n",
    "pred = class_names[predict_fn([text])[0]]\n",
    "alternative =  class_names[1 - predict_fn([text])[0]]\n",
    "print('Prediction: %s' % pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explanation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "explanation = explainer.explain(text, threshold=0.95, use_proba=False, use_unk=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "use_unk=True means we will perturb examples by replacing words with UNKs. Let us now take a look at the anchor. The word 'good' basically guarantees a positive prediction. This is because the UNKs do not take instances like 'not good' into account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Anchor: good\n",
      "Precision: 1.00\n",
      "\n",
      "Examples where anchor applies and model predicts positive:\n",
      "UNK UNK UNK good book UNK\n",
      "UNK is a good book .\n",
      "UNK is a good book UNK\n",
      "UNK is UNK good book .\n",
      "UNK UNK UNK good book .\n",
      "UNK is a good book .\n",
      "UNK is UNK good UNK UNK\n",
      "UNK UNK UNK good UNK .\n",
      "This UNK a good UNK UNK\n",
      "This is a good UNK .\n",
      "\n",
      "Examples where anchor applies and model predicts negative:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('Anchor: %s' % (' AND '.join(explanation['names'])))\n",
    "print('Precision: %.2f' % explanation['precision'])\n",
    "print('\\nExamples where anchor applies and model predicts %s:' % pred)\n",
    "print('\\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_true']]))\n",
    "print('\\nExamples where anchor applies and model predicts %s:' % alternative)\n",
    "print('\\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_false']]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing the perturbation distribution\n",
    "Let's try this with another perturbation distribution, namely one that replaces words by similar words instead of UNKs.\n",
    "\n",
    "Explanation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "explanation = explainer.explain(text, threshold=0.95, use_proba=True, use_unk=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The anchor now shows that we need more to guarantee the positive prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Anchor: good AND book\n",
      "Precision: 0.95\n",
      "\n",
      "Examples where anchor applies and model predicts positive:\n",
      "Another includes both good book .\n",
      "Any explains that good book .\n",
      "SOME refers another good book .\n",
      "This makes an good book .\n",
      "SOME encapsulates every good book .\n",
      "That consists this good book .\n",
      "THE carries this good book .\n",
      "Both is another good book .\n",
      "Every sits another good book .\n",
      "BOTH leads the good book .\n",
      "\n",
      "Examples where anchor applies and model predicts negative:\n",
      "SOME falls another good book .\n",
      "THE feels another good book .\n",
      "This starts some good book .\n",
      "THis starts another good book .\n",
      "All requires a good book .\n",
      "Some goes a good book .\n",
      "This happens some good book .\n",
      "Some starts a good book .\n",
      "Both feels some good book .\n",
      "Both feels another good book .\n"
     ]
    }
   ],
   "source": [
    "print('Anchor: %s' % (' AND '.join(explanation['names'])))\n",
    "print('Precision: %.2f' % explanation['precision'])\n",
    "print('\\nExamples where anchor applies and model predicts %s:' % pred)\n",
    "print('\\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_true']]))\n",
    "print('\\nExamples where anchor applies and model predicts %s:' % alternative)\n",
    "print('\\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_false']]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}