DP_autoFAQ.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B7fbDvssiBrV"
   },
   "source": [
    "# Simple intent recognition and question answering with DeepPavlov\n",
    "\n",
    "This notebook consists of the [DeepPavlov](https://github.com/deepmipt/DeepPavlov) code snippets. The snippets show how to interact with the text classification models that were specifically developed to be effective when training data is limited. The popular use case scenario for these models is to classify user utterances into one of the FAQ questions and retrieve the corresponding answer (autoFAQ models). As a testbed, we use the students’ FAQ from the [MIPT website](https://mipt.ru/english/). The FAQ contains the most popular first-year students' questions with corresponding answers.\n",
    "The framework allows you to train models, fine-tune hyperparameters, and test models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x_hoGWCfhxLh"
   },
   "source": [
    "# Requirements\n",
    "\n",
    "First, install all required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 683
    },
    "colab_type": "code",
    "id": "qhvF6Petcd0r",
    "outputId": "ec2f36c8-adc2-415d-cc1f-24031bc8d9ed"
   },
   "outputs": [],
   "source": [
    "!pip install -q deeppavlov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EVWji-oNOl2I"
   },
   "source": [
    "# Model Description\n",
    "\n",
    "DeepPavlov contains several text classification models that work well on few training pairs. All the models are based on two major text representations: fastText word embeddings and tf-idf representation. The models described in the separated configuration files under the [config/faq](https://github.com/deepmipt/DeepPavlov/tree/master/deeppavlov/configs/faq) folder. The config file consists of four main sections: **dataset_reader**, **dataset_iterator**, **chainer**, and **train**.\n",
    "\n",
    "The **dataset_iterator** specifies how to split the data into train, valid, test sets. The **chainer** section of the configuration files contains a pipeline of the required components to interact with the models, i.e. tokenizer, lemmatizer, tf-idf vectorizer, and others. The tokenizer splits a string into tokens, lemmatizer converts all tokens into lemmas. The tf-idf vectorizer transforms the lemmas into tf-idf vectors. The component’s input and output are defined in the **in** and **out** keys correspondingly.\n",
    "\n",
    "Run the following cell to see a [configuration file](https://github.com/deepmipt/DeepPavlov/blob/master/deeppavlov/configs/faq/tfidf_logreg_en_faq.json) based on logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load https://raw.githubusercontent.com/deepmipt/DeepPavlov/master/deeppavlov/configs/faq/tfidf_logreg_en_faq.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u1hsQECYf5yX"
   },
   "source": [
    "# Interacting with the model\n",
    "\n",
    "The DeepPavlov framework contains several models pre-trained on the aforementioned MIPT FAQ corpus. The files with the pre-trained models defined in the **metadata: download** section of the model's configuration file. You can interact with the model by running it from the command line with **interact** parameter and the name of  the model's configuration file (-d indicates to download all required files). But first, **install** all the model requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 2111
    },
    "colab_type": "code",
    "id": "UydjsTwLfryF",
    "outputId": "17386d79-1c3b-4739-b502-67e2432a0b20"
   },
   "outputs": [],
   "source": [
    "!python -m deeppavlov install tfidf_logreg_en_faq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 615
    },
    "colab_type": "code",
    "id": "H_YQYuj2_Xpx",
    "outputId": "678c140a-74fb-4187-b70c-4b441c2c84c2"
   },
   "outputs": [],
   "source": [
    "!python -m deeppavlov interact tfidf_logreg_en_faq -d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "48TD-v-FO25O"
   },
   "source": [
    "Alternatively, you can **build_model** from the Python code as on the example below. In addition, please make sure that you can navigate the configuration files by using Autocomplete (Tab key) with **configs** module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "code",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224
    },
    "colab_type": "code",
    "id": "wMZqyzBYc2eV",
    "outputId": "14910e68-f733-47bf-91d1-1c7646f79e6b"
   },
   "outputs": [],
   "source": [
    "from deeppavlov import configs\n",
    "from deeppavlov.core.common.file import read_json\n",
    "from deeppavlov.core.commands.infer import build_model\n",
    "\n",
    "faq = build_model(configs.faq.tfidf_logreg_en_faq, download = True)\n",
    "a = faq([\"I need help\"])\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TJ_Uqj7K52En"
   },
   "source": [
    "# Training the model\n",
    "\n",
    "You can train a model by running the library with **train** parameter, wherein the model will be trained on the dataset defined in the dataset_reader section of the configuration file. If **metrics** key along with either **validate_best** or **test_best** is defined in the train section, the model will be validated/tested on the corresponding set in the dataset_iterator section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xfOBVt_U5zE4"
   },
   "outputs": [],
   "source": [
    "!python -m deeppavlov train tfidf_logreg_en_faq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "93GqENh-PAHc"
   },
   "source": [
    "Let's modify the training data and retrain the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FEXzImAIPGEt"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "wget -q http://files.deeppavlov.ai/faq/school/faq_school_en.csv -O faq.csv\n",
    "echo \"What's DeepPavlov?, DeepPavlov is an open-source conversational AI library\" >> faq.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aN-Yb8Vd4oD6"
   },
   "outputs": [],
   "source": [
    "from deeppavlov import configs, train_model\n",
    "\n",
    "model_config = read_json(configs.faq.tfidf_logreg_en_faq)\n",
    "model_config[\"dataset_reader\"][\"data_path\"] = \"/content/faq.csv\"\n",
    "model_config[\"dataset_reader\"][\"data_url\"] = None\n",
    "faq = train_model(model_config)\n",
    "a = faq([\"tell me about DeepPavlov\"])\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1s2RLyyYUogN"
   },
   "source": [
    "# About Us\n",
    "\n",
    "We are iPavlov, our story started in 2017 when we decided to build a conversational AI framework that on the one hand will contain all required NLP components to build chatbots and on the other hand will be easy to use. Our work resulted in releasing DeepPavlov library. Our lab at MIPT is honored with Facebook AI Academic Partnership and NVIDIA GPU Research Center status. We successfully combine research and extreme coding in our week-long DeepHack.me hackathons — DeepHack.Game, DeepHack.Q&A and DeepHack.RL. We serve a global AI community by organizing NIPS Conversational Challenge to evaluate state-of-the-art techniques in the field of dialog systems and collect open source dialog datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yom6JD53QcXk"
   },
   "source": [
    "# Useful links\n",
    "\n",
    "[DeepPavlov repository](https://github.com/deepmipt/DeepPavlov)\n",
    "\n",
    "[DeepPavlov demo page](https://demo.ipavlov.ai)\n",
    "\n",
    "[DeepPavlov documentation](https://docs.deeppavlov.ai)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "DP:autoFAQ.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}