howtouse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Walktrough example : Train Document Emebeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a simple example on how it is possible to use the scripts in this repository to train document embeddings on a given dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First the usaul import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import logging\n",
    "from configparser import ConfigParser\n",
    "from load import DatasetGenerator,ModelsLoader,load_and_process\n",
    "from preprocess import GensimPreprocessor,DocumentsTagged\n",
    "from train_models import ModelsTrainer\n",
    "from quality_check import QualityChecker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "If you wish to use it with a configuration file please take a look at one in the folder `/configs`. In this walkthrough I will use `/configs/train_newspapers_standard.config` (train document embeddings for newspaper classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to use the configuraiton file you can use the python built-in *configparser* in this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./configs/train_newspapers_standard.config']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "configuration = ConfigParser(allow_no_value=False)\n",
    "configuration.read('./configs/train_newspapers_standard.config')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will simply load a special dictionary with all the parameters you can set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LOADING', 'PREPROCESS', 'PARAMETERS', 'TRAIN', 'LABELS']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "configuration.sections()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LOAD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load = es\n",
      "json_path = \n",
      "es_server = localhost\n",
      "es_indices = new_guardian,new_independent,new_telegraph,new_dailymail\n",
      "es_query = {\"query\": {\"bool\" : { \"must\" : [{\"range\": {\"publication_date\": {\"gte\": \"2016-01-01\",\"lte\": \"2017-10-31\"}}},{\"query_string\": {\"fields\" : [\"body\"], \"query\" : \"brexit OR referendum\"}}] }}}\n",
      "doc_ids = url,url,url,url\n",
      "doc_texts = body,body,body,body\n",
      "constuct_new_models = True\n"
     ]
    }
   ],
   "source": [
    "for key,value in configuration['LOADING'].items(): print(key,value, sep = ' = ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DatasetGenerator` will take these parameters and load data accordingly. `load` is the source of the data. Right now you can either load data from a ES instance or from JSON files. If you want to use ES you need to specify: the indices you want to retrieve from, a general query that will be used for all the searched operation, the fields that contains a unique identifier for the document and the one which stores the actual data (ORDER MATTERS). If you load data from JSON files instead you need to specify the path where the file is stored. In this case the script assumes that __doc_ids and *doc_bodys  are the same for all JSONs!__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = DatasetGenerator.load_data_from_config(configuration['LOADING'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case you wish to load data independently from a configuration file you can use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = DatasetGenerator.load_elasticsearch(es_server = ['localhost'], es_indices = ['new_guardian'],\n",
    "                                           es_query = {\"query\" : {\"match_all\" : {}}, \"sort\" : [\"_doc\"]},\n",
    "                                           doc_ids = ['url'],\n",
    "                                           doc_texts = ['body'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case I will you some data I have stored locally in a JSON file (since I did not find a way to connect this notebook to the server)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-01-23 16:36:58,459 : INFO : load: Loading data from `/home/ele/Scrivania/HPI/data/websci/samuele.garda/politics/2017.json`\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'author': ['https://www.facebook.com/bbcnews'], 'publication_date': '27 September 2017', 'news_keywords': '---', 'tags': ['http://www.bbc.co.uk/news/uk-politics-41407356'], 'words': 'Liam Fox, Boris Johnson and Priti Patel argued open markets are the best vehicle for reducing poverty and aiding prosperity at an event in London. Free of the \"constraints\" of the EU, the UK must be an \"agitator\" for free trade, the foreign secretary said.Meanwhile an ex-Tory leader has warned the UK must prepare for no Brexit deal.Critics say failure to do a Brexit deal could result in new trade barriers but Iain Duncan Smith said the EU must agree to open trade discussions by December or the UK should make arrangements to leave without a deal. Boeing warned over UK government contractsAccusing the EU of \"arrogant behaviour... bordering on the deliberately offensive\", he said the UK must \"throw resources\" at a no deal scenario, arguing the UK\\'s reach in terms of trade was second to none. As talks on the terms of the UK\\'s withdrawal from the EU continue in Brussels, European Council president Donald Tusk said on Tuesday that he believed not enough progress has been made to move to the next phase of negotiations, including trade and the UK\\'s future relationship with the bloc.Ministers have said discussions on future UK-EU relations should begin as soon as possible and there are \"no excuses\" for the current logjam to continue after Prime Minister Theresa May gave assurances on financial contributions and the rights of EU citizens in the UK in her Florence speech last week.Speaking on Tuesday during a visit to Slovakia, Mr Johnson said \"it was time to talk about the future\".At the launch of a new think tank on Wednesday, Mr Johnson, alongside International Trade Secretary Liam Fox and International Development Secretary Priti Patel, stressed the importance of the UK seizing opportunities to forge independent trading arrangements with growing economies around the world.\"Free trade is not only the key to economic success, but also serves as a force for peace and progress in every sense, giving millions more people the chance to lift themselves out of poverty,\" the foreign secretary said..\"We must ensure that Global Britain breaks free of the constraints of the EU and becomes the world\\'s leading proselytiser and agitator for free trade.\"His comments come amid a growing transatlantic trade dispute following the US government\\'s threat to impose a 220% tariff on imports of the C-Series jet, parts of which are made in Northern Ireland by Canadian firm Bombardier. Defence Secretary Sir Michael Fallon warned the dispute \"could jeopardise\" the UK\\'s defence contracts with US firm Boeing, a rival to Bombardier. ', 'taxonomy': 'uk politics', 'article_tag': '---', 'premium': False, 'category': 'politics', 'keywords': '---', 'list_of_tags': '---', 'title': \"UK must make 'moral case' for free trade outside EU - BBC NewsBBC News\", 'subtitle': 'Ministers argue open markets are the best way to cut global poverty and boost prosperity after Brexit.', 'opinion': False}\n"
     ]
    }
   ],
   "source": [
    "data = DatasetGenerator.load_json(folder = '/home/ele/Scrivania/HPI/data/websci/samuele.garda/politics',\n",
    "                                 doc_id = 'url',\n",
    "                                 doc_text = 'body')\n",
    "print(data.__next__())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have noticed that the fields `url` and `body` have been converted to `tags` and `words`. This is for compatibility reason with `gensim`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's load some model as wll. If there are no models to train you can use the `construct_new_models = True`. This will create as many model as the possible configurations in the `PARAMETERS` section (please refer to `gensim` documentation for more details). On the other hand, if you have already trained some model and you want to fine tune them you can load all the models froma a directory or a single one. Again you can use the specific functions if you do not want to stick with configuration files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "single_model = ModelsLoader.load_single_model('./models/Doc2Vec(dbow,d100,n20,mc5,t32)')\n",
    "models = ModelsLoader.load_from_dir('models')\n",
    "new_models = ModelsLoader.construct_models(parameters_conf= {'size': [100, 200], 'min_count': [5, 10]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the beloved configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-01-23 16:37:01,215 : INFO : load: Loaded : 10 models\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<gensim.models.doc2vec.Doc2Vec object at 0x7fd00be706d8>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70390>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be702e8>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70630>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70588>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70780>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be701d0>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be704e0>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70438>, <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70828>]\n"
     ]
    }
   ],
   "source": [
    "models_to_train = ModelsLoader.load_models_from_config(configuration['LOADING'],configuration['PARAMETERS'])\n",
    "print(models_to_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PREPROCESS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can move to the preprocessing part. For now the only options allowed are: removing stopword (a predifined set provided by `gensim`) and/or stemming. The preprocessor is the only part bounded to a configuration file. This is because when using the trained models it is extremely important that the new unseed input is processed in the same way as the training examples. If you do not want to work with configuration file you will have to come up with your own preprocessing pipeline. You simpy need a function that works on the `words` key of the dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm_stopwords = True\n",
      "stem = True\n"
     ]
    }
   ],
   "source": [
    "for key,value in configuration['PREPROCESS'].items(): print(key,value, sep = ' = ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".{'author': ['https://www.facebook.com/bbcnews'], 'publication_date': '27 September 2017', 'news_keywords': '---', 'tags': ['http://www.bbc.co.uk/news/uk-scotland-scotland-politics-41413587'], 'words': ['Ms', 'Dugdale', 'told', 'the', 'BBC', 'there', 'had', 'been', 'a', 'lot', 'of', 'internal', 'problems', 'in', 'the', 'party', 'ahead', 'of', 'her', 'sudden', 'resignation', 'Her', 'allies', 'have', 'claimed', 'there', 'was', 'a', 'plot', 'against', 'her', 'after', 'interim', 'leader', 'Alex', 'Rowley', 'was', 'caught', 'on', 'tape', 'backing', 'leadership', 'candidate', 'Richard', 'Leonard', 'The', 'row', 'unfolded', 'as', 'Jeremy', 'Corbyn', 'spoke', 'at', 'the', 'Labour', 'conference', 'in', 'Brighton', 'During', 'his', 'speech', 'Mr', 'Corbyn', 'said', 'that', 'Labour', 'was', 'on', 'the', 'way', 'back', 'in', 'Scotland', 'and', 'thanked', 'Ms', 'Dugdale', 'for', 'her', 'work', 'The', 'dispute', 'which', 'has', 'involved', 'several', 'of', 'the', 'party', 's', 'most', 'prominent', 'MSPs', 'has', 'been', 'thrown', 'into', 'the', 'spotlight', 'by', 'the', 'contest', 'between', 'Mr', 'Leonard', 'and', 'Anas', 'Sarwar', 'to', 'succeed', 'Ms', 'Dugdale', 'as', 'Scottish', 'Labour', 'leader', 'Internal', 'problems', 'Ms', 'Dugdale', 'quit', 'in', 'August', 'with', 'her', 'deputy', 'Mr', 'Rowley', 'subsequently', 'revealing', 'that', 'he', 'was', 'only', 'given', 'ten', 'minutes', 'notice', 'of', 'the', 'move', 'Speaking', 'on', 'BBC', 'Radio', 'Five', 'Live', 'Ms', 'Dugdale', 'confirmed', 'this', 'was', 'true', 'and', 'added', 'I', 'm', 'sure', 'lots', 'of', 'people', 'will', 'be', 'thinking', 'wow', 'that', 'speaks', 'to', 'a', 'lot', 'of', 'internal', 'problems', 'in', 'the', 'Labour', 'party', 'They', 'd', 'be', 'right', 'She', 'also', 'said', 'she', 'had', 'quit', 'as', 'leader', 'because', 'it', 'frees', 'me', 'up', 'to', 'talk', 'about', 'things', 'I', 'm', 'passionate', 'about', 'such', 'as', 'Brexit', 'noting', 'that', 'I', 'wasn', 't', 'able', 'to', 'do', 'that', 'as', 'leader', 'of', 'the', 'Scottish', 'Labour', 'Party', 'because', 'at', 'that', 'point', 'my', 'job', 'was', 'to', 'hold', 'everybody', 'together', 'And', 'she', 'denied', 'she', 'had', 'been', 'pushed', 'out', 'of', 'the', 'role', 'by', 'opponents', 'saying', 'I', 'wasn', 't', 'bullied', 'out', 'of', 'my', 'position', 'I', 'did', 'this', 'very', 'much', 'on', 'my', 'own', 'terms', 'However', 'Mr', 'Rowley', 'has', 'faced', 'accusations', 'of', 'being', 'involved', 'in', 'a', 'plot', 'to', 'replace', 'Ms', 'Dugdale', 'with', 'Mr', 'Leonard', 'who', 'like', 'Mr', 'Rowley', 'is', 'on', 'the', 'left', 'of', 'the', 'party', 'The', 'interim', 'leader', 'vowed', 'not', 'to', 'endorse', 'either', 'candidate', 'in', 'the', 'leadership', 'race', 'but', 'was', 'caught', 'on', 'tape', 'prior', 'to', 'a', 'conference', 'fringe', 'event', 'saying', 'that', 'Mr', 'Leonard', 'was', 'the', 'best', 'candidate', 'Mr', 'Rowley', 'was', 'unaware', 'he', 'was', 'being', 'recorded', 'but', 'the', 'Scottish', 'Sun', 'quoted', 'him', 'as', 'saying', 'Although', 'I', 'm', 'neutral', 'in', 'the', 'leadership', 'I', 'also', 'believe', 'that', 'Richard', 'Leonard', 'has', 'everything', 'that', 'we', 'need', 'to', 'win', 'in', '2021', 'I', 'really', 'do', 'So', 'when', 'to', 'our', 'surprise', 'the', 'job', 'became', 'vacant', 'it', 'just', 'seemed', 'to', 'me', 'that', 'from', 'a', 'left', 'perspective', 'the', 'person', 'that', 'was', 'most', 'suited', 'to', 'do', 'that', 'job', 'would', 'be', 'Richard', 'Leonard', 'Complete', 'betrayal', 'Mr', 'Rowley', 'whose', 'daughter', 'Danielle', 'a', 'Labour', 'MP', 'is', 'running', 'Mr', 'Leonard', 's', 'campaign', 'also', 'said', 'party', 'figures', 'privately', 'didn', 't', 'believe', 'Kezia', 'would', 'be', 'there', 'for', 'the', 'next', 'Holyrood', 'election', 'in', '2021', 'saying', 'that', 'our', 'view', 'was', 'that', 'Richard', 'was', 'the', 'best', 'person', 'and', 'therefore', 'we', 'should', 'go', 'with', 'that', 'plan', 'He', 'also', 'said', 'he', 'had', 'been', 'a', 'backer', 'of', 'Mr', 'Leonard', 'for', 'some', 'time', 'now', 'but', 'stressed', 'that', 'we', 'certainly', 'weren', 't', 'putting', 'any', 'pressure', 'on', 'it', 'His', 'comments', 'were', 'described', 'as', 'hypocrisy', 'and', 'incredibly', 'disappointing', 'by', 'fellow', 'Labour', 'MSP', 'Jackie', 'Baillie', 'a', 'backer', 'of', 'Mr', 'Sarwar', 'who', 'is', 'seen', 'as', 'being', 'the', 'centrist', 'candidate', 'and', 'has', 'been', 'a', 'critic', 'of', 'Jeremy', 'Corbyn', 'in', 'the', 'past', 'Ms', 'Baillie', 'said', 'there', 'was', 'evidence', 'of', 'a', 'plot', 'going', 'on', 'behind', 'the', 'scenes', 'for', 'months', 'against', 'Ms', 'Dugdale', 's', 'leadership', 'calling', 'this', 'a', 'complete', 'betrayal', 'of', 'the', 'membership', 'and', 'every', 'value', 'we', 'hold', 'dear', 'She', 'later', 'told', 'BBC', 'Scotland', 'Richard', 'Leonard', 'may', 'deny', 'this', 'but', 'there', 'are', 'people', 'within', 'his', 'campaign', 'team', 'who', 'have', 'clearly', 'been', 'involved', 'I', 'think', 'Richard', 'would', 'want', 'to', 'distance', 'himself', 'from', 'them', 'This', 'was', 'echoed', 'by', 'another', 'Sarwar', 'backing', 'MSP', 'Daniel', 'Johnson', 'who', 'said', 'that', 'the', 'admission', 'of', 'a', 'plot', 'is', 'shocking', 'and', 'unacceptable', 'However', 'Ms', 'Baillie', 's', 'comments', 'were', 'in', 'turn', 'decried', 'by', 'Leonard', 'supporting', 'MSP', 'Neil', 'Findlay', 'as', 'barrel', 'scraping', 'desperation', 'The', 'party', 's', 'MPs', 'were', 'also', 'drawn', 'into', 'the', 'row', 'with', 'Martin', 'Whitfield', 'saying', 'Mr', 'Leonard', 'had', 'questions', 'to', 'answer', 'over', 'the', 'affair', 'while', 'Ms', 'Rowley', 'insisted', 'that', 'her', 'father', 'had', 'always', 'been', 'very', 'supportive', 'of', 'Ms', 'Dugdale', 'A', 'spokesman', 'for', 'Mr', 'Leonard', 's', 'campaign', 'team', 'said', 'If', 'there', 'were', 'any', 'attempts', 'to', 'undermine', 'Kezia', 'they', 'did', 'not', 'involve', 'Richard', 'and', 'he', 'knew', 'nothing', 'of', 'them', 'Any', 'statements', 'that', 'he', 'did', 'are', 'completely', 'false', 'A', 'Scottish', 'Labour', 'spokesman', 'said', 'Alex', 'was', 'having', 'what', 'he', 'believed', 'to', 'be', 'a', 'private', 'conversation', 'with', 'a', 'student', 'and', 'a', 'political', 'activist', 'He', 'has', 'not', 'and', 'will', 'not', 'publicly', 'back', 'a', 'candidate', 'He', 'has', 'no', 'intention', 'of', 'relinquishing', 'his', 'role', 'as', 'interim', 'leader', 'of', 'the', 'Labour', 'Party', 'in', 'Scotland', 'until', 'after', 'the', 'current', 'leadership', 'contest', 'is', 'over', 'Unifying', 'message', 'Mr', 'Sarwar', 's', 'campaign', 'has', 'been', 'dogged', 'by', 'controversy', 'over', 'his', 'family', 's', 'company', 'which', 'saw', 'him', 'relinquish', 'his', 'shares', 'after', 'criticism', 'from', 'opponents', 'including', 'First', 'Minister', 'Nicola', 'Sturgeon', 'The', 'rival', 'camps', 'issued', 'comments', 'over', 'the', 'leadership', 'row', 'even', 'as', 'Mr', 'Corbyn', 'was', 'making', 'his', 'keynote', 'speech', 'to', 'the', 'party', 's', 'UK', 'conference', 'in', 'Brighton', 'Mr', 'Corbyn', 'insisted', 'Labour', 'was', 'on', 'the', 'way', 'back', 'in', 'Scotland', 'becoming', 'once', 'again', 'the', 'champion', 'of', 'social', 'justice', 'He', 'added', 'Thank', 'you', 'Kezia', 'and', 'thank', 'you', 'Alex', 'And', 'whoever', 'next', 'leads', 'Scottish', 'Labour', 'our', 'unifying', 'socialist', 'message', 'will', 'continue', 'to', 'inspire', 'both', 'south', 'and', 'north', 'of', 'the', 'border'], 'taxonomy': 'scotland politics', 'article_tag': '---', 'premium': False, 'category': 'politics', 'keywords': '---', 'list_of_tags': '---', 'title': \"Scottish Labour MSPs embroiled in row over leadership 'plot' - BBC NewsBBC News\", 'subtitle': 'Scottish Labour MSPs trade barbs over claims of a \"plot\" to replace Kezia Dugdale with Richard Leonard.', 'opinion': False}\n"
     ]
    }
   ],
   "source": [
    "gp = GensimPreprocessor(configuration['PREPROCESS'])\n",
    "data_preprocessed = gp.preprocess_data(data)\n",
    "print(data_preprocessed.__next__())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now gensim models expect a `namedtuple` as input. Since the original gensim class for converting the input (`TaggedDocument`) is throwing away all extra information except `tags`,`words` , if you think you will need some other field in further processing you can use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_doc2vec = DocumentsTagged(data_preprocessed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each docuemnt field will be transformed to an attribute of the class `TaggedDocument`. You can access these attributes with the `.` operation. Since we all still working with generator we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-01-23 16:37:07,509 : INFO : load: Documents in /home/ele/Scrivania/HPI/data/websci/samuele.garda/politics/2017.json discarded beacuse of missing field : 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-01-23 16:37:07,513 : INFO : load: Retrieved 458 documents\n",
      "2018-01-23 16:37:07,515 : INFO : utils: 'load_and_process'  0.65 s\n"
     ]
    }
   ],
   "source": [
    "corpus = load_and_process(data_doc2vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'politics'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[0].category"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TRAIN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is possible to actually train the models. First of all we need to initaliaze the models with the corpus we loaded (i.e. creating the actual shallow neural network architecture for training embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alpha = 0.025\n",
      "min_alpha = 0.001\n",
      "epochs = 20\n",
      "shuffle = True\n",
      "adapt_alpha = False\n",
      "chekpoint = ./models/newspaper/std\n",
      "quality_check_infered = 5\n",
      "quality_check_trained = 5\n"
     ]
    }
   ],
   "source": [
    "for key,value in configuration['TRAIN'].items(): print(key,value, sep = ' = ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-01-23 16:37:12,159 : INFO : train_models: Initializing models with corpus...\n",
      "2018-01-23 16:37:12,166 : INFO : doc2vec: collecting all words and their counts\n",
      "2018-01-23 16:37:12,169 : INFO : doc2vec: PROGRESS: at example #0, processed 0 words (0/s), 0 word types, 0 tags\n",
      "2018-01-23 16:37:12,311 : INFO : doc2vec: collected 17378 word types and 458 unique tags from a corpus of 458 examples and 289581 words\n",
      "2018-01-23 16:37:12,312 : INFO : word2vec: Loading a fresh vocabulary\n",
      "2018-01-23 16:37:12,341 : INFO : word2vec: min_count=5 retains 5097 unique words (29% of original 17378, drops 12281)\n",
      "2018-01-23 16:37:12,342 : INFO : word2vec: min_count=5 leaves 269830 word corpus (93% of original 289581, drops 19751)\n",
      "2018-01-23 16:37:12,373 : INFO : word2vec: deleting the raw counts dictionary of 17378 items\n",
      "2018-01-23 16:37:12,375 : INFO : word2vec: sample=0 downsamples 0 most-common words\n",
      "2018-01-23 16:37:12,379 : INFO : word2vec: downsampling leaves estimated 269830 word corpus (100.0% of prior 269830)\n",
      "2018-01-23 16:37:12,384 : INFO : word2vec: estimated required memory for 5097 words and 100 dimensions: 47676900 bytes\n",
      "2018-01-23 16:37:12,411 : INFO : doc2vec: using concatenative 2100-dimensional layer1\n",
      "2018-01-23 16:37:12,413 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:12,524 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:12,645 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:12,747 : INFO : doc2vec: using concatenative 1100-dimensional layer1\n",
      "2018-01-23 16:37:12,749 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:12,863 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:12,988 : INFO : doc2vec: using concatenative 2200-dimensional layer1\n",
      "2018-01-23 16:37:12,989 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:13,127 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:13,247 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:13,371 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:13,482 : INFO : doc2vec: using concatenative 4200-dimensional layer1\n",
      "2018-01-23 16:37:13,483 : INFO : word2vec: resetting layer weights\n",
      "2018-01-23 16:37:13,615 : INFO : utils: 'init_models'  1.46 s\n"
     ]
    }
   ],
   "source": [
    "models = ModelsTrainer.init_models(models_to_train,corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('Doc2Vec(dm-c,d100,n20,w10,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be706d8>), ('Doc2Vec(dbow,d200,n20,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70390>), ('Doc2Vec(dbow,d100,n20,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be702e8>), ('Doc2Vec(dm-c,d100,n20,w5,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70630>), ('Doc2Vec(dm-s,d200,n20,w10,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70588>), ('Doc2Vec(dm-c,d200,n20,w5,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70780>), ('Doc2Vec(dm-s,d100,n20,w5,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be701d0>), ('Doc2Vec(dm-s,d200,n20,w5,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be704e0>), ('Doc2Vec(dm-s,d100,n20,w10,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70438>), ('Doc2Vec(dm-c,d200,n20,w10,mc5,t4)', <gensim.models.doc2vec.Doc2Vec object at 0x7fd00be70828>)])\n"
     ]
    }
   ],
   "source": [
    "print(models)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the actual training can begin. Now you have to chose whether to stick with the gensim standard function for training models or use a manually controlled learning rate decay. This can be set via the `adapt_alpha` parameter. If you think you know what you are doing set go for the manual control. This will allow you to set an inital learning rate and the minimum value you want to reach at the end of the process. For now the decay is set to be `alpha_delta = (alpha - min_alpha) / epochs`. Please take a look at `train_models` if you want to modify this. Finally if you use this procedure and set `shuffle = True` the documents will be shuffled at each epoch. This is done because it as been seen (https://rare-technologies.com/doc2vec-tutorial/) that manually controlling the learning rate and shuffling the corpus at each epoch can lead to better embeddings. Since the opinions about this are conflicting the final word is left to you.  Otherwise all the parameters are ignored and a single shuffle operation is done before training. The checkpoint is used to set a folder where each model is saved at the end of the training process. We can still use the configuration file or the specific functions or the configuration file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "alpha = 0.025\n",
    "min_alpha = 0.001\n",
    "epochs = 10\n",
    "\n",
    "ModelsTrainer.train_from_config(models,corpus,configuration['TRAIN'])\n",
    "\n",
    "\n",
    "ModelsTrainer.train_manual_lr(models = models ,corpus = corpus ,epochs = epochs,\n",
    "                              alpha = alpha,minalpha = min_alpha,alpha_delta = (alpha - min_alpha) / epochs,\n",
    "                              checkpoint = 'models')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since for illustrational purposes 10 models are a bit too much I will use the specific function. Do not forget to create a checkpoint folder!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-01-23 16:37:25,124 : INFO : train_models: Training models with standar learning rate decay...\n",
      "\n",
      "2018-01-23 16:37:25,128 : INFO : train_models: Training model Doc2Vec(dm-c,d100,n20,w10,mc5,t4)\n",
      "2018-01-23 16:37:52,788 : INFO : train_models: Doc2Vec(dm-c,d100,n20,w10,mc5,t4) completed training in 27.657s\n",
      "\n",
      "2018-01-23 16:37:52,872 : INFO : utils: 'train_standard'  27.75 s\n"
     ]
    }
   ],
   "source": [
    "logging.getLogger(\"gensim\").setLevel(logging.WARNING) # gensim is quiete verbose...\n",
    "\n",
    "model = {next(iter(models.keys())) : next(iter(models.values()))} # pick just one model here\n",
    "\n",
    "os.mkdir('./models')\n",
    "\n",
    "ModelsTrainer.train_standard(models = model ,corpus = corpus ,epochs = 3,shuffle_docs  = True,checkpoint = './models')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally the last step is to evaluate the quality of our embeddings. This kind of inspection can be performed via `QualityChecker` class. Moreover you can inherit from this class if you wish to implement your own evaluation metrics!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "checker = QualityChecker(models=model,corpus=corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-01-23 16:38:06,300 : INFO : quality_check: Doc2Vec(dm-c,d100,n20,w10,mc5,t4) - training - most similar for ['http://www.bbc.co.uk/news/uk-politics-41887401'] : [('http://www.bbc.co.uk/news/uk-politics-41604675', 0.9974935054779053), ('http://www.bbc.co.uk/news/uk-politics-41941414', 0.9966021776199341), ('http://www.bbc.co.uk/news/uk-politics-41437636', 0.9963968992233276), ('http://www.bbc.co.uk/news/uk-politics-39294904', 0.9953902959823608), ('http://www.bbc.co.uk/news/uk-politics-41642051', 0.9949260354042053)]\n",
      "\n",
      "2018-01-23 16:38:06,437 : INFO : quality_check: Doc2Vec(dm-c,d100,n20,w10,mc5,t4) - inferred - most similar for ['http://www.bbc.co.uk/news/uk-politics-41887401'] : [('http://www.bbc.co.uk/news/uk-politics-41544588', 0.973179817199707), ('http://www.bbc.co.uk/news/uk-politics-41585430', 0.9663538336753845), ('http://www.bbc.co.uk/news/uk-politics-41961389', 0.9655001759529114), ('http://www.bbc.co.uk/news/uk-scotland-scotland-politics-41939416', 0.9604285955429077), ('http://www.bbc.co.uk/news/uk-politics-41899727', 0.9578949213027954)]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "checker.base_check_from_config(config_train=configuration['TRAIN'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "checker.inferred_most_similar(topk = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "checker.trained_most_similar(topk=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This basic check is to assess:\n",
    "\n",
    "    if the training vectors capture the structure of the data. Then you can manually see if the documents with the highest cosine similarity are effectively similar (deploy trained data)\n",
    "    \n",
    "    if the inferred vector for a random document have the highest cosine similarity with the original vector (deploy model on unseen examples)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "DocEmbeddings",
   "language": "python",
   "name": "doc_embeddings"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}