notebooks/pdi/egu-lda.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EGU Topic Modeling and prediction\n",
    "\n",
    "This notebook uses topic modeling to analyze the EGU conference using the abstracts submitted for the years 2011 to 2018. The notebook can be used to detect trends and visualize topics along these years using one or many categories. The abstracts were parsed into text and ingested into Apache Solr.\n",
    "\n",
    "We parsed the PDFs using PDFMiner's utility pdf2txt and ingested the resulting text files into Solr\n",
    "\n",
    "```sh\n",
    "ls *.pdf | xargs -n1 -P8 bash -c 'pdf2txt.py -o output/$0.txt -t text $0'\n",
    "```\n",
    "\n",
    "**Example Document in Solr**\n",
    " \n",
    " \n",
    " \n",
    "```json\n",
    "doc = {\n",
    "\"entities\":[\n",
    "    \"Jeffrey Obelcz  and Warren T. Wood\",\n",
    "    \"NRC Postdoctoral Fellow\",\n",
    "    \"Naval Research Lab\",\n",
    "    \"Seaﬂoor Sciences\",\n",
    "    \"United States jbobelcz@gmail.com\",\n",
    "    \"Naval\",\n",
    "    \"Research Lab\",\n",
    "    \"Seaﬂoor Sciences\",\n",
    "    \"United States\"],\n",
    "\"id\": \"EGU2018-9778\",\n",
    "\"sessions\": [\"ESSI4.3\"],\n",
    "\"file\": [\"EGU2018-9778\"],\n",
    "\"presentation\": [\"Posters\"],\n",
    "\"year\": [2018],\n",
    "\"title\": [\"Towards a Quantitative Understanding of Parameters Driving Submarine Slope Failure: A Machine Learning Approach\"],\n",
    "\"category\": [\"ESSI\"],\n",
    "\"abstract\":[\"Submarine slope failure is a ubiquitous process and dominant pathway for sediment and organic carbon ﬂux from continental margins to the deep sea. Slope failure occurs over a wide range of temporal and spatial scales ...\"]\n",
    "}\n",
    "```\n",
    "\n",
    "## Application\n",
    "\n",
    "This notebook can be used to analyze what a corpora of scientific text talks about, in this case we used EGU abstracts but it can be used on any text corpora.\n",
    "\n",
    "The topics are examined using LDAVis, which displays the topics on an X-Y plot (intertopic distance map). Topic are represented by circles whose areas are proportional to the relative prevalences of the topics in the corpus. In the display the user can enter a topic number (note 1-base numbering vs. 0-base numbering in Topic List below cells 5 and 7); the terms are displayed on the right, ranked by significance (weight). A topic can be selected on the fly by hovering over its circle; clicking selects that topic. A user can also click on a term in the RH panel to show the topics in which that term occurs. \n",
    "\n",
    "The slider at the top of the RH panel allows the user to vary the “saliency”, i.e. uniqueness of the terms to that topic. A value of 0.6 is optimal, according to the authors of the algorithm. Blue bars represent overall term frequency while red bars show term frequency within the topic, which will be different when the saliency Is selected be less than one.\n",
    "\n",
    "Please see the annotated image of the LDAVis display.\n",
    "\n",
    "![](https://raw.githubusercontent.com/USCDataScience/pdi-topics/master/notebooks/pdi/img/pylda.png)\n",
    "\n",
    "\n",
    "\n",
    "The final cell of this notebook lists the titles and IDs of abstracts belonging to a specified topic.\n",
    "\n",
    "* category: the Program Group, e.g. CL, AS etc. Keep in mind that these codes have changed through the years.\n",
    "* presentation: Presentation type: oral, poster, pico etc.\n",
    "* session: the session that the abstract appeared in; sessions may be co-organized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### EGU Programme Groups Codes:\n",
    "\n",
    "-----\n",
    "\n",
    "### Disciplinary Sessions \n",
    "\n",
    "Geodynamics (GD) -\n",
    "Geosciences Instrumentation & Data Systems (GI) -\n",
    "Geomorphology (GM) -\n",
    "Geochemistry, Mineralogy, Petrology & Volcanology (GMPV) -\n",
    "Hydrological Sciences (HS) -\n",
    "Natural Hazards (NH) -\n",
    "Nonlinear Processes in Geosciences (NP) -\n",
    "Ocean Sciences (OS) -\n",
    "Planetary & Solar System Sciences (PS) -\n",
    "Seismology (SM) -\n",
    "Stratigraphy, Sedimentology & Palaeontology (SSP) -\n",
    "Soil System Sciences (SSS) -\n",
    "Solar-Terrestrial Sciences (ST) -\n",
    "Tectonics & Structural Geology (TS) -\n",
    "Atmospheric Sciences (AS) -\n",
    "Biogeosciences (BG) -\n",
    "Climate: Past, Present, Future (CL) -\n",
    "Cryospheric Sciences (CR) -\n",
    "Earth Magnetism & Rock Physics (EMRP) -\n",
    "Energy, Resources and the Environment (ERE) -\n",
    "Earth & Space Science Informatics (ESSI) -\n",
    "Geodesy (G) -\n",
    "Geodynamics (GD) -\n",
    "Geosciences Instrumentation & Data Systems (GI) -\n",
    "Geomorphology (GM) -\n",
    "Geochemistry, Mineralogy, Petrology & Volcanology (GMPV) -\n",
    "Hydrological Sciences (HS) -\n",
    "Natural Hazards (NH) -\n",
    "Nonlinear Processes in Geosciences (NP) -\n",
    "Ocean Sciences (OS) -\n",
    "Planetary & Solar System Sciences (PS) -\n",
    "Seismology (SM) -\n",
    "Stratigraphy, Sedimentology & Palaeontology (SSP) -\n",
    "Soil System Sciences (SSS) -\n",
    "Solar-Terrestrial Sciences (ST) -\n",
    "Tectonics & Structural Geology (TS) -\n",
    "\n",
    "### Union Sessions\n",
    "\n",
    "Union Symposia (US)\n",
    "Great Debates (GDB)\n",
    "Medal Lectures (ML)\n",
    "Short courses (SC)\n",
    "Educational and Outreach Symposia (EOS)\n",
    "EGU Plenary, Ceremonies and Networking (PCN)\n",
    "Feedback and administrative meetings (FAM)\n",
    "Townhall and splinter meetings (TSM)\n",
    "Side events (SEV)\n",
    "Press conferences (PC)\n",
    "\n",
    "#### Interdisciplinary Events (IE)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 1: Import requirements\n",
    "\n",
    "import urllib\n",
    "import json\n",
    "import string\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# NLP libraries\n",
    "import spacy\n",
    "from gensim import corpora, models\n",
    "from gensim.models.ldamulticore import LdaMulticore, LdaModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 2: Querying Solr and building our corpus out of the matching documents \n",
    "\n",
    "# terms = ['ice', 'climate'] to include only abstracts with specified terms\n",
    "terms = ['*']\n",
    "years = ['2018']\n",
    "entities = ['*']\n",
    "sessions = ['*']\n",
    "\n",
    "# Return \"page_size\" documents with each Solr query until complete\n",
    "page_size = 5000\n",
    "cursorMark = '*'\n",
    "\n",
    "solr_documents = []\n",
    "solr_root = 'http://qa.pdi-solr.apps.nsidc.org/solr/egu/select?indent=on&'\n",
    "more_results = True\n",
    "\n",
    "\n",
    "if terms[0] != '*':\n",
    "    terms_wirldcard = ['*' + t + '*' for t in terms]\n",
    "else:\n",
    "    terms_wirldcard = ['*']\n",
    "    \n",
    "if sessions[0] != '*':\n",
    "    sessions_wirldcard = ['*' + s + '*' for s in sessions]\n",
    "else:\n",
    "    sessions_wirldcard = ['*']\n",
    "    \n",
    "if entities[0] != '*':\n",
    "    entities_wirldcard = ['*' + e + '*' for e in entities]\n",
    "else:\n",
    "    entities_wirldcard = ['*']\n",
    "\n",
    "terms_query = '%20OR%20abstract:'.join(terms_wirldcard)\n",
    "years_query = '%20OR%20year:'.join(years)  \n",
    "entities_query = '%20OR%20entities:'.join(entities_wirldcard)\n",
    "sessions_query = '%20OR%20sessions:'.join(sessions_wirldcard)\n",
    "query_string = 'q=(abstract:{})%20AND(year:{})' + \\\n",
    "                '%20AND%20(entities:{})%20AND%20(sessions:{})&wt=json&rows={}&cursorMark={}&sort=id+asc'\n",
    "while (more_results):    \n",
    "    solr_query = query_string.format(terms_query,\n",
    "                                     years_query,\n",
    "                                     entities_query,\n",
    "                                     sessions_query,\n",
    "                                     page_size,\n",
    "                                     cursorMark)\n",
    "    solr_url = solr_root + solr_query\n",
    "    print('Querying: \\n' + solr_url)\n",
    "    req = urllib.request.Request(solr_url)\n",
    "    # parsing response\n",
    "    r = urllib.request.urlopen(req).read()\n",
    "    json_response = json.loads(r.decode('utf-8'))\n",
    "    solr_documents.extend(json_response['response']['docs'])\n",
    "    nextCursorMark = json_response['nextCursorMark']\n",
    "    if (nextCursorMark == cursorMark):\n",
    "        more_results = False\n",
    "        break\n",
    "    else: \n",
    "        cursorMark = nextCursorMark\n",
    "\n",
    "total_found = json_response['response']['numFound']\n",
    "print(\"Processing {0} out of {1} total. \\n\".format(len(solr_documents), total_found))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 3, remove stop words and create an array of documents\n",
    "import string\n",
    "\n",
    "my_stop_words = {'et_al','change', 'different'}\n",
    "\n",
    "def remove_stop_words(text):\n",
    "    cleaned_test = [w for w in text if w not in my_stop_words]\n",
    "    cleaned_test = [w for w in cleaned_test if len(w) > 2]\n",
    "    return cleaned_test\n",
    "\n",
    "document_list = []\n",
    "# bigram corpus will contain an array of documents and their tokens, with bigram tokens included\n",
    "bigram_corpus = []\n",
    "\n",
    "for doc in solr_documents:\n",
    "    bigrams = remove_stop_words(doc['bigrams'][0].split())\n",
    "    if 'sessions' in doc:\n",
    "        sessions = doc['sessions'][0]\n",
    "    else:\n",
    "        sessions = 'NAN'\n",
    "    if 'category' in doc:\n",
    "        category = doc['category'][0]\n",
    "    else:\n",
    "        category = 'NAN'\n",
    "    document_list.append({ 'id': doc['id'],\n",
    "                                   'text': bigrams,\n",
    "                                   'year': str(doc['year'][0]),\n",
    "                                   'title': doc['title'][0],\n",
    "                                   'category': category.replace('<',''),\n",
    "                                   'sessions':sessions})\n",
    "    bigram_corpus.append(bigrams)\n",
    "\n",
    "df = pd.DataFrame.from_dict(document_list)\n",
    "axis_category = pd.DataFrame(df.groupby(['category', 'year'])['category'].count()).rename(columns={'category': 'count'})\n",
    "print(axis_category)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 4: Using GENSIM to do topic modelling\n",
    "\n",
    "# num pases should be adjusted, 3 is just a guesstimate of when convergence will be achieved.\n",
    "num_passes = 2\n",
    "num_topics = 20\n",
    "words_per_topic = 9\n",
    "print_topics = False\n",
    "\n",
    "dictionary = corpora.Dictionary(bigram_corpus)\n",
    "lda_corpus = [dictionary.doc2bow(text) for text in bigram_corpus]\n",
    "\n",
    "lda_model = LdaMulticore(lda_corpus,\n",
    "                         num_topics=num_topics,\n",
    "                         id2word=dictionary,\n",
    "                         passes=num_passes,\n",
    "                         workers=2\n",
    "                        )\n",
    "\n",
    "topics = lda_model.print_topics(num_topics=num_topics, num_words=words_per_topic)\n",
    "if print_topics:\n",
    "    print (\"Topic List: \\n\")\n",
    "    for topic in topics:\n",
    "        t = str((int(topic[0])+ 1))\n",
    "        print('Topic ' + t + ': ', topic[1:])\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import pyLDAvis.gensim\n",
    "print (\"\\nPyLDAVis: \\n\")\n",
    "pyLDAvis.enable_notebook()\n",
    "pyLDAvis.gensim.prepare(corpus=lda_corpus,\n",
    "                        topic_model=lda_model,\n",
    "                        dictionary=dictionary,\n",
    "                        sort_topics=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 4: Listing papers containing a particular ngram\n",
    "\n",
    "# use unigrams or bigrams\n",
    "terms = set(['carbon_cycle'])\n",
    "\n",
    "top_n = 10\n",
    "\n",
    "def createLink(doc):\n",
    "    baseURL = 'https://meetingorganizer.copernicus.org/EGU' + str(doc['year']) + '/' + doc['id'] + '.pdf'\n",
    "    return baseURL\n",
    "\n",
    "# Ode to Python's comprehension lists\n",
    "matches  = [doc for doc in document_list if terms == set(doc['text']).intersection(terms)]\n",
    "\n",
    "from IPython.core.display import display, HTML\n",
    "# Let's predict the first 10 documents\n",
    "for doc in matches[0:top_n]:\n",
    "    display(HTML('<br>Abstract <a href=\"{}\" target=\"_blank\">{}</a> '.format(\n",
    "        createLink(doc),\n",
    "        doc['id'])))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now that we have a trained model we can classify a new unseen document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 5: Classifying an unseen document using our GENSIM model\n",
    "\n",
    "# For practical purposes we use a mocked up document but we can easily query Solr or another store to get the content we want to classify\n",
    "# Eventually all this should be served in as a web service \n",
    "# taken from https://meetingorganizer.copernicus.org/EGU2018/EGU2014-2415.pdf\n",
    "\n",
    "unseen_document = \"\"\"\n",
    "Waves  in  the  Southern  Ocean  are  the  largest  in  the  planet.  In  the  Southern  Hemisphere,  the  absence  of  large\n",
    "landmasses at high latitudes allows the wind to feed energy into the ocean over a virtually unlimited fetch. The\n",
    "enormous amount air-sea momentum exchanged over the Southern Ocean plays a substantial role on the global\n",
    "climate. However, large biases affect the estimation of wave regime around the Antarctic continent making climate\n",
    "prediction susceptible to uncertainty.\n",
    " \"\"\"\n",
    "\n",
    "parsed_doc = list(unseen_document.split())\n",
    "vec = dictionary.doc2bow(parsed_doc)\n",
    "predicted_topics = lda_model[vec]\n",
    "predicted_topics = [(p[0]+1, p[1]) for p in predicted_topics]\n",
    "print(predicted_topics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmarking Model Coherence "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 5: Plotting model coherence, this takes some time depending on iterations and model used.\n",
    "\n",
    "\n",
    "# lda_model.log_perplexity(lda_corpus)\n",
    "\n",
    "# Enable logging for gensim - optional\n",
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "from gensim.models import CoherenceModel\n",
    "\n",
    "initial_topics = 4 \n",
    "max_topics = 50\n",
    "\n",
    "def compute_coherence_values(dictionary, corpus, limit=10, start=2, step=2):\n",
    "    \"\"\"\n",
    "    Compute c_v coherence for various number of topics\n",
    "\n",
    "    Parameters:\n",
    "    ----------\n",
    "    dictionary : Gensim dictionary\n",
    "    corpus : Gensim corpus\n",
    "    texts : List of input texts\n",
    "    limit : Max num of topics\n",
    "\n",
    "    Returns:\n",
    "    -------\n",
    "    model_list : List of LDA topic models\n",
    "    coherence_values : Coherence values corresponding to the LDA model with respective number of topics\n",
    "    \"\"\"\n",
    "    coherence_values = []\n",
    "    model_list = []\n",
    "    for num_topics in range(start, limit, step):\n",
    "        print(\"Processing {0} topics \\n\".format(num_topics))\n",
    "#         model = LdaModel(corpus=corpus,\n",
    "#                           id2word=dictionary,\n",
    "#                           num_topics=num_topics, \n",
    "#                           random_state=100,\n",
    "#                           update_every=0,\n",
    "#                           chunksize=100000,\n",
    "#                           passes=1,\n",
    "#                           alpha='auto',\n",
    "#                           per_word_topics=False)\n",
    "        \n",
    "        model = LdaMulticore(corpus=corpus,\n",
    "                         num_topics=num_topics,\n",
    "                         id2word=dictionary,\n",
    "                         passes=5,\n",
    "                         workers=8\n",
    "                        )\n",
    "\n",
    "\n",
    "        model_list.append(model)\n",
    "        coherencemodel = CoherenceModel(model=model, corpus=lda_corpus, texts=bigram_corpus, coherence='c_v')\n",
    "        coherence_values.append(coherencemodel.get_coherence())\n",
    "\n",
    "    return model_list, coherence_values\n",
    "\n",
    "model_list, coherence_values = compute_coherence_values(dictionary=dictionary,\n",
    "                                                        corpus=lda_corpus,\n",
    "                                                        start=initial_topics,\n",
    "                                                        limit=max_topics,\n",
    "                                                        step=2)\n",
    "\n",
    "limit=max_topics; start=initial_topics; step=2;\n",
    "x = range(start, limit, step)\n",
    "plt.plot(x, coherence_values)\n",
    "plt.xlabel(\"Num Topics\")\n",
    "plt.ylabel(\"Coherence score\")\n",
    "plt.legend((\"coherence_values\"), loc='best')\n",
    "plt.show()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}