Presentation-TagMe.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container {width:95% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container {width:95% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## TagMe: Entity Linking on the Fly "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Entity Linking\n",
    "\n",
    "The task of recognizing and disambiguating named entities to a knowledge base.\n",
    "\n",
    "<center><img src=\"Figures/iowa.png\" height=\"50%\" width=\"50%\" align=\"center\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## TagMe\n",
    "\n",
    "TagMe (Ferragina and Scaiella, CIKM 2010): language independent annotations of short texts.\n",
    "\n",
    "A must-have end-to-end baseline in Entity Linking. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why I like TagMe\n",
    "\n",
    "- It answers to a pressing need in DH and CSS.\n",
    "\n",
    "- It adds semantics without dealing (directly) with it.\n",
    "\n",
    "- It is language independent.\n",
    "\n",
    "- It is very intuitive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Mention Identification via Link Probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Obama', 'won', 'the', '2008', 'Iowa', 'caucuses', 'while', 'Hillary', 'was', 'leading', 'in', 'the', 'polls', '.', 'Obama won', 'won the', 'the 2008', '2008 Iowa', 'Iowa caucuses', 'caucuses while']\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "sentence = \"Obama won the 2008 Iowa caucuses while Hillary was leading in the polls.\"\n",
    "ngram_up_to = 3\n",
    "\n",
    "def get_all_ngrams(text,ngram_up_to):\n",
    "    \"\"\"Returns all ngrams from a text up to a certain number.\n",
    "    \n",
    "    Args:\n",
    "        text: a string.\n",
    "        ngram_up_to: a integer.\n",
    "    Returns:\n",
    "        A list of ngrams.\n",
    "    \"\"\"\n",
    "    \n",
    "    tokens = nltk.word_tokenize(text)\n",
    "    ngrams = [\" \".join(x) for n in range(1,ngram_up_to+1) for x in nltk.ngrams(tokens,n)]        \n",
    "    return ngrams\n",
    " \n",
    "ngrams = get_all_ngrams(sentence,ngram_up_to)\n",
    "print (ngrams[:20]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"Figures/wiki-size.png\" height=\"50%\" width=\"50%\" align=\"center\"></center>\n",
    "<center><img src=\"Figures/size-gb.png\" height=\"30%\" width=\"30%\" align=\"center\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all ready!\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "\n",
    "with open(\"Resources/overall_mentions_freq.pickle\", \"rb\") as f:\n",
    "    mentions_freq = pickle.load(f)\n",
    "\n",
    "with open(\"Resources/overall_ngrams_freq_cleaned.pickle\", \"rb\") as f:\n",
    "    ngrams_freq = pickle.load(f)\n",
    "        \n",
    "with open(\"Resources/overall_entity_freq.pickle\", \"rb\") as f:\n",
    "    entity_freq = pickle.load(f)\n",
    "        \n",
    "with open(\"Resources/mention_overall_dict.pickle\", \"rb\") as f:\n",
    "    mention_to_entities = pickle.load(f)\n",
    "\n",
    "with open(\"Resources/entity_overall_dict.pickle\", \"rb\") as f:\n",
    "    entity_inlinks = pickle.load(f)\n",
    "    \n",
    "print (\"all ready!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Abraham%20Lincoln', 'Alaska', 'Apollo%2011', 'The%20Amazing%20Spider-Man']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entity_inlinks[\"Barack%20Obama\"][:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.007931134539123707"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_link_proba(ngram):\n",
    "    \"\"\"Returns probability of a ngram to be an entity mention.\n",
    "    \n",
    "    Args:\n",
    "        ngram: a string.\n",
    "    Returns:\n",
    "        A probability score.\n",
    "        If a ngram is missing from the mentions_freq or the ngrams_freq dictionaries,\n",
    "        then it returns None.       \n",
    "    \"\"\"\n",
    "    \n",
    "    global mentions_freq, ngrams_freq\n",
    "    \n",
    "    try:\n",
    "        mention_freq = mentions_freq[ngram]\n",
    "        ngram_freq = ngrams_freq[ngram]\n",
    "    except KeyError:\n",
    "        return None\n",
    "    else:\n",
    "        return mention_freq / ngram_freq        \n",
    "\n",
    "get_link_proba(\"Obama\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Obama', 0.007931134539123707), ('won', 0.0005864751645721125), ('the', 3.886926584799571e-07), ('2008', 0.009146341463414634), ('Iowa', 0.16056687851559645), ('caucuses', 0.029350104821802937), ('while', 3.0125018828136766e-06), ('Hillary', 0.012371862849063274), ('was', 8.64148583781452e-07), ('leading', 0.0003902267091300462), ('in', 2.4781263311631732e-06), ('the', 3.886926584799571e-07), ('polls', 0.005082005082005082), ('.', 2.4849563271031407e-06), ('won the', 1.5227885303567893e-05), ('Iowa caucuses', 0.41025641025641024)]\n"
     ]
    }
   ],
   "source": [
    "candidate_mentions = [(ngram, get_link_proba(ngram)) for ngram in ngrams if ngram in ngrams_freq]\n",
    "\n",
    "print (candidate_mentions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Iowa caucuses', 'Hillary', '2008', 'Obama', 'polls', 'won', 'leading', 'won the', 'while', 'was']\n"
     ]
    }
   ],
   "source": [
    "def remove_substrings(candidate_mentions):\n",
    "    \"\"\"Removes very short candidate mentions (<2 chars) \n",
    "    and ngrams that are substrings of a more popular one.\n",
    "    \n",
    "    Args:\n",
    "        candidate_mentions: list of tuples (candidate_mention (str), probability score (float)).\n",
    "    Returns:\n",
    "        A final list of mentions as strings.      \n",
    "    \"\"\"\n",
    "    \n",
    "    candidate_mentions = [(cand,score) for cand,score in candidate_mentions if len(cand)>1]\n",
    "        \n",
    "    candidate_mentions.sort(key=lambda cand: cand[1], reverse=True)\n",
    "    \n",
    "    mentions = []\n",
    "    for cand,score in candidate_mentions:\n",
    "        if not any([cand in mention for mention in mentions]):\n",
    "            mentions.append(cand)\n",
    "    return mentions\n",
    " \n",
    "mentions = remove_substrings(candidate_mentions)\n",
    "\n",
    "print (mentions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['/wiki/Hillary_Clinton', 29],\n",
       " ['/wiki/Edmund_Hillary', 5],\n",
       " ['/wiki/Hilary_(name)', 1]]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scripts import output_helpers\n",
    "\n",
    "output_helpers.show_top_entity_candidates(\"Hillary\",mention_to_entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## 2. Disambiguation via Commonness and Relatedness\n",
    "\n",
    "<img src=\"Figures/tagme-alg.png\" height=\"100%\" width=\"100%\" align=\"center\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hillary%20Clinton 0.8285714285714286\n",
      "Edmund%20Hillary 0.14285714285714285\n",
      "Hilary%20%28name%29 0.02857142857142857\n"
     ]
    }
   ],
   "source": [
    "def get_commoness(mention,entity_candidate):\n",
    "    \"\"\"Compute how common it is that a given mention would point to a specific entity.\n",
    "    \n",
    "    Args:\n",
    "        mention: a string, entry in the global dictionary mentions_freq.\n",
    "        entity_candidate: a string, entry in the global dictionary mention_to_entities.\n",
    "    Returns:\n",
    "        A commoness score (between 0.0 and 1.0)      \n",
    "    \"\"\"\n",
    "    \n",
    "    global mention_to_entities, mentions_freq\n",
    "    \n",
    "    entity_mention_freq = mention_to_entities[mention][entity_candidate]\n",
    "    mention_freq = mentions_freq[mention]\n",
    "    return entity_mention_freq / mention_freq\n",
    "\n",
    "mention = \"Hillary\"\n",
    "candidates = mention_to_entities[mention]\n",
    "\n",
    "for cand in candidates:\n",
    "    commoness = get_commoness(mention,cand)\n",
    "    print (cand, commoness)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.788401339278572\n",
      "0.3054468828291247\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "n_all_entities = len(entity_inlinks)\n",
    "\n",
    "def get_relatedness(e1, e2):\n",
    "    \"\"\"Compute Witten & Milne (AAAI, 2008) relatedness between two entities.\n",
    "    \n",
    "    Args:\n",
    "        e1,e2: two entities.\n",
    "    Returns:\n",
    "        A relatedness score (between 0.0 and 1.0)      \n",
    "    \"\"\"\n",
    "    \n",
    "    global n_all_entities, entity_inlinks\n",
    "        \n",
    "    entities = (e1, e2)\n",
    "    entities_in_links = [set(entity_inlinks[entity]) for entity in entities]\n",
    "    n_ent_in_links = [len(entity) for entity in entities_in_links]\n",
    "        \n",
    "    conjunct_in_links = len([in_link for in_link in entities_in_links[0] if in_link in entities_in_links[1]])\n",
    "        \n",
    "    if conjunct_in_links==0:\n",
    "        return 0.0\n",
    "    \n",
    "    num = math.log(max(n_ent_in_links)) - math.log(conjunct_in_links)\n",
    "    den = math.log(n_all_entities) - math.log(min(n_ent_in_links))\n",
    "    rel = 1 - (num / den)\n",
    "    if rel == 0:\n",
    "        return 0.0\n",
    "    return rel\n",
    "            \n",
    "print (get_relatedness(\"Barack%20Obama\",\"Hillary%20Clinton\"))\n",
    "print (get_relatedness(\"Barack%20Obama\",\"Edmund%20Hillary\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22436376077998585"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_relevance_score(entity,other_mentions):\n",
    "    \"\"\"Compute relevance score for an entity given surrounding mentions\n",
    "    \n",
    "    Args:\n",
    "        entity: the entity under study.\n",
    "        other_mentions: list of tuples (mention,[candidate entities])\n",
    "    Returns:\n",
    "        A relevance score (between 0.0 and 1.0)      \n",
    "    \"\"\"\n",
    "    \n",
    "    relevance_score = 0.0\n",
    "        \n",
    "    for other_mention, other_cands in other_mentions:\n",
    "            \n",
    "        vote = 0.0\n",
    "            \n",
    "        for other_cand in other_cands:\n",
    "            rel = get_relatedness(entity,other_cand)\n",
    "            comm = get_commoness(other_mention,other_cand)\n",
    "            vote+= comm * rel\n",
    "\n",
    "        avg_vote = vote/len(other_cands)\n",
    "        relevance_score+=avg_vote\n",
    "    return relevance_score\n",
    "\n",
    "mention = \"Hillary\"\n",
    "                \n",
    "candidates = mention_to_entities[mention]\n",
    "\n",
    "other_mentions = [(other_m, mention_to_entities[other_m]) for other_m in mentions if other_m != mention]\n",
    "\n",
    "get_relevance_score(\"Hillary%20Clinton\",other_mentions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Hillary', 'Hillary%20Clinton')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_best_match(mention,candidates,other_mentions,thr= 0.03):\n",
    "    \"\"\"Retrieve best candidate match, given a mention in context \n",
    "    \n",
    "    Args:\n",
    "        mention: a mention of an entity.\n",
    "        candidates: a list of entity candidates for the mention.\n",
    "        other_mentions: list of tuples (mention,[list of candidate entities for that mention])\n",
    "        thr: optional variable, pruning for very uncommon candidate-mention pairs (default is 0.02)\n",
    "\n",
    "    Returns:\n",
    "        The best entity candidate for the mention      \n",
    "    \"\"\"    \n",
    "    candidates = [cand for cand in candidates if get_commoness(mention,cand)>thr]\n",
    "    \n",
    "    cand_scores = []\n",
    "    \n",
    "    for cand in candidates:\n",
    "        relevance_score = get_relevance_score(cand,other_mentions)\n",
    "        cand_scores.append([cand,relevance_score])\n",
    "    \n",
    "    cand_scores.sort(key=lambda x: x[1],reverse=True)\n",
    "    best_entity = cand_scores[0][0]\n",
    "    return (mention, best_entity)\n",
    "\n",
    "get_best_match(mention,candidates,other_mentions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"Figures/hillary.png\" height=\"30%\" width=\"30%\" align=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"Figures/hillary_aspects.png\" height=\"100%\" width=\"100%\" align=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "F. Nanni, S.P. Ponzetto and L. Dietz, \"Entity-Aspect Linking\", JCDL 2018, \"Vannevar Bush\" Best Paper Award\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Obama won the 2008 Iowa caucuses while <a href=\"https://en.wikipedia.org/wiki/Hillary%20Clinton#2008_presidential_campaign\">Hillary</a> was leading in the polls."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import json\n",
    "from scripts import EAL\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "mention , entity = get_best_match(mention,candidates,other_mentions)\n",
    "\n",
    "with open(\"Aspects/\"+entity+\".json\") as json_file:   \n",
    "    aspects = json.load(json_file)\n",
    "    aspect = EAL.rank(sentence,aspects)[0][0]\n",
    "    output = output_helpers.generate_wikilink(sentence,mention,entity,aspect)\n",
    "    display(HTML(output))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}