notebooks/DoctorBook.ipynb

{
 "metadata": {
  "name": "DoctorBook"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Dr. Book - Linking a Book to Open Data Sources"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Problem Statement:</h3>\n",
      "\n",
      "<p>For any book authored using the Docbook XML structure (http://docbook.org/tdg51/en/html/) we will extract people, places, topics and concepts to render a webpage linking these resources to Open Data resources such as Freebase (wikipedia), OpenStreetMap, and Flickr. We will use Pandas to do text analysis on the book to produce a word cloud and answer other questions about the text.</p>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Updates as of 4/16/2012</h3>\n",
      "<p>\n",
      "    <h4>What we are currently working on</h4>\n",
      "    <ul>\n",
      "        <li>NLTK-based entity extractor.</li>\n",
      "        <li>Write a function that fetches it's wikipedia url (if it has one) for each extracted term.</li>\n",
      "    </ul>\n",
      "    <h4>What we will get done this week.</h4>\n",
      "    <ul>\n",
      "        <li>Complete entity extraction.</li>\n",
      "        <li>Complete Wiki url scraping.</li>\n",
      "    </ul>\n",
      "    <h4>What are problems you need help with to take next steps on your project?</h4>\n",
      "    <ul>\n",
      "        <li>Not necessarily an issue, but we need to connect pandas DataFrames/Series to the front-end.</li>\n",
      "        <li>We also have to figure out what visualization works best.</li>\n",
      "        <li>Searching for other resources marked up in DocBook format.</li>\n",
      "    </ul>\n",
      "</p>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>What open data sets might you be working with?</h3>\n",
      "<p>\n",
      "    <ul>\n",
      "        <li>Base: eBook in structured Docbook XML</li>\n",
      "        <li>Linked Data: Wikipedia, DBPedia - API, Freebase, OpenMaps, Flickr</li>\n",
      "    </ul>\n",
      "</p>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Possible ties to Wikipedia (or any Wikimedia related projects)?</h3>\n",
      "<p>We will use Wikipedia, Freebase and OpenStreeMap</p>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>What are some concrete immediate first/next steps in your projects?</h3>\n",
      "<p>\n",
      "    <ul>\n",
      "        <li>Finding parts of text to link to wikipedia</li>\n",
      "        <li>Resolving text to wikipedia entries</li>\n",
      "    </ul>\n",
      "</p>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>What challenges do you anticipate running into?</h3>\n",
      "<p>\n",
      "<ul>\n",
      "    <li>Learning to access APIs</li>\n",
      "    <li>Data mashups</li>\n",
      "    <li>Front-end design / structure / d3 visualization</li>\n",
      "</ul>\n",
      "</p>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>What skills will a team need to develop to solve this problem?</h3>\n",
      "<p>Database management, visualization, statistical analysis, extensive JavaScript knowledge (backbone.js)</p>\n",
      "<p>\n",
      "    <ul>\n",
      "        <li>Application Structure:</li>\n",
      "    </ul>\n",
      "    <ul>\n",
      "        <li>Back-end:</li>\n",
      "\t\t<li>Database - MarkLogic, BaseX</li>\n",
      "\t\t<li>Python Flask WebServer</li>\n",
      "\t\t<li>Semantic analysis - Python NLTK (Natural Language Toolkit)</li>\n",
      "    </ul>\n",
      "    <ul>\n",
      "        <li>Front-end:</li>\n",
      "\t\t<li>Backbone.js (MVC)</li>\n",
      "\t\t<li>d3.js (visualization & interaction)</li>\n",
      "    </ul>\n",
      "</p>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}