examples/text/qa_information_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\";\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"; "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# QA-Based Information Extraction\n",
    "\n",
    "\n",
    "As of v0.28.x, **ktrain** now includes a “universal” information extractor, which uses a Question-Answering model to extract any information of interest from documents.\n",
    "\n",
    "Suppose you have a table (e.g., an Excel spreadsheet) that looks like the DataFrame below. (In this example, each document is a single sentence, but each row can potenially be an entire report with many paragraphs.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>There is a risk of Donald Trump running again in 2024.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>This risk was consistent across patients stratified by history of CVD, risk factors \\nbut no CVD, and neither CVD nor risk factors.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Risk factors associated with subsequent death include older age, hypertension, diabetes, \\nischemic heart disease, obesity and chronic lung disease; however, sometimes \\nthere are no obvious risk factors .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Three major risk factors for COVID-19 were sex (male), age (≥60), and severe pneumonia.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>His speciality is medical risk assessments, and he is 30 years old.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Results: A total of nine studies including 356 patients were included in this study, \\nthe mean age was 52.4 years and 221 (62.1%) were male.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                            Text\n",
       "0                                                                                                                            Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .\n",
       "1                                                                                                                                                         There is a risk of Donald Trump running again in 2024.\n",
       "2                                                                            This risk was consistent across patients stratified by history of CVD, risk factors \\nbut no CVD, and neither CVD nor risk factors.\n",
       "3  Risk factors associated with subsequent death include older age, hypertension, diabetes, \\nischemic heart disease, obesity and chronic lung disease; however, sometimes \\nthere are no obvious risk factors .\n",
       "4                                                                                                                        Three major risk factors for COVID-19 were sex (male), age (≥60), and severe pneumonia.\n",
       "5                                                                                                                                            His speciality is medical risk assessments, and he is 30 years old.\n",
       "6                                                                  Results: A total of nine studies including 356 patients were included in this study, \\nthe mean age was 52.4 years and 221 (62.1%) were male."
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [\n",
    "'Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .',\n",
    "'There is a risk of Donald Trump running again in 2024.',\n",
    "\"\"\"This risk was consistent across patients stratified by history of CVD, risk factors \n",
    "but no CVD, and neither CVD nor risk factors.\"\"\",\n",
    "\"\"\"Risk factors associated with subsequent death include older age, hypertension, diabetes, \n",
    "ischemic heart disease, obesity and chronic lung disease; however, sometimes \n",
    "there are no obvious risk factors .\"\"\",\n",
    "'Three major risk factors for COVID-19 were sex (male), age (≥60), and severe pneumonia.',\n",
    "'His speciality is medical risk assessments, and he is 30 years old.',\n",
    "\"\"\"Results: A total of nine studies including 356 patients were included in this study, \n",
    "the mean age was 52.4 years and 221 (62.1%) were male.\"\"\"]\n",
    "import pandas as pd\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "df = pd.DataFrame(data, columns=['Text'])\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's pretend your boss wants you to extract both the reported risk factors from each document and the sample sizes for the reported studies.  This can easily be accomplished with the `AnswerExtractor` in **ktrain**, a kind of universal information extractor based on a Question-Answering model. (Note that, by default, `AnswerExtractor` uses TensorFlow.  To use PyTorch, supply `framework='pt'` as a parameter.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "      <th>Risk Factors</th>\n",
       "      <th>Sample Size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .</td>\n",
       "      <td>sex, obesity, genetic factors and mechanical factors</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>There is a risk of Donald Trump running again in 2024.</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>This risk was consistent across patients stratified by history of CVD, risk factors \\nbut no CVD, and neither CVD nor risk factors.</td>\n",
       "      <td>neither CVD nor risk factors</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Risk factors associated with subsequent death include older age, hypertension, diabetes, \\nischemic heart disease, obesity and chronic lung disease; however, sometimes \\nthere are no obvious risk factors .</td>\n",
       "      <td>older age, hypertension, diabetes,  ischemic heart disease, obesity and chronic lung disease</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Three major risk factors for COVID-19 were sex (male), age (≥60), and severe pneumonia.</td>\n",
       "      <td>sex (male), age (≥60), and severe pneumonia</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>His speciality is medical risk assessments, and he is 30 years old.</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Results: A total of nine studies including 356 patients were included in this study, \\nthe mean age was 52.4 years and 221 (62.1%) were male.</td>\n",
       "      <td>None</td>\n",
       "      <td>356</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                            Text  \\\n",
       "0                                                                                                                            Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .   \n",
       "1                                                                                                                                                         There is a risk of Donald Trump running again in 2024.   \n",
       "2                                                                            This risk was consistent across patients stratified by history of CVD, risk factors \\nbut no CVD, and neither CVD nor risk factors.   \n",
       "3  Risk factors associated with subsequent death include older age, hypertension, diabetes, \\nischemic heart disease, obesity and chronic lung disease; however, sometimes \\nthere are no obvious risk factors .   \n",
       "4                                                                                                                        Three major risk factors for COVID-19 were sex (male), age (≥60), and severe pneumonia.   \n",
       "5                                                                                                                                            His speciality is medical risk assessments, and he is 30 years old.   \n",
       "6                                                                  Results: A total of nine studies including 356 patients were included in this study, \\nthe mean age was 52.4 years and 221 (62.1%) were male.   \n",
       "\n",
       "                                                                                   Risk Factors  \\\n",
       "0                                          sex, obesity, genetic factors and mechanical factors   \n",
       "1                                                                                          None   \n",
       "2                                                                  neither CVD nor risk factors   \n",
       "3  older age, hypertension, diabetes,  ischemic heart disease, obesity and chronic lung disease   \n",
       "4                                                   sex (male), age (≥60), and severe pneumonia   \n",
       "5                                                                                          None   \n",
       "6                                                                                          None   \n",
       "\n",
       "  Sample Size  \n",
       "0        None  \n",
       "1        None  \n",
       "2        None  \n",
       "3        None  \n",
       "4        None  \n",
       "5        None  \n",
       "6         356  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ktrain.text.qa import AnswerExtractor\n",
    "ae = AnswerExtractor()\n",
    "df = ae.extract(df.Text.values, df, [('What are the risk factors?', 'Risk Factors'), \n",
    "                                     ('How many individuals in sample?', 'Sample Size')])\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, all that's required is that you phrase the type information you want to extract as a question (e.g., *What are the risk factors?*) and provide a label (e.g., *Risk Factors*).  The above command will return a new DataFrame with additional columns containing the information of interest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Examples\n",
    "\n",
    "QA-based information extraction is surprisingly versatile.  Here, we use it to extract **URLs**, **dates**, and **amounts**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "      <th>Amount</th>\n",
       "      <th>URL</th>\n",
       "      <th>Date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Closing price for Square on October 8th was $238.57, for details - https://finance.yahoo.com</td>\n",
       "      <td>238.57</td>\n",
       "      <td>https://finance.yahoo.com</td>\n",
       "      <td>October 8th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The film \"The Many Saints of Newark\" was released on 10/01/2021.</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>10/01/2021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Release delayed until the 1st of October due to COVID-19</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>1st of October</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Price of Bitcoin fell to forty thousand dollars</td>\n",
       "      <td>forty thousand dollars</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Documentation can be found at: amaiya.github.io/causalnlp</td>\n",
       "      <td>None</td>\n",
       "      <td>amaiya.github.io/causalnlp</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                           Text  \\\n",
       "0  Closing price for Square on October 8th was $238.57, for details - https://finance.yahoo.com   \n",
       "1                              The film \"The Many Saints of Newark\" was released on 10/01/2021.   \n",
       "2                                      Release delayed until the 1st of October due to COVID-19   \n",
       "3                                               Price of Bitcoin fell to forty thousand dollars   \n",
       "4                                     Documentation can be found at: amaiya.github.io/causalnlp   \n",
       "\n",
       "                   Amount                         URL            Date  \n",
       "0                  238.57   https://finance.yahoo.com     October 8th  \n",
       "1                    None                        None      10/01/2021  \n",
       "2                    None                        None  1st of October  \n",
       "3  forty thousand dollars                        None            None  \n",
       "4                    None  amaiya.github.io/causalnlp            None  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [\"Closing price for Square on October 8th was $238.57, for details - https://finance.yahoo.com\",\n",
    "        \"\"\"The film \"The Many Saints of Newark\" was released on 10/01/2021.\"\"\",\n",
    "           \"Release delayed until the 1st of October due to COVID-19\",\n",
    "           \"Price of Bitcoin fell to forty thousand dollars\",\n",
    "           \"Documentation can be found at: amaiya.github.io/causalnlp\",\n",
    "]\n",
    "df = pd.DataFrame(data, columns=['Text'])\n",
    "df = ae.extract(df.Text.values, df, [('What is the amount?', 'Amount'),\n",
    "                                     ('What is the URL?', 'URL'), \n",
    "                                     ('What is the date?', 'Date')])\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our last example, let's extract universities from a sample of the 20 Newsgroup dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "      <th>University</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier)\\nSubject: Converting images to HP LaserJet III?\\nNntp-Posting-Host: hampton\\nOrganization: The City University\\nLines: 14\\n\\nDoes anyone know of a good way (standard PC application/PD utility) to\\nconvert tif/img/tga files into LaserJet III format.  We would also like to\\ndo the same, converting to HPGL (HP plotter) files.\\n\\nPlease email any response.\\n\\nIs this the correct group?\\n\\nThanks in advance.  Michael.\\n-- \\nMichael Collier (Programmer)                 The Computer Unit,\\nEmail: M.P.Collier@uk.ac.city                The City University,\\nTel: 071 477-8000 x3769                      London,\\nFax: 071 477-8565                            EC1V 0HB.\\n</td>\n",
       "      <td>The City University</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: ani@ms.uky.edu (Aniruddha B. Deglurkar)\\nSubject: help: Splitting a trimming region along a mesh \\nOrganization: University Of Kentucky, Dept. of Math Sciences\\nLines: 28\\n\\n\\n\\n\\tHi,\\n\\n\\tI have a problem, I hope some of the 'gurus' can help me solve.\\n\\n\\tBackground of the problem:\\n\\tI have a rectangular mesh in the uv domain, i.e  the mesh is a \\n\\tmapping of a 3d Bezier patch into 2d. The area in this domain\\n\\twhich is inside a trimming loop had to be rendered. The trimming\\n\\tloop is a set of 2d Bezier curve segments.\\n\\tFor the sake of notation: the mesh is made up of cells.\\n\\n\\tMy problem is this :\\n\\tThe trimming area has to be split up into individual smaller\\n\\tcells bounded by the trimming curve segments. If a cell\\n\\tis wholly inside the area...then it is output as a whole ,\\n\\telse it is trivially rejected. \\n\\n\\tDoes any body know how thiss can be done, or is there any algo. \\n\\tsomewhere for doing this.\\n\\n\\tAny help would be appreciated.\\n\\n\\tThanks, \\n\\tAni.\\n-- \\nTo get irritated is human, to stay cool, divine.\\n</td>\n",
       "      <td>University Of Kentucky</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: djohnson@cs.ucsd.edu (Darin Johnson)\\nSubject: Re: harrassed at work, could use some prayers\\nOrganization: =CSE Dept., U.C. San Diego\\nLines: 63\\n\\n(Well, I'll email also, but this may apply to other people, so\\nI'll post also.)\\n\\n&gt;I've been working at this company for eight years in various\\n&gt;engineering jobs.  I'm female.  Yesterday I counted and realized that\\n&gt;on seven different occasions I've been sexually harrassed at this\\n&gt;company.\\n\\n&gt;I dreaded coming back to work today.  What if my boss comes in to ask\\n&gt;me some kind of question...\\n\\nYour boss should be the person bring these problems to.  If he/she\\ndoes not seem to take any action, keep going up higher and higher.\\nSexual harrassment does not need to be tolerated, and it can be an\\nenormous emotional support to discuss this with someone and know that\\nthey are trying to do something about it.  If you feel you can not\\ndiscuss this with your boss, perhaps your company has a personnel\\ndepartment that can work for you while preserving your privacy.  Most\\ncompanies will want to deal with this problem because constant anxiety\\ndoes seriously affect how effectively employees do their jobs.\\n\\nIt is unclear from your letter if you have done this or not.  It is\\nnot inconceivable that management remains ignorant of employee\\nproblems/strife even after eight years (it's a miracle if they do\\nnotice).  Perhaps your manager did not bring to the attention of\\nhigher ups?  If the company indeed does seem to want to ignore the\\nentire problem, there may be a state agency willing to fight with\\nyou.  (check with a lawyer, a women's resource center, etc to find out)\\n\\nYou may also want to discuss this with your paster, priest, husband,\\netc.  That is, someone you know will not be judgemental and that is\\nsupportive, comforting, etc.  This will bring a lot of healing.\\n\\n&gt;So I returned at 11:25, only to find that ever single\\n&gt;person had already left for lunch.  They left at 11:15 or so.  No one\\n&gt;could be bothered to call me at the other building, even though my\\n&gt;number was posted.\\n\\nThis happens to a lot of people.  Honest.  I believe it may seem\\nto be due to gross insensitivity because of the feelings you are\\ngoing through.  People in offices tend to be more insensitive while\\nworking than they normally are (maybe it's the hustle or stress or...)\\nI've had this happen to me a lot, often because they didn't realize\\nmy car was broken, etc.  Then they will come back and wonder why I\\ndidn't want to go (this would tend to make me stop being angry at\\nbeing ignored and make me laugh).  Once, we went off without our\\nboss, who was paying for the lunch :-)\\n\\n&gt;For this\\n&gt;reason I hope good Mr. Moderator allows me this latest indulgence.\\n\\nWell, if you can't turn to the computer for support, what would\\nwe do?  (signs of the computer age :-)\\n\\nIn closing, please don't let the hateful actions of a single person\\nharm you.  They are doing it because they are still the playground\\nbully and enjoy seeing the hurt they cause.  And you should not\\naccept the opinions of an imbecile that you are worthless - much\\nwiser people hold you in great esteem.\\n-- \\nDarin Johnson\\ndjohnson@ucsd.edu\\n  - Luxury!  In MY day, we had to make do with 5 bytes of swap...\\n</td>\n",
       "      <td>U.C.San Diego</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: s0612596@let.rug.nl (M.M. Zwart)\\nSubject: catholic church poland\\nOrganization: Faculteit der Letteren, Rijksuniversiteit Groningen, NL\\nLines: 10\\n\\nHello,\\n\\nI'm writing a paper on the role of the catholic church in Poland after 1989. \\nCan anyone tell me more about this, or fill me in on recent books/articles(\\nin english, german or french). Most important for me is the role of the \\nchurch concerning the abortion-law, religious education at schools,\\nbirth-control and the relation church-state(government). Thanx,\\n\\n                                                 Masja,\\n\"M.M.Zwart\"&lt;s0612596@let.rug.nl&gt;\\n</td>\n",
       "      <td>Rijksuniversiteit Groningen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From: stanly@grok11.columbiasc.ncr.com (stanly)\\nSubject: Re: Elder Brother\\nOrganization: NCR Corp., Columbia SC\\nLines: 15\\n\\nIn article &lt;Apr.8.00.57.41.1993.28246@athos.rutgers.edu&gt; REXLEX@fnal.gov writes:\\n&gt;In article &lt;Apr.7.01.56.56.1993.22824@athos.rutgers.edu&gt; shrum@hpfcso.fc.hp.com\\n&gt;Matt. 22:9-14 'Go therefore to the main highways, and as many as you find\\n&gt;there, invite to the wedding feast.'...\\n\\n&gt;hmmmmmm.  Sounds like your theology and Christ's are at odds. Which one am I \\n&gt;to believe?\\n\\nIn this parable, Jesus tells the parable of the wedding feast. \"The kingdom\\nof heaven is like unto a certain king which made a marriage for his son\".\\nSo the wedding clothes were customary,  and \"given\" to those who \"chose\" to\\nattend.  This man \"refused\" to wear the clothes.  The wedding clothes are\\nequalivant to the \"clothes of righteousness\".  When Jesus died for our sins,\\nthose \"clothes\" were then provided.  Like that man, it is our decision to\\nput the clothes on.\\n</td>\n",
       "      <td>rutgers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>From: vbv@lor.eeap.cwru.edu (Virgilio (Dean) B. Velasco Jr.)\\nSubject: Re: The arrogance of Christians\\nOrganization: Case Western Reserve Univ. Cleveland, Ohio (USA)\\nLines: 28\\n\\nIn article &lt;Apr.22.00.56.15.1993.2073@geneva.rutgers.edu&gt; hayesstw@risc1.unisa.ac.za (Steve Hayes) writes:\\n\\n&gt;A similar analogy might be a medical doctor who believes that a blood \\n&gt;transfusion is necessary to save the life of a child whose parents are \\n&gt;Jehovah's Witnesses and so have conscientious objections to blood \\n&gt;transfusion. The doctor's efforts to persuade them to agree to a blood \\n&gt;transfusion could be perceived to be arrogant in precisely the same way as \\n&gt;Christians could be perceived to be arrogant.\\n\\n&gt;The truth or otherwise of the belief that a blood transfusion is necessary \\n&gt;to save the life of the child is irrelevant here. What matters is that the \\n&gt;doctor BELIEVES it to be true, and could be seen to be trying to foce his \\n&gt;beliefs on the parents, and this could well be perceived as arrogance.\\n\\nLet me carry that a step further.  Most doctors would not claim to be \\ninfallible.  Indeed, they would generally admit that they could conceivably\\nbe wrong, e.g. that in this case, a blood tranfusion might not turn out to \\nbe necessary after all.  However, the doctors would have enough confidence\\nand conviction to claim, out of genuine concern, that is IS necessary.  As\\nfallible human beings, they must acknowledge the possibility that they are\\nwrong.  However, they would also say that such doubts are not reasonable,\\nand stand by their convictions.\\n\\n-- \\nVirgilio \"Dean\" Velasco Jr, Department of Electrical Eng'g and Applied Physics \\n\\t CWRU graduate student, roboticist-in-training and Q wannabee\\n    \"Bullwinkle, that man's intimidating a referee!\"   |    My boss is a \\n   \"Not very well.  He doesn't look like one at all!\"  |  Jewish carpenter.\\n</td>\n",
       "      <td>Case Western Reserve Univ. Cleveland, Ohio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>From: jodfishe@silver.ucs.indiana.edu (joseph dale fisher)\\nSubject: Re: anger\\nOrganization: Indiana University\\nLines: 34\\n\\nIn article &lt;Apr.17.01.10.44.1993.2232@geneva.rutgers.edu&gt; news@cbnewsk.att.com writes:\\n&gt;&gt;Paul Conditt writes:\\n[insert deletion of Paul's and Aaron's discourse on anger, ref Galatians\\n5:19-20]\\n&gt;\\n&gt;I don't know why it is so obvious.  We are not speaking of acts of the \\n&gt;flesh.  We are just speaking of emotions.  Emotions are not of themselves\\n&gt;moral or immoral, good or bad.  Emotions just are.  The first step is\\n&gt;not to label his emotion as good or bad or to numb ourselves so that\\n&gt;we hide our true feelings, it is to accept ourselves as we are, as God\\n&gt;accepts us.  \\n\\nOh, but they definitely can be.  Please look at Colossians 3:5-10 and\\nEphesians 4:25-27.  Emotions can be controlled and God puts very strong\\nemphasis on self-control, otherwise, why would he have Paul write to\\nTimothy so much about making sure to teach self-control? \\n\\n[insert deletion of remainder of paragraph]\\n\\n&gt;\\n&gt;Re-think it, Aaron.  Don't be quick to judge.  He has forgiven those with\\n&gt;AIDS, he has dealt with and taken responsibility for his feelings and made\\n&gt;appropriate choices for action on such feelings.  He has not given in to\\n&gt;his anger.\\n\\nPlease, re-think and re-read for yourself, Joe.  Again, the issue is\\nself-control especially over feelings and actions, for our actions stem\\nfrom our feelings in many instances.  As for God giving in to his anger,\\nthat comes very soon.\\n\\n&gt;\\n&gt;Joe Moore\\n\\nJoe Fisher\\n</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>From: aldridge@netcom.com (Jacquelin Aldridge)\\nSubject: Re: Teenage acne\\nOrganization: NETCOM On-line Communication Services (408 241-9760 guest)\\nLines: 57\\n\\npchurch@swell.actrix.gen.nz (Pat Churchill) writes:\\n\\n\\n&gt;My 14-y-o son has the usual teenage spotty chin and greasy nose.  I\\n&gt;bought him Clearasil face wash and ointment.  I think that is probably\\n&gt;enough, along with the usual good diet.  However, he is on at me to\\n&gt;get some product called Dalacin T, which used to be a\\n&gt;doctor's-prescription only treatment but is not available over the\\n&gt;chemist's counter.  I have asked a couple of pharmacists who say\\n&gt;either his acne is not severe enough for Dalacin T, or that Clearasil\\n&gt;is OK.  I had the odd spots as a teenager, nothing serious.  His\\n&gt;father was the same, so I don't figure his acne is going to escalate\\n&gt;into something disfiguring.  But I know kids are senstitive about\\n&gt;their appearance.  I am wary because a neighbour's son had this wierd\\n&gt;malady that was eventually put down to an overdose of vitamin A from\\n&gt;acne treatment.  I want to help - but with appropriate treatment.\\n\\n&gt;My son also has some scaliness around the hairline on his scalp.  Sort\\n&gt;of teenage cradle cap.  Any pointers/advice on this?  We have tried a\\n&gt;couple of anti dandruff shampoos and some of these are inclined to\\n&gt;make the condition worse, not better.\\n\\n&gt;Shall I bury the kid till he's 21 :)\\n\\n:) No...I was one of the lucky ones. Very little acne as a teenager. I\\ndidn't have any luck with clearasil. Even though my skin gets oily it\\nreally only gets miserable pimples when it's dry. \\n\\nFrequent lukewarm water rinses on the face might help. Getting the scalp\\nthing under control might help (that could be as simple as submerging under\\nthe bathwater till it's softened and washing it out). Taking a one a day\\nvitamin/mineral might help. I've heard iodine causes trouble and that it  \\nis used in fast food restaurants to sterilize equipment which might be\\nwhere the belief that greasy foods cause acne came from. I notice grease \\non my face, not immediately removed will cause acne (even from eating\\nmeat).\\n\\nKeeping hair rinse, mousse, dip, and spray off the face will help. Warm\\nwater bath soaks or cloths on the face to soften the oil in the pores will\\nhelp prevent blackheads. Body oil is hydrophilic, loves water and it\\nsoftens and washes off when it has a chance. That's why hair goes limp with\\noilyness. \\n\\nBecoming convinced that the best thing to do with\\na whitehead is leave it alone will save him days of pimple misery. Any\\nprying of black or whiteheads can cause infections, the red spots of\\npimples. Usually a whitehead will break naturally in a day and there won't\\nbe an infection afterwards.\\n\\nTell him that it's normal to have some pimples but the cosmetic industry\\nmakes it's money off of selling people on the idea that they are an\\nincredible defect to be hidden at any cost (even that of causing more pimples). \\n\\n\\n-Jackie-\\n\\n\\n</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>From: geb@cs.pitt.edu (Gordon Banks)\\nSubject: Re: Blindsight\\nReply-To: geb@cs.pitt.edu (Gordon Banks)\\nOrganization: Univ. of Pittsburgh Computer Science\\nLines: 18\\n\\nIn article &lt;werner-240393161954@tol7mac15.soe.berkeley.edu&gt; werner@soe.berkeley.edu (John Werner) writes:\\n&gt;In article &lt;19213@pitt.UUCP&gt;, geb@cs.pitt.edu (Gordon Banks) wrote:\\n&gt;&gt; \\n&gt;&gt; Explain.  I thought there were 3 types of cones, equivalent to RGB.\\n&gt;\\n&gt;You're basically right, but I think there are just 2 types.  One is\\n&gt;sensitive to red and green, and the other is sensitive to blue and yellow. \\n&gt;This is why the two most common kinds of color-blindness are red-green and\\n&gt;blue-yellow.\\n&gt;\\n\\nYes, I remember that now.  Well, in that case, the cones are indeed\\ncolor sensitive, contrary to what the original respondent had claimed.\\n-- \\n----------------------------------------------------------------------------\\nGordon Banks  N3JXP      | \"Skepticism is the chastity of the intellect, and\\ngeb@cadre.dsl.pitt.edu   |  it is shameful to surrender it too soon.\" \\n----------------------------------------------------------------------------\\n</td>\n",
       "      <td>Univ. of Pittsburgh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>From: libman@hsc.usc.edu (Marlena Libman)\\nSubject: Need advice with doctor-patient relationship problem\\nOrganization: University of Southern California, Los Angeles, CA\\nLines: 64\\nNNTP-Posting-Host: hsc.usc.edu\\n\\nI need advice with a situation which occurred between me and a physican\\nwhich upset me.  I saw this doctor for a problem with recurring pain.\\nHe suggested medication and a course of treatment, and told me that I\\nneed to call him 7 days after I begin the medication so that he may\\nmonitor its effectiveness, as well as my general health.\\n\\nI did exactly as he asked, and made the call (reaching his secretary).\\nI explained to her that I was following up at the doctor's request,\\nand that I was worried because the pain episodes were becoming more\\nfrequent and the medication did not seem effective.\\n\\nThe doctor called me back, and his first words were, \"Whatever you want,\\nyou'd better make it quick.  I'm very busy and don't have time to chit-\\nchat with you!\"  I told him I was simply following his instructions to\\ncall on the 7th day to status him, and that I was feeling worse.  I \\nthen asked if perhaps there was a better time for us to talk when he\\nhad more time.  He responded, \"Just spit it out now because no time is\\na good time.\"  (Said in a raised voice.)  I started to feel upset and\\ntried to explain quickly what was going on with my condition but my\\nnervousness interfered with my choice of words and I kind of stuttered\\nand then said \"well, never mind\" and he said he'll talk to various\\ncolleagues about other medications and he'll call me some other time.\\n\\nThis doctor called me that evening and said because I didn't express\\nmyself well, he was confused about what I wanted.  At this point I\\nwas pretty upset and I told him (in an amazingly polite voice considering\\nhow angry I felt) that his earlier manner had hurt my feelings.  He told\\nme that he just doesn't have time to \"rap with patients\" and thought\\nthat was what I wanted.  I told him that to assume I was calling to\\n\"rap\" was insulting, and said again that I was just following through\\non his orders.  He responded that he resented the implication that he \\nfelt I was making that he was not interested in learning about what his\\npatients have to say about their condition status.  He then gave me\\nthis apology: \"I am sorry that there was a miscommunication and you\\nmistakenly thought I was insulting.  I am not trying to insult you\\nbut I am not that knowledgeable about pain, and I don't have a lot of\\ntime to deal with that.\"  He then told me to call him the next day\\nfor further instructions on how do deal with my pain and medication.\\n\\nI am still upset and have not yet called.\\n\\nMy questions: (1) Should I continue to have this doctor manage my care?\\n(2) Since I am in pain off and on, I realize that this may cause me to\\nbe more anxietous so am I perhaps over-reacting or overly sensitive?\\nIf this doctor refers me to his colleague who knows more about the type\\nof pain I have, he still wants me to status him on my condition but\\nnow I am afraid to call him.\\n\\n\\t\\t\\t--Marlena\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n</td>\n",
       "      <td>University of Southern California</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   Text  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    From: sd345@city.ac.uk (Michael Collier)\\nSubject: Converting images to HP LaserJet III?\\nNntp-Posting-Host: hampton\\nOrganization: The City University\\nLines: 14\\n\\nDoes anyone know of a good way (standard PC application/PD utility) to\\nconvert tif/img/tga files into LaserJet III format.  We would also like to\\ndo the same, converting to HPGL (HP plotter) files.\\n\\nPlease email any response.\\n\\nIs this the correct group?\\n\\nThanks in advance.  Michael.\\n-- \\nMichael Collier (Programmer)                 The Computer Unit,\\nEmail: M.P.Collier@uk.ac.city                The City University,\\nTel: 071 477-8000 x3769                      London,\\nFax: 071 477-8565                            EC1V 0HB.\\n   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       From: ani@ms.uky.edu (Aniruddha B. Deglurkar)\\nSubject: help: Splitting a trimming region along a mesh \\nOrganization: University Of Kentucky, Dept. of Math Sciences\\nLines: 28\\n\\n\\n\\n\\tHi,\\n\\n\\tI have a problem, I hope some of the 'gurus' can help me solve.\\n\\n\\tBackground of the problem:\\n\\tI have a rectangular mesh in the uv domain, i.e  the mesh is a \\n\\tmapping of a 3d Bezier patch into 2d. The area in this domain\\n\\twhich is inside a trimming loop had to be rendered. The trimming\\n\\tloop is a set of 2d Bezier curve segments.\\n\\tFor the sake of notation: the mesh is made up of cells.\\n\\n\\tMy problem is this :\\n\\tThe trimming area has to be split up into individual smaller\\n\\tcells bounded by the trimming curve segments. If a cell\\n\\tis wholly inside the area...then it is output as a whole ,\\n\\telse it is trivially rejected. \\n\\n\\tDoes any body know how thiss can be done, or is there any algo. \\n\\tsomewhere for doing this.\\n\\n\\tAny help would be appreciated.\\n\\n\\tThanks, \\n\\tAni.\\n-- \\nTo get irritated is human, to stay cool, divine.\\n   \n",
       "2  From: djohnson@cs.ucsd.edu (Darin Johnson)\\nSubject: Re: harrassed at work, could use some prayers\\nOrganization: =CSE Dept., U.C. San Diego\\nLines: 63\\n\\n(Well, I'll email also, but this may apply to other people, so\\nI'll post also.)\\n\\n>I've been working at this company for eight years in various\\n>engineering jobs.  I'm female.  Yesterday I counted and realized that\\n>on seven different occasions I've been sexually harrassed at this\\n>company.\\n\\n>I dreaded coming back to work today.  What if my boss comes in to ask\\n>me some kind of question...\\n\\nYour boss should be the person bring these problems to.  If he/she\\ndoes not seem to take any action, keep going up higher and higher.\\nSexual harrassment does not need to be tolerated, and it can be an\\nenormous emotional support to discuss this with someone and know that\\nthey are trying to do something about it.  If you feel you can not\\ndiscuss this with your boss, perhaps your company has a personnel\\ndepartment that can work for you while preserving your privacy.  Most\\ncompanies will want to deal with this problem because constant anxiety\\ndoes seriously affect how effectively employees do their jobs.\\n\\nIt is unclear from your letter if you have done this or not.  It is\\nnot inconceivable that management remains ignorant of employee\\nproblems/strife even after eight years (it's a miracle if they do\\nnotice).  Perhaps your manager did not bring to the attention of\\nhigher ups?  If the company indeed does seem to want to ignore the\\nentire problem, there may be a state agency willing to fight with\\nyou.  (check with a lawyer, a women's resource center, etc to find out)\\n\\nYou may also want to discuss this with your paster, priest, husband,\\netc.  That is, someone you know will not be judgemental and that is\\nsupportive, comforting, etc.  This will bring a lot of healing.\\n\\n>So I returned at 11:25, only to find that ever single\\n>person had already left for lunch.  They left at 11:15 or so.  No one\\n>could be bothered to call me at the other building, even though my\\n>number was posted.\\n\\nThis happens to a lot of people.  Honest.  I believe it may seem\\nto be due to gross insensitivity because of the feelings you are\\ngoing through.  People in offices tend to be more insensitive while\\nworking than they normally are (maybe it's the hustle or stress or...)\\nI've had this happen to me a lot, often because they didn't realize\\nmy car was broken, etc.  Then they will come back and wonder why I\\ndidn't want to go (this would tend to make me stop being angry at\\nbeing ignored and make me laugh).  Once, we went off without our\\nboss, who was paying for the lunch :-)\\n\\n>For this\\n>reason I hope good Mr. Moderator allows me this latest indulgence.\\n\\nWell, if you can't turn to the computer for support, what would\\nwe do?  (signs of the computer age :-)\\n\\nIn closing, please don't let the hateful actions of a single person\\nharm you.  They are doing it because they are still the playground\\nbully and enjoy seeing the hurt they cause.  And you should not\\naccept the opinions of an imbecile that you are worthless - much\\nwiser people hold you in great esteem.\\n-- \\nDarin Johnson\\ndjohnson@ucsd.edu\\n  - Luxury!  In MY day, we had to make do with 5 bytes of swap...\\n   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      From: s0612596@let.rug.nl (M.M. Zwart)\\nSubject: catholic church poland\\nOrganization: Faculteit der Letteren, Rijksuniversiteit Groningen, NL\\nLines: 10\\n\\nHello,\\n\\nI'm writing a paper on the role of the catholic church in Poland after 1989. \\nCan anyone tell me more about this, or fill me in on recent books/articles(\\nin english, german or french). Most important for me is the role of the \\nchurch concerning the abortion-law, religious education at schools,\\nbirth-control and the relation church-state(government). Thanx,\\n\\n                                                 Masja,\\n\"M.M.Zwart\"<s0612596@let.rug.nl>\\n   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            From: stanly@grok11.columbiasc.ncr.com (stanly)\\nSubject: Re: Elder Brother\\nOrganization: NCR Corp., Columbia SC\\nLines: 15\\n\\nIn article <Apr.8.00.57.41.1993.28246@athos.rutgers.edu> REXLEX@fnal.gov writes:\\n>In article <Apr.7.01.56.56.1993.22824@athos.rutgers.edu> shrum@hpfcso.fc.hp.com\\n>Matt. 22:9-14 'Go therefore to the main highways, and as many as you find\\n>there, invite to the wedding feast.'...\\n\\n>hmmmmmm.  Sounds like your theology and Christ's are at odds. Which one am I \\n>to believe?\\n\\nIn this parable, Jesus tells the parable of the wedding feast. \"The kingdom\\nof heaven is like unto a certain king which made a marriage for his son\".\\nSo the wedding clothes were customary,  and \"given\" to those who \"chose\" to\\nattend.  This man \"refused\" to wear the clothes.  The wedding clothes are\\nequalivant to the \"clothes of righteousness\".  When Jesus died for our sins,\\nthose \"clothes\" were then provided.  Like that man, it is our decision to\\nput the clothes on.\\n   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          From: vbv@lor.eeap.cwru.edu (Virgilio (Dean) B. Velasco Jr.)\\nSubject: Re: The arrogance of Christians\\nOrganization: Case Western Reserve Univ. Cleveland, Ohio (USA)\\nLines: 28\\n\\nIn article <Apr.22.00.56.15.1993.2073@geneva.rutgers.edu> hayesstw@risc1.unisa.ac.za (Steve Hayes) writes:\\n\\n>A similar analogy might be a medical doctor who believes that a blood \\n>transfusion is necessary to save the life of a child whose parents are \\n>Jehovah's Witnesses and so have conscientious objections to blood \\n>transfusion. The doctor's efforts to persuade them to agree to a blood \\n>transfusion could be perceived to be arrogant in precisely the same way as \\n>Christians could be perceived to be arrogant.\\n\\n>The truth or otherwise of the belief that a blood transfusion is necessary \\n>to save the life of the child is irrelevant here. What matters is that the \\n>doctor BELIEVES it to be true, and could be seen to be trying to foce his \\n>beliefs on the parents, and this could well be perceived as arrogance.\\n\\nLet me carry that a step further.  Most doctors would not claim to be \\ninfallible.  Indeed, they would generally admit that they could conceivably\\nbe wrong, e.g. that in this case, a blood tranfusion might not turn out to \\nbe necessary after all.  However, the doctors would have enough confidence\\nand conviction to claim, out of genuine concern, that is IS necessary.  As\\nfallible human beings, they must acknowledge the possibility that they are\\nwrong.  However, they would also say that such doubts are not reasonable,\\nand stand by their convictions.\\n\\n-- \\nVirgilio \"Dean\" Velasco Jr, Department of Electrical Eng'g and Applied Physics \\n\\t CWRU graduate student, roboticist-in-training and Q wannabee\\n    \"Bullwinkle, that man's intimidating a referee!\"   |    My boss is a \\n   \"Not very well.  He doesn't look like one at all!\"  |  Jewish carpenter.\\n   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         From: jodfishe@silver.ucs.indiana.edu (joseph dale fisher)\\nSubject: Re: anger\\nOrganization: Indiana University\\nLines: 34\\n\\nIn article <Apr.17.01.10.44.1993.2232@geneva.rutgers.edu> news@cbnewsk.att.com writes:\\n>>Paul Conditt writes:\\n[insert deletion of Paul's and Aaron's discourse on anger, ref Galatians\\n5:19-20]\\n>\\n>I don't know why it is so obvious.  We are not speaking of acts of the \\n>flesh.  We are just speaking of emotions.  Emotions are not of themselves\\n>moral or immoral, good or bad.  Emotions just are.  The first step is\\n>not to label his emotion as good or bad or to numb ourselves so that\\n>we hide our true feelings, it is to accept ourselves as we are, as God\\n>accepts us.  \\n\\nOh, but they definitely can be.  Please look at Colossians 3:5-10 and\\nEphesians 4:25-27.  Emotions can be controlled and God puts very strong\\nemphasis on self-control, otherwise, why would he have Paul write to\\nTimothy so much about making sure to teach self-control? \\n\\n[insert deletion of remainder of paragraph]\\n\\n>\\n>Re-think it, Aaron.  Don't be quick to judge.  He has forgiven those with\\n>AIDS, he has dealt with and taken responsibility for his feelings and made\\n>appropriate choices for action on such feelings.  He has not given in to\\n>his anger.\\n\\nPlease, re-think and re-read for yourself, Joe.  Again, the issue is\\nself-control especially over feelings and actions, for our actions stem\\nfrom our feelings in many instances.  As for God giving in to his anger,\\nthat comes very soon.\\n\\n>\\n>Joe Moore\\n\\nJoe Fisher\\n   \n",
       "7                                                                                                                                                                                                                                                                                       From: aldridge@netcom.com (Jacquelin Aldridge)\\nSubject: Re: Teenage acne\\nOrganization: NETCOM On-line Communication Services (408 241-9760 guest)\\nLines: 57\\n\\npchurch@swell.actrix.gen.nz (Pat Churchill) writes:\\n\\n\\n>My 14-y-o son has the usual teenage spotty chin and greasy nose.  I\\n>bought him Clearasil face wash and ointment.  I think that is probably\\n>enough, along with the usual good diet.  However, he is on at me to\\n>get some product called Dalacin T, which used to be a\\n>doctor's-prescription only treatment but is not available over the\\n>chemist's counter.  I have asked a couple of pharmacists who say\\n>either his acne is not severe enough for Dalacin T, or that Clearasil\\n>is OK.  I had the odd spots as a teenager, nothing serious.  His\\n>father was the same, so I don't figure his acne is going to escalate\\n>into something disfiguring.  But I know kids are senstitive about\\n>their appearance.  I am wary because a neighbour's son had this wierd\\n>malady that was eventually put down to an overdose of vitamin A from\\n>acne treatment.  I want to help - but with appropriate treatment.\\n\\n>My son also has some scaliness around the hairline on his scalp.  Sort\\n>of teenage cradle cap.  Any pointers/advice on this?  We have tried a\\n>couple of anti dandruff shampoos and some of these are inclined to\\n>make the condition worse, not better.\\n\\n>Shall I bury the kid till he's 21 :)\\n\\n:) No...I was one of the lucky ones. Very little acne as a teenager. I\\ndidn't have any luck with clearasil. Even though my skin gets oily it\\nreally only gets miserable pimples when it's dry. \\n\\nFrequent lukewarm water rinses on the face might help. Getting the scalp\\nthing under control might help (that could be as simple as submerging under\\nthe bathwater till it's softened and washing it out). Taking a one a day\\nvitamin/mineral might help. I've heard iodine causes trouble and that it  \\nis used in fast food restaurants to sterilize equipment which might be\\nwhere the belief that greasy foods cause acne came from. I notice grease \\non my face, not immediately removed will cause acne (even from eating\\nmeat).\\n\\nKeeping hair rinse, mousse, dip, and spray off the face will help. Warm\\nwater bath soaks or cloths on the face to soften the oil in the pores will\\nhelp prevent blackheads. Body oil is hydrophilic, loves water and it\\nsoftens and washes off when it has a chance. That's why hair goes limp with\\noilyness. \\n\\nBecoming convinced that the best thing to do with\\na whitehead is leave it alone will save him days of pimple misery. Any\\nprying of black or whiteheads can cause infections, the red spots of\\npimples. Usually a whitehead will break naturally in a day and there won't\\nbe an infection afterwards.\\n\\nTell him that it's normal to have some pimples but the cosmetic industry\\nmakes it's money off of selling people on the idea that they are an\\nincredible defect to be hidden at any cost (even that of causing more pimples). \\n\\n\\n-Jackie-\\n\\n\\n   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  From: geb@cs.pitt.edu (Gordon Banks)\\nSubject: Re: Blindsight\\nReply-To: geb@cs.pitt.edu (Gordon Banks)\\nOrganization: Univ. of Pittsburgh Computer Science\\nLines: 18\\n\\nIn article <werner-240393161954@tol7mac15.soe.berkeley.edu> werner@soe.berkeley.edu (John Werner) writes:\\n>In article <19213@pitt.UUCP>, geb@cs.pitt.edu (Gordon Banks) wrote:\\n>> \\n>> Explain.  I thought there were 3 types of cones, equivalent to RGB.\\n>\\n>You're basically right, but I think there are just 2 types.  One is\\n>sensitive to red and green, and the other is sensitive to blue and yellow. \\n>This is why the two most common kinds of color-blindness are red-green and\\n>blue-yellow.\\n>\\n\\nYes, I remember that now.  Well, in that case, the cones are indeed\\ncolor sensitive, contrary to what the original respondent had claimed.\\n-- \\n----------------------------------------------------------------------------\\nGordon Banks  N3JXP      | \"Skepticism is the chastity of the intellect, and\\ngeb@cadre.dsl.pitt.edu   |  it is shameful to surrender it too soon.\" \\n----------------------------------------------------------------------------\\n   \n",
       "9                                                                                                                        From: libman@hsc.usc.edu (Marlena Libman)\\nSubject: Need advice with doctor-patient relationship problem\\nOrganization: University of Southern California, Los Angeles, CA\\nLines: 64\\nNNTP-Posting-Host: hsc.usc.edu\\n\\nI need advice with a situation which occurred between me and a physican\\nwhich upset me.  I saw this doctor for a problem with recurring pain.\\nHe suggested medication and a course of treatment, and told me that I\\nneed to call him 7 days after I begin the medication so that he may\\nmonitor its effectiveness, as well as my general health.\\n\\nI did exactly as he asked, and made the call (reaching his secretary).\\nI explained to her that I was following up at the doctor's request,\\nand that I was worried because the pain episodes were becoming more\\nfrequent and the medication did not seem effective.\\n\\nThe doctor called me back, and his first words were, \"Whatever you want,\\nyou'd better make it quick.  I'm very busy and don't have time to chit-\\nchat with you!\"  I told him I was simply following his instructions to\\ncall on the 7th day to status him, and that I was feeling worse.  I \\nthen asked if perhaps there was a better time for us to talk when he\\nhad more time.  He responded, \"Just spit it out now because no time is\\na good time.\"  (Said in a raised voice.)  I started to feel upset and\\ntried to explain quickly what was going on with my condition but my\\nnervousness interfered with my choice of words and I kind of stuttered\\nand then said \"well, never mind\" and he said he'll talk to various\\ncolleagues about other medications and he'll call me some other time.\\n\\nThis doctor called me that evening and said because I didn't express\\nmyself well, he was confused about what I wanted.  At this point I\\nwas pretty upset and I told him (in an amazingly polite voice considering\\nhow angry I felt) that his earlier manner had hurt my feelings.  He told\\nme that he just doesn't have time to \"rap with patients\" and thought\\nthat was what I wanted.  I told him that to assume I was calling to\\n\"rap\" was insulting, and said again that I was just following through\\non his orders.  He responded that he resented the implication that he \\nfelt I was making that he was not interested in learning about what his\\npatients have to say about their condition status.  He then gave me\\nthis apology: \"I am sorry that there was a miscommunication and you\\nmistakenly thought I was insulting.  I am not trying to insult you\\nbut I am not that knowledgeable about pain, and I don't have a lot of\\ntime to deal with that.\"  He then told me to call him the next day\\nfor further instructions on how do deal with my pain and medication.\\n\\nI am still upset and have not yet called.\\n\\nMy questions: (1) Should I continue to have this doctor manage my care?\\n(2) Since I am in pain off and on, I realize that this may cause me to\\nbe more anxietous so am I perhaps over-reacting or overly sensitive?\\nIf this doctor refers me to his colleague who knows more about the type\\nof pain I have, he still wants me to status him on my condition but\\nnow I am afraid to call him.\\n\\n\\t\\t\\t--Marlena\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n   \n",
       "\n",
       "                                   University  \n",
       "0                         The City University  \n",
       "1                      University Of Kentucky  \n",
       "2                               U.C.San Diego  \n",
       "3                 Rijksuniversiteit Groningen  \n",
       "4                                     rutgers  \n",
       "5  Case Western Reserve Univ. Cleveland, Ohio  \n",
       "6                                        None  \n",
       "7                                        None  \n",
       "8                         Univ. of Pittsburgh  \n",
       "9           University of Southern California  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load text data\n",
    "categories = ['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med']\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "train_b = fetch_20newsgroups(subset='train', categories=categories, shuffle=True)\n",
    "df = pd.DataFrame(train_b.data[:10], columns=['Text']) # let's examine the first 10 posts\n",
    "df = ae.extract(df.Text.values, df, [('What is the university?', 'University')])\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Customizing the `AnswerExtractor` to Your Use Case\n",
    "\n",
    "If there are false positives (or false negatives), you can adjust the `min_conf` parameter (i.e., minimum confidence threshold) until you’re happy (default is `min_conf=6`).  If `return_conf=True`, then columns showing the confidence scores of each extraction will also be included in the resultant DataFrame.\n",
    "\n",
    "If adjusting the confidence threshold is not sufficient to address the false positives and false negatives you're seeing, you can also try fine-tuning the QA model to your custom dataset by providing only a small handful examples:\n",
    "\n",
    "**Example:**\n",
    "```python\n",
    "data = [\n",
    "{\"question\": \"What is the URL?\", \n",
    "\"context\": \"Closing price for Square on October 8th was $238.57, for details - https://finance.yahoo.com\", \n",
    " \"answers\": \"https://finance.yahoo.com\"},\n",
    " {\"question\": \"What is the URL?\", \n",
    "  \"context\": \"HTTP is a protocol for fetching resources.\", \n",
    "  \"answers\": None}, \n",
    "]\n",
    "from ktrain.text.qa import AnswerExtractor\n",
    "ae = AnswerExtractor(bert_squad_model='distilbert-base-cased-distilled-squad')\n",
    "ae.finetune(data)\n",
    "```\n",
    "\n",
    "Note that, by default, the `AnswerExtractor` uses a `bert-large-*` model that requires a lot of memory to train.  If fine-tuning, you may want to switch to a smaller model like DistilBERT, as shown in the example above.\n",
    "\n",
    "Finally, the `finetune` method accepts other parameters such as `batch_size` and `max_seq_length` that you can adjust depending on your speed requirements, dataset characteristics, and system resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}