aTextUsageExamples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis of Text in Python and Java"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analytsis of Text (aText) is a Natural Language Processing (NLP) package developed over many years using machine and deep learning and symbolic AI techniques. The full aText package implements the most fundamental aspects of deep linguistics processing as listed below. Many real-world applications have been written using the functionalities of aText, including legal contract verification and documents classification, investors sentiment analysis, call center chat sentiment analysis with embedded acoustic features, semantic search for accurate documents retrieval, translation of natural language query to SQL, image captioning, and extraction of information about competitive asset managers. Patents on several proprietary algorithms implemented in the package are pending.\n",
    "\n",
    "Machine and deep learning:\n",
    "- supervised document classifications (NBC, kNBC, SVM, Fisher Kernel)\n",
    "- unsupervised classifications and topic modeling (LSA, pLSA, LDA)\n",
    "- deep learning models based on RNN/LSTM and word2vec\n",
    "\n",
    "Linguistics processing:\n",
    "- parsing (tokenization, stemming, POS tagging)\n",
    "- named-entity extraction and co-reference resolution\n",
    "- subject-verb-object triple extraction\n",
    "\n",
    "Web and social media:\n",
    "- sentiment and social network analyses\n",
    "- text summarization at the sentence and clauses levels\n",
    "- web-scraping (e.g. Amazon and TripAdvisor) of reviews\n",
    "- social media (e.g. Facebook, Twitter) and email analyses\n",
    "\n",
    "The original package has been written in platform-independent Java with a graphical user interface. For developers, aText functionalities have been made accessible via APIs to the jar file. A Python wrapper has been added to expose most of the functions for data scientists and engineers. The code in the notebook file aTextUsageExample.ipynb provides the following example basic usage of the package aText - basic sentiment analysis, document classification via Naive Bayesian Classifier (NBC), and sentence-level summarization.\n",
    "\n",
    "INSTALLATION & RUNNING EXAMPLES\n",
    "\n",
    "- Execute \"pip install atext\"\n",
    "- Requirements: Anaconda latest with Python 3.7, Java 1.8, packages py4j and requests, 250MB of space in the Python package directory.\n",
    "- Download data files from the githb repository: https://github.com/sjskdas/aText\n",
    "- Start Jupyter Notebook from the directory and run examples in aTextUsageExamples.ipynb\n",
    "\n",
    "POSSIBLE ISSUES:\n",
    "- If an invalid or corrupt jar file error occurs then download the jar file from this location and then copy to the aText directory under python site-packages: https://drive.google.com/open?id=1Ajhdrxw3n-I6OE1nODdp6W0DbuW6aImb\n",
    "- If a permission error occurs when opening the example notebook file then start jupyter notebook as an administrator.\n",
    "\n",
    "For any further query or to obtain the other Python functions or the full Java version of the package, please contact at atext@machineanalytics.com. For details of the popular algorithms in aText, please consult the book Computational Business Analytics by Dr. Subrata Das, published by Chapman & Hall/CRC Press, 2014."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sjskd\\Anaconda3\\python.exe\n",
      "C:\\Users\\sjskd\\Anaconda3\n"
     ]
    }
   ],
   "source": [
    "# pip install atext\n",
    "import aText\n",
    "\n",
    "import os.path\n",
    "import sys\n",
    "print(sys.executable)\n",
    "python_dir, python_exe = os.path.split(sys.executable)\n",
    "print(python_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# kill the spawned JVM process once done with aText, else it will continue to running in the background\n",
    "aText.stop_atext()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aText started\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'This is a test!'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first usage of aText takes time to download some necessary files\n",
    "# make sure to have write permission for the Lib/site-package directory\n",
    "aText.start_atext()\n",
    "aText.test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from py4j.java_gateway import JavaGateway\n",
    "gateway = JavaGateway()                 # connect to the JV\n",
    "misc_app = gateway.entry_point          # get the an application instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9 9\n",
      "18\n"
     ]
    }
   ],
   "source": [
    "# addition routine to test that the JVM is spawned\n",
    "random = gateway.jvm.java.util.Random()   # create a java.util.Random instance\n",
    "number1 = random.nextInt(10)              # call the Random.nextInt method\n",
    "number2 = random.nextInt(10)\n",
    "print(number1, number2)\n",
    "value = misc_app.addition(number1, number2) # call the addition method\n",
    "print(value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Positive Negative\n",
      "0.90 0.10\n",
      "0.22 0.78\n"
     ]
    }
   ],
   "source": [
    "# sentiment expressed in reviews - positive or negative\n",
    "good_review = \"Wonderful for families - a find   We were lucky enough to learn about the Hotel Suisse from friends in our hometown and it was such a treat  We loved staying there  The location could not be better and the family room was huge  Our friends stayed at the Hassler during the smae time up the block and our room was 3 times the size  I loved the security the place had as well with only a few guests at a time and our kids loved Rome  The hotel is a little tricky as it is on the second floor of a mixed use building after several design shops  This hotel was the best value of all of the hotels we stayed in on our 3 week trip  I would not go back to Rome and not stay at the Suisse  \"\n",
    "topics = misc_app.sentiment_topics()\n",
    "print(*topics)\n",
    "prob = misc_app.sentiment(good_review) # call the sentiment method\n",
    "# print the probably distribution among sentiments\n",
    "print(\"{:0.2f} {:0.2f}\".format(*prob))\n",
    "bad_review = \"awful  had to move out   This was our third trip to Rome in the last year and we thought we would stay a bit nearer the centre of the city  The Forum is not a four star hotel by any stretch of the imagination  We were given a room on the fourth floor  which we thought would be good because of the view over the forum site  Unfortunately the air conditioning did not work  and this necessitated the windows being left open at night  The street noise and bar opposite made sleeping difficult  The room was cramped  the breakfast very sub-standard  The second night we were awoken at midnight by staff scraping tables across the restaurant floor until the early hours of the morning  despite several complaints to the hotel desk  We arranged to move hotel and the final straw was the presence of a large number of homeless people outside the hotel who used the area as a toilet and dosshouse  we saw several rats running around  Do yourselves a favour  look elsewhere\"\n",
    "prob = misc_app.sentiment(bad_review) # call the sentiment method\n",
    "print(\"{:0.2f} {:0.2f}\".format(*prob))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alcohol Facts:\n",
      "Beer # Whiskey # Wine\n",
      "Number of nodes: 56\n",
      "Probability distribution: 0.10 0.00 0.90 \n",
      "\n",
      "Sentiment in Reviews:\n",
      "Positive # Negative\n",
      "Number of nodes: 19\n",
      "Probability distribution: 0.01 0.99 \n",
      "Positive # Negative\n",
      "Number of nodes: 19\n",
      "Probability distribution: 1.00 0.00 \n",
      "\n",
      "Product Reviews:\n",
      "Panasonic Microwave # Frigidaire Air Conditioner # Hoover WindTunnel Upright # Samsung Galaxy Phone # Dell Inspiron Laptop # Sony Smart LED TV\n",
      "Number of nodes: 7\n",
      "Probability distribution: 0.03 0.01 0.00 0.00 0.03 0.92 \n",
      "Panasonic Microwave # Frigidaire Air Conditioner # Hoover WindTunnel Upright # Samsung Galaxy Phone # Dell Inspiron Laptop # Sony Smart LED TV\n",
      "Number of nodes: 7\n",
      "Probability distribution: 0.02 0.87 0.03 0.01 0.06 0.01 \n"
     ]
    }
   ],
   "source": [
    "# aText NBC prepares the data via tokenization, stop word elimination, and stemming.\n",
    "def print_nbc_result(input_text):\n",
    "    prob = misc_app.nbc_infer(input_text)\n",
    "    classes = misc_app.nbc_classes()\n",
    "    print(' # '.join(classes)) # print class variables\n",
    "    size = misc_app.nbc_size()\n",
    "    print('Number of nodes: {}'.format(size)) # print size of network\n",
    "    # print output class probability probability distribution\n",
    "    print('Probability distribution: ' + ('{:0.2f} '*len(prob)).format(*prob))\n",
    "  \n",
    "# NBC 3-class small classifier\n",
    "print(\"\\nAlcohol Facts:\")\n",
    "excelFile = 'AlcoholPython.xlsx'\n",
    "# call the nbc with the excel filename, sheet name, MI threshold\n",
    "# MI threshold can vary between 0 to 1 - higher value means lower number of nodes, 0.0 keeps all\n",
    "misc_app.NBC(excelFile, \"Training\", 0.0) \n",
    "input_text = \"Italy is the largest producer of wine\"\n",
    "print_nbc_result(input_text)\n",
    "\n",
    "# NBC 2-class classifier\n",
    "print(\"\\nSentiment in Reviews:\")\n",
    "excelFile = 'SentimentPython.xlsx'\n",
    "misc_app.NBC(excelFile, \"Training\", 0.02) \n",
    "input_text = \"Never again I have just come home from Rome and four nights at the Hotel Locarno The only good thing was its location and the breakfast I found the hotel terribly shabby and rundown The rooms I saw 2 were dark and tiny with small short and hard beds The closet was so small that the hangers did not fit and the clothes kept falling of The lighting was very poor we had to use a torch to read in bed There The staff at the reception desk was not very helpful and very patronizing There was nothing chic about this hotel\"\n",
    "print_nbc_result(input_text)\n",
    "input_text = \"Lovely Place It s a lovely hotel in a quiet area but also very close to the main-shopping street and a 10-15 minutes walk to the leaning tower Staff is extremely helpful friendly and always made us feel at home There are some german native speakers and all staff seem to have excellent english skills At lunch they provided us some simple but excellent tuscan plates with cheese bazil and prosciutto toscano and there are also some restaurants in the area around the hotel So don t worry about good italian food - The breakfast room is a particulary nice room with wall-paintings good atmosphere and a big variety of pastries We used this hotel also as base for our trips to florence and lucca as train goes frequently and costs not so much 5 Euros one trip In the evening we had some delighting cocktails on the terrace on the back of the hotel barman Domenyco was fantastic Thanks to all and we will surely come back to this place Tip Have a ice-cream at Piazza Garibaldi \"\n",
    "print_nbc_result(input_text)\n",
    "\n",
    "# NBC 6-class classifier\n",
    "print(\"\\nProduct Reviews:\")\n",
    "excelFile = 'ReviewsPython.xlsx'\n",
    "misc_app.NBC(excelFile, \"Training\", 0.2) \n",
    "input_text = \"I wrote a solid positive review and gave this thing four stars. Well, less than a month later, dead pixels are making a line right through the screen at a bit left of center. I never touched the actual TV after initial setup, it just sat on the stand and crapped out. I really, really wanted to like this. I mean, it's a Sony! But solid picture doesn't mean anything if it craps out within a month. Well, time to see if customer service is worth anything.\"\n",
    "print_nbc_result(input_text)\n",
    "input_text = \"This air conditioner works great. In 100 degree temps this little unit kept 14' by 20' room super cool. Only flaw would be that it doesn't auto shut off when desired temp is reached. For me that was cosmetic, other may want that feature.\"\n",
    "print_nbc_result(input_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TOPICS: John Smith\n",
      "\n",
      "SUMMARY: John Smith is a scientist.  He hired Subrata.  John did not break the pot.  \n",
      "\n",
      "TOPICS: Dow Chemical Co, ClearForest Corp, technology\n",
      "\n",
      "SUMMARY: In 2001, Dow Chemical Co.  merged with Union Carbide Corp., requiring the integration of 35,000 Union Carbide research and development reports into Dow’s document management system.  Dow partnered with ClearForest Corp., a commercial developer of text-driven business solutions, to help integrate the new combined document collection.  Using technology it had developed, ClearForest indexed the documents, identifying chemical substances, products, companies, and people for inclusion in the combined database.  Dow was able to add more than 80 years’ worth of Union Carbide research to its information management system and approximately 100,000 new chemical substances to its registry.  When the project was omplete, Dow estimated it had spent some $3 million less than it would have if it had used its own methods for indexing documents.  \n"
     ]
    }
   ],
   "source": [
    "# sentence-level extractive summarization method without any training requirment\n",
    "\n",
    "input_text = \"John Smith is a scientist. He hired Subrata. Subrata is a friend of Sam. John did not break the pot.\"\n",
    "# call the summary method with input text, percentage outpt, and number of extracted topics\n",
    "s = misc_app.summary(input_text, 75, 1) # keep 75% and extract one topic\n",
    "print(s)\n",
    "\n",
    "input_text = \" \\\n",
    "In 2001, Dow Chemical Co. merged with Union Carbide Corp., requiring the integration of 35,000 Union Carbide research and development reports into Dow’s document management system. Dow partnered with ClearForest Corp., a commercial developer of text-driven business solutions, to help integrate the new combined document collection. Using technology it had developed, ClearForest indexed the documents, identifying chemical substances, products, companies, and people for inclusion in the combined database. Dow was able to add more than 80 years’ worth of Union Carbide research to its information management system and approximately 100,000 new chemical substances to its registry. When the project was omplete, Dow estimated it had spent some $3 million less than it would have if it had used its own methods for indexing documents. Dow also estimated it had reduced the time it would have spent sorting documents by 50% and data errors by 10%–15%.\\\n",
    "This scenario is an example of how the world is changing when it comes to managing electronic information. In the future, books and magazines will be used only for special purposes, as electronic documents become the primary means of storing, accessing, and sorting written communication. As many fields become overwhelmed with information, it will become physically impossible for any individual to process all the information available on a particular topic. Massive amounts of data will reside in cyberspace, generating demand for text mining technology and solutions.\\\n",
    "Text mining has been defined as “the discovery by computer of new, previously unknown, information by automatically extracting information from different written resources”. The Dow-Union Carbide scenario reflects how text mining can be applied in practical business situations. Many other domains (such as health care, government, education, and manufacturing) can also benefit from text-mining tools. Here, we explore them to guide organizations looking for the most effective solutions.\\\n",
    "Text mining is similar to data mining, except it is designed to handle structured data from databases or XML files, working with unstructured or semistructured data sets (such as email, full-text documents, and HTML files). As a result, text mining is a much better solution for companies (such as Dow) where large volumes of diverse information must be merged and managed. To date, however, most research and development efforts have centered on data mining using structured data.\\\n",
    "Machine intelligence is a problem for text mining. Natural language has developed to help humans communicate with one another and record information. Computers are a long way from comprehending natural language. Humans are able to distinguish and apply linguistic patterns to text, overcoming obstacles (such as slang, spelling variations, and contextual meaning). Computers do not handle them easily. Meanwhile, although our language capabilities allow us to comprehend unstructured data, we lack the computer’s ability to process text in large volumes at high speeds. The key to text mining is creating technology that combines a human’s linguistic capabilities with the speed and accuracy of a computer. \\\n",
    "\"\n",
    "# call the summary method with input text, percentage outpt, and number of extracted topics\n",
    "s = misc_app.summary(input_text, 25, 3) # keep 25% and extract three topics\n",
    "print('\\n' + s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}