Minreng Wu_Lucene and Elasticsearch_Feb27.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basics of Lucene and Elasticsearch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lucene\n",
    "\n",
    "Lucene is a high-performance, scalable information retrieval (IR) library. Lucene was originally written completely in Java by [Doug Cutting](https://en.wikipedia.org/wiki/Doug_Cutting) and can let users add searching capabilities to their applications easily. \n",
    "\n",
    "Lucene has been ported to other programming languages including Object Pascal, Perl, C#, C++, Python, Ruby and PHP, so currently it's not only used in Java programs but also the other programming languages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How it works?\n",
    "\n",
    "The architecture of Lucene is very similar to the abstract IR architecture Prof. Caverlee discussed in the class:\n",
    "\n",
    "![architecture_of_lucene](files/pics/architecture_of_lucene.png)\n",
    "\n",
    "It can be divided into two parts: **indexing** and **searching**.\n",
    "\n",
    "**Indexing**: gathering data (acquiring contents) => building documents => analyzing and indexing documents\n",
    "\n",
    "**Searching**: building query => searching query in the index library => rendering query results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index Structure\n",
    "\n",
    "In Lucene, we also use inverted index to index the terms in the documents. But it's quite different. See the structure of the inverted index in lucene as follows:\n",
    "\n",
    "```\n",
    "|+ field1(name,type)\n",
    "    |+ term1\n",
    "        |+ doc1\n",
    "          \t|+ termFreq = 2\n",
    "                |+ [position1,offset1,payload1]\n",
    "                |+ [position2,offset2,payload2]\n",
    "        |+ doc2\n",
    "          \t|+ termFreq = 1\n",
    "            \t\t|+ [position3,offset3,payload3]\n",
    "        |+...\n",
    "    |+ term2\n",
    "    |+...\n",
    "|+ field2(name,type)\n",
    "|+ ...\n",
    "```\n",
    "\n",
    "We can see that there're lots of modifications made on the index structure of Lucene compared to the original inverted index version. \n",
    "\n",
    "*   Term: a single term after tokenized and stemmed\n",
    "*   Field: the collection of Terms\n",
    "*   Document: the collection of Fields\n",
    "\n",
    "-   DocIDList: the list of the document IDs\n",
    "-   TermFreq: term frequency\n",
    "-   Position: the position information about where a term occurs in the index (after tokenized)\n",
    "-   Offset: the position information about where a term occurs in the original doc \n",
    "-   Payload: additional per-position metadata information such as character offsets and user payloads (e.g. improve the score of a specific term)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to use Lucene in Python?\n",
    "\n",
    "There's a Python version library [PyLucene](http://lucene.apache.org/pylucene/index.html) for accessing Lucene. Here's a [installation instructions](http://lucene.apache.org/pylucene/install.html) introducing how to install PyLucene in your PC.\n",
    "\n",
    "For convenience, I choose to use docker to run PyLucene on my PC, and below is the docker image I pulled from the docker hub:\n",
    "\n",
    "```bash\n",
    "docker pull coady/pylucene\n",
    "```\n",
    "\n",
    "Then we can run and enter the docker container:\n",
    "\n",
    "```bash\n",
    "docker run -it coady/pylucene /bin/bash\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to use Lucene in Python?\n",
    "\n",
    "There's a Python version library [PyLucene](http://lucene.apache.org/pylucene/index.html) for accessing Lucene. Here's a [installation instructions](http://lucene.apache.org/pylucene/install.html) introducing how to install PyLucene in your PC.\n",
    "\n",
    "For convenience, I choose to use docker to run PyLucene on my PC, and below is the docker image I pulled from the docker hub:\n",
    "\n",
    "```bash\n",
    "docker pull coady/pylucene\n",
    "```\n",
    "\n",
    "Then we can run and enter the docker container:\n",
    "\n",
    "```bash\n",
    "docker run -it coady/pylucene /bin/bash\n",
    "```\n",
    "\n",
    "And I first collect 10 famous quotes from the Internet and save them as 10 single documents respectively. In each document, the content is some text like below:\n",
    "\n",
    ">   \"Few things in the world are more powerful than a positive push. A smile. A world of optimism and hope. A 'you can do it' when things are tough.\" ― Richard M. DeVos\n",
    "\n",
    "Then I will use PyLucene to index all these 10 documents and then search the documents by some queries using PyLucene. Let's see how we can get it.\n",
    "\n",
    "First, we need to import the required libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os, lucene, threading, time\n",
    "from datetime import datetime\n",
    "import glob\n",
    "from java.io import File\n",
    "from java.nio.file import Paths\n",
    "from org.apache.lucene.analysis.miscellaneous import LimitTokenCountAnalyzer\n",
    "from org.apache.lucene.analysis.core import WhitespaceAnalyzer\n",
    "from org.apache.lucene.analysis.standard import StandardAnalyzer\n",
    "from org.apache.lucene.document import Document, Field, FieldType, TextField\n",
    "from org.apache.lucene.index import FieldInfo, IndexWriter, IndexWriterConfig, IndexReader, DirectoryReader, IndexOptions, Term\n",
    "from org.apache.lucene.store import SimpleFSDirectory\n",
    "from org.apache.lucene.util import Version\n",
    "from org.apache.lucene.analysis.standard import StandardAnalyzer\n",
    "from org.apache.lucene.search import IndexSearcher, TermQuery\n",
    "from org.apache.lucene.queryparser.classic import QueryParser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we should initialize the Java VM for running Lucene:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<jcc.JCCEnv at 0x7faffc3d92b0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# initialize Java VM\n",
    "lucene.initVM()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later, we can index the documents in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Indexing:  /usr/src/doc/sample6.txt\n",
      "Done:  /usr/src/doc/sample6.txt\n",
      "Indexing:  /usr/src/doc/sample8.txt\n",
      "Done:  /usr/src/doc/sample8.txt\n",
      "Indexing:  /usr/src/doc/sample4.txt\n",
      "Done:  /usr/src/doc/sample4.txt\n",
      "Indexing:  /usr/src/doc/sample1.txt\n",
      "Done:  /usr/src/doc/sample1.txt\n",
      "Indexing:  /usr/src/doc/sample9.txt\n",
      "Done:  /usr/src/doc/sample9.txt\n",
      "Indexing:  /usr/src/doc/sample5.txt\n",
      "Done:  /usr/src/doc/sample5.txt\n",
      "Indexing:  /usr/src/doc/sample10.txt\n",
      "Done:  /usr/src/doc/sample10.txt\n",
      "Indexing:  /usr/src/doc/sample3.txt\n",
      "Done:  /usr/src/doc/sample3.txt\n",
      "Indexing:  /usr/src/doc/sample7.txt\n",
      "Done:  /usr/src/doc/sample7.txt\n",
      "Indexing:  /usr/src/doc/sample2.txt\n",
      "Done:  /usr/src/doc/sample2.txt\n",
      "\n",
      "Total processed: 10\n"
     ]
    }
   ],
   "source": [
    "# define the document(input) directory path and the output index directory path\n",
    "docdir = \"/usr/src/doc\"\n",
    "indir = \"/usr/src/index\"\n",
    "DIRTOINDEX = docdir\n",
    "INDEXIDR = Paths.get(indir)\n",
    "\n",
    "# initialize analyzer and index writer\n",
    "indexdir = SimpleFSDirectory(INDEXIDR)\n",
    "analyzer = StandardAnalyzer()\n",
    "config = IndexWriterConfig(analyzer)\n",
    "index_writer = IndexWriter(indexdir, config)\n",
    "\n",
    "# index the 10 documents in the input directory\n",
    "for tfile in glob.glob(os.path.join(DIRTOINDEX, '*.txt')):\n",
    "    print(\"Indexing: \", tfile)\n",
    "    document = Document()\n",
    "    content = open(tfile, 'r').read()\n",
    "    document.add(Field(\"text\", content, TextField.TYPE_STORED))\n",
    "    index_writer.addDocument(document)\n",
    "    print(\"Done: \", tfile)\n",
    "    \n",
    "# show total number of documents and commit the changes\n",
    "print(\"\\nTotal processed: %s\" % index_writer.numRamDocs())\n",
    "index_writer.commit()\n",
    "index_writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analyzer in Lucene is used to analyze text while indexing and searching documents (In other words, Analyzer is used to split the text into tokens). Here we use the `StandardAnalyzer` which will lowercase all words and remove all stopwords and punctuations. \n",
    "\n",
    "And here we specify a field called `text` as a `TextField` type which would be indexed, tokenized, stored respectively. (Compared to `StringField` type, which would only be indexed but not tokenized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we indexed the documents, we can search some docs in our index library by the queries. For example, we can query the word \"success\"  in our index library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify the index directory and create the IndexSeacher\n",
    "INDEXDIR = \"/usr/src/index\"\n",
    "indir = Paths.get(INDEXDIR)\n",
    "indir = SimpleFSDirectory(indir)\n",
    "ir = DirectoryReader.open(indir)\n",
    "lucene_searcher = IndexSearcher(ir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"text:success\"\n",
    "lucene_analyzer = StandardAnalyzer()\n",
    "parsed_query = QueryParser(query, lucene_analyzer).parse(query)\n",
    "hits = lucene_searcher.search(parsed_query, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 3 document(s) that matched query 'text:success':\n",
      "doc=1 score=0.63861465 shardIndex=0\n",
      "b'\"Success is just a never-ending process of trying to become better and better at whatever you want\" - John Traver\\n'\n",
      "doc=7 score=0.42715713 shardIndex=0\n",
      "b'\\xe2\\x80\\x9cEnthusiasm is one of the most powerful engines of success. When you do a thing, do it with all your might. Put your whole soul into it. Stamp it with your own personality. Be active, be energetic, be enthusiastic and faithful, and you will accomplish your object. Nothing great was ever achieved without enthusiasm.\\xe2\\x80\\x9d - Ralph Waldo Emerson\\n'\n",
      "doc=9 score=0.39788035 shardIndex=0\n",
      "b'\\xe2\\x80\\x9cDon\\xe2\\x80\\x99t wish it was easier, wish you were better. Don\\xe2\\x80\\x99t wish for less problems, wish for more skills. Don\\xe2\\x80\\x99t wish for less challenges, wish for more wisdom. The major value in life is not what you get. The major value in life is what you become. Success is not to be pursued; it is to be attracted by the person you become.\\xe2\\x80\\x9d - Jim Rohn\\n'\n"
     ]
    }
   ],
   "source": [
    "print(\"Found %d document(s) that matched query '%s':\" % (hits.totalHits.value, parsed_query))\n",
    "for hit in hits.scoreDocs:\n",
    "    print(hit.toString())\n",
    "    doc = lucene_searcher.doc(hit.doc)\n",
    "    print(doc.get(\"text\").encode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first specify the path of our index library and use `DirectoryReader` to create `IndexReader` and  `IndexSearcher`. Then we can form a query `text:success`: the first partition `text` represents the field name; the second partition `success` represents the content we query. `100` represents the maximum number of documents we would retrieve.\n",
    "\n",
    "\n",
    "\n",
    "The search result will be sorted by the BM25 score. we can also use the TF-IDF algorithm by manually setting  `indexSearcher.setSimilarity(ClassicSimilarity())`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Elasticsearch\n",
    "\n",
    "Elasticsearch is a distributed, RESTful search and analytics engine based on the [Lucene](https://en.wikipedia.org/wiki/Lucene) library. Since it packs up search engine operations into RESTful APIs, it's easy and friendly to the users who only know little about search engines. Compared to another full-text search engine [Solr](https://lucene.apache.org/solr/) which is also based on Lucene, it also implements distributed indexing and is the better option for cloud and distributed environments that need good scalability and performance. Besides, Elasticsearch also provides the ability of analyzing big volumes of data in near real-time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Concepts\n",
    "\n",
    "**Index**:  An index is a collection of documents that have somewhat similar characteristics. For example, you can have an index for customer data, another index for a product catalog, and yet another index for order data. An index is identified by a name (that must be all lowercase) and this name is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it.\n",
    "\n",
    "You can consider Index as the place where Elasticsearch stores data and compare it to the **database** concept in the traditional DBMS like MySQL.\n",
    "\n",
    "\n",
    "\n",
    "**Type**: A type is a logical category/partition of your index whose semantics is completely up to you. In general, a type is defined for documents that have a set of common fields. For example, let’s assume you run a blogging platform and store all your data in a single index. In this index, you may define a type for user data, another type for blog data, and yet another type for comments data. \n",
    "\n",
    "You can consider Type as the structure definition of the data in an index and compare it to the **table** concept in the traditional DBMS like MySQL.\n",
    "\n",
    "\n",
    "\n",
    "**Document**: A document is a basic unit of information that can be indexed. For example, you can have a document for a single customer, another document for a single product, and yet another for a single order. This document is expressed in [JSON](http://json.org/) (JavaScript Object Notation) which is a ubiquitous internet data interchange format.\n",
    "\n",
    "You can consider Document as the stored data in Elasticsearch storage and compare it to the **Record** concept in the traditional DBMS like MySQL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to Store Data into Elasticsearch?\n",
    "\n",
    "Suppose that now we need to store some poems into Elasticsearch. For a poem, it has some basic properties like title, author, number of words and content. We can first create an Elasticseach **index** called `Poems`. Then define a **type** mapping called `Poem`:\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"mappings\":{\n",
    "        \"poem\":{\n",
    "            \"properties\":{\n",
    "                \"title\":{\n",
    "                    \"type\":\"keyword\"\n",
    "                },\n",
    "                \"author\":{\n",
    "                    \"type\":\"keyword\"\n",
    "                },\n",
    "                \"numOfWords\":{\n",
    "                    \"type\":\"integer\"\n",
    "                },\n",
    "                \"content\":{\n",
    "                    \"type\":\"text\"\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "Then we have a **document** of the famous poem \"*Sonnet XVIII*\" written by William Shakespeare like the following:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"title\": \"Sonnet XVIII\",\n",
    "  \"author\": \"William Shakespeare\",\n",
    "  \"numOfWords\": 114,\n",
    "  \"content\": \"Shall I compare thee to a summer's day?\\nThou art more lovely and more temperate:\\nRough winds do shake the darling buds of May,\\nAnd summer's lease hath all too short a date:\\nSometime too hot the eye of heaven shines,\\nAnd often is his gold complexion dimm'd;\\nAnd every fair from fair sometime declines,\\nBy chance, or nature's changing course, untrimm'd;\\nBut thy eternal summer shall not fade\\nNor lose possession of that fair thou ow'st;\\nNor shall Death brag thou wander'st in his shade,\\nWhen in eternal lines to time thou grow'st;\\nSo long as men can breathe or eyes can see,\\nSo long lives this, and this gives life to thee.\"\n",
    "}\n",
    "```\n",
    "\n",
    "We can see that the entities in ES are represented in JSON (a structure similar to dictionary). Every single **document** JSON represents a record in the **index** `Poems`. The **type** mapping JSON defines the data type of each property and what each property stands for. \n",
    "\n",
    "In the **type** mapping, `keyword`, `text` and `integer` are 3 different datatypes of properties. It's similar to the Lucene field type `StringField` and `TextField` we mentioned before. For string datatypes, `text` means this property will be analyzed, tokenized and indexed (see [Text datatype](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/text.html#text)), while `keyword` will not be tokenized and ES will only match its exact value (see [Keyword datatype](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/keyword.html)). There're many other filed datatypes supported by ES (See [Field datatypes](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping-types.html#mapping-types))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have already prepared all the required data, now we can try to store the data into ES. As we mentioned before, all the APIs are provided in RESTful interfaces in ES, so we can operate ES only by sending HTTP requests.\n",
    "\n",
    "Create **index** `Poems`:\n",
    "\n",
    "```bash\n",
    "$ curl -XPOST 'http://127.0.0.1:9200/poems'\n",
    "\n",
    ">> {\"acknowledged\":true,\"shards_acknowledged\":true,\"index\":\"poems\"}\n",
    "```\n",
    "\n",
    "Create **type** mapping `Poem`:\n",
    "\n",
    "```bash\n",
    "$ curl -XPOST -H \"Content-Type: application/json\" \\\n",
    "--data '{\"poem\":{\"properties\":{\"title\":{\"type\":\"keyword\"},\"author\":{\"type\":\"keyword\"},\"numOfWords\":{\"type\":\"integer\"},\"content\":{\"type\":\"text\"}}}}' \\\n",
    "'http://127.0.0.1:9200/poems/_mapping/poem'\n",
    "\n",
    ">> {\"acknowledged\":true}\n",
    "```\n",
    "\n",
    "Insert the **document** \"*Sonnet XVIII*\":\n",
    "\n",
    "```bash\n",
    "$ curl -XPOST -H \"Content-Type: application/json\" \\\n",
    "--data '{\"title\":\"Sonnet XVIII\",\"author\":\"William Shakespeare\",\"numOfWords\":114,\"content\":\"Shall I compare thee to a summer'\\''s day?\\nThou art more lovely and more temperate:\\nRough winds do shake the darling buds of May,\\nAnd summer'\\''s lease hath all too short a date:\\nSometime too hot the eye of heaven shines,\\nAnd often is his gold complexion dimm'\\''d;\\nAnd every fair from fair sometime declines,\\nBy chance, or nature'\\''s changing course, untrimm'\\''d;\\nBut thy eternal summer shall not fade\\nNor lose possession of that fair thou ow'\\''st;\\nNor shall Death brag thou wander'\\''st in his shade,\\nWhen in eternal lines to time thou grow'\\''st;\\nSo long as men can breathe or eyes can see,\\nSo long lives this, and this gives life to thee.\"}' \\\n",
    "'http://127.0.0.1:9200/poems/poem'\n",
    "\n",
    ">> {\n",
    "    \"_index\":\"poems\",\n",
    "    \"_type\":\"poem\",\n",
    "    \"_id\":\"AXCDwBoxQ_YPTVvO9IMP\",\n",
    "    \"_version\":1,\n",
    "    \"result\":\"created\",\n",
    "    \"_shards\":{\n",
    "        \"total\":2,\n",
    "        \"successful\":1,\n",
    "        \"failed\":0\n",
    "    },\n",
    "    \"created\":true\n",
    "}\n",
    "```\n",
    "\n",
    "Here the `id` is generated automatically.\n",
    "\n",
    "Then we can query this document by the `author` \"William Shakespeare\":\n",
    "\n",
    "```bash\n",
    "$ curl -X GET \"127.0.0.1:9200/poems/poem/_search?pretty\" \\\n",
    "-H 'Content-Type: application/json' \\\n",
    "--data '{\n",
    "  \"query\": {\n",
    "    \"bool\": {\n",
    "      \"should\": [\n",
    "        {\"match\": { \"author\":  \"William\" }}\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}'\n",
    "\n",
    ">>\n",
    "{\n",
    "  \"took\" : 105,\n",
    "  \"timed_out\" : false,\n",
    "  \"_shards\" : {\n",
    "    \"total\" : 5,\n",
    "    \"successful\" : 5,\n",
    "    \"skipped\" : 0,\n",
    "    \"failed\" : 0\n",
    "  },\n",
    "  \"hits\" : {\n",
    "    \"total\" : 1,\n",
    "    \"max_score\" : 0.2876821,\n",
    "    \"hits\" : [\n",
    "      {\n",
    "        \"_index\" : \"poems\",\n",
    "        \"_type\" : \"poem\",\n",
    "        \"_id\" : \"AXCDwBoxQ_YPTVvO9IMP\",\n",
    "        \"_score\" : 0.2876821,\n",
    "        \"_source\" : {\n",
    "          \"title\" : \"Sonnet XVIII\",\n",
    "          \"author\" : \"William Shakespeare\",\n",
    "          \"numOfWords\" : 114,\n",
    "          \"content\" : \"Shall I compare thee to a summer's day?\\nThou art more lovely and more temperate:\\nRough winds do shake the darling buds of May,\\nAnd summer's lease hath all too short a date:\\nSometime too hot the eye of heaven shines,\\nAnd often is his gold complexion dimm'd;\\nAnd every fair from fair sometime declines,\\nBy chance, or nature's changing course, untrimm'd;\\nBut thy eternal summer shall not fade\\nNor lose possession of that fair thou ow'st;\\nNor shall Death brag thou wander'st in his shade,\\nWhen in eternal lines to time thou grow'st;\\nSo long as men can breathe or eyes can see,\\nSo long lives this, and this gives life to thee.\"\n",
    "        }\n",
    "      }\n",
    "    ]\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "There're many other search approaches. See [Search APIs](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python Elasticsearch Examples\n",
    "\n",
    "We can also use an official client [elasticsearch-py](https://github.com/elastic/elasticsearch-py) to connect Elasticsearch cluster in Python directly.\n",
    "\n",
    "First, we need to connect to our ES cluster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import os\n",
    "from elasticsearch5 import Elasticsearch\n",
    "\n",
    "# connect to ES cluster\n",
    "es = Elasticsearch(host=\"127.0.0.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create an index named \"quotes\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'error': {'root_cause': [{'type': 'index_already_exists_exception',\n",
       "    'reason': 'index [quotes/fCk3WGCbRTyckjJ-3UQK6g] already exists',\n",
       "    'index_uuid': 'fCk3WGCbRTyckjJ-3UQK6g',\n",
       "    'index': 'quotes'}],\n",
       "  'type': 'index_already_exists_exception',\n",
       "  'reason': 'index [quotes/fCk3WGCbRTyckjJ-3UQK6g] already exists',\n",
       "  'index_uuid': 'fCk3WGCbRTyckjJ-3UQK6g',\n",
       "  'index': 'quotes'},\n",
       " 'status': 400}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create index named \"quotes\"\n",
    "es.indices.create(index='quotes', ignore=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we create a corresponding type mapping called \"quote\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'acknowledged': True}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create type mapping called \"quote\"\n",
    "type_mapping_json = {\"quote\":{\"properties\":{\"author\":{\"type\":\"keyword\"},\"numOfWords\":{\"type\":\"integer\"},\"content\":{\"type\":\"text\"}}}}\n",
    "es.indices.put_mapping(index=\"quotes\", doc_type=\"quote\", body=type_mapping_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can parse the quote files we used in the Lucene section and create documents in ES."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing:  /usr/src/doc/sample6.txt\n",
      "Parsing:  /usr/src/doc/sample8.txt\n",
      "Parsing:  /usr/src/doc/sample4.txt\n",
      "Parsing:  /usr/src/doc/sample1.txt\n",
      "Parsing:  /usr/src/doc/sample9.txt\n",
      "Parsing:  /usr/src/doc/sample5.txt\n",
      "Parsing:  /usr/src/doc/sample10.txt\n",
      "Parsing:  /usr/src/doc/sample3.txt\n",
      "Parsing:  /usr/src/doc/sample7.txt\n",
      "Parsing:  /usr/src/doc/sample2.txt\n"
     ]
    }
   ],
   "source": [
    "# parse files and create documents in ES\n",
    "docs_path = \"/usr/src/doc\"\n",
    "for tfile in glob.glob(os.path.join(docs_path, '*.txt')):\n",
    "    print(\"Parsing: \", tfile)\n",
    "    text = open(tfile, 'r').read()\n",
    "    content = text[0:text.rfind(\"-\")].strip(' \\\"\\r\\n')\n",
    "    author = text[text.rfind(\"-\")+1:].strip(\" \\r\\n\")\n",
    "    word_count = len(content.split())\n",
    "    \n",
    "    # insert docs into ES\n",
    "    es.index(index=\"quotes\", doc_type=\"quote\", body={\"author\": author, \"numOfWords\": word_count, \"content\": content})\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Search the documents which contains the term \"success\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'took': 95,\n",
       " 'timed_out': False,\n",
       " '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},\n",
       " 'hits': {'total': 3,\n",
       "  'max_score': 0.93629104,\n",
       "  'hits': [{'_index': 'quotes',\n",
       "    '_type': 'quote',\n",
       "    '_id': 'AXCJwWCYjPBjFZvI57Hj',\n",
       "    '_score': 0.93629104,\n",
       "    '_source': {'author': 'Jim Rohn',\n",
       "     'numOfWords': 62,\n",
       "     'content': '“Don’t wish it was easier, wish you were better. Don’t wish for less problems, wish for more skills. Don’t wish for less challenges, wish for more wisdom. The major value in life is not what you get. The major value in life is what you become. Success is not to be pursued; it is to be attracted by the person you become.”'}},\n",
       "   {'_index': 'quotes',\n",
       "    '_type': 'quote',\n",
       "    '_id': 'AXCJwWBBjPBjFZvI57Hb',\n",
       "    '_score': 0.22009256,\n",
       "    '_source': {'author': 'John Traver',\n",
       "     'numOfWords': 17,\n",
       "     'content': 'Success is just a never-ending process of trying to become better and better at whatever you want'}},\n",
       "   {'_index': 'quotes',\n",
       "    '_type': 'quote',\n",
       "    '_id': 'AXCJwWCGjPBjFZvI57Hh',\n",
       "    '_score': 0.1383129,\n",
       "    '_source': {'author': 'Ralph Waldo Emerson',\n",
       "     'numOfWords': 54,\n",
       "     'content': '“Enthusiasm is one of the most powerful engines of success. When you do a thing, do it with all your might. Put your whole soul into it. Stamp it with your own personality. Be active, be energetic, be enthusiastic and faithful, and you will accomplish your object. Nothing great was ever achieved without enthusiasm.”'}}]}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.search(index=\"quotes\", doc_type=\"quote\", body={\"query\": {\"term\": {\"content\": \"success\"}}})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the result is very similar to that of Lucene we did before. All the hits will be sorted by the score.\n",
    "\n",
    "Then we can also search multiple terms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'took': 13,\n",
       " 'timed_out': False,\n",
       " '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},\n",
       " 'hits': {'total': 3,\n",
       "  'max_score': 1.4461608,\n",
       "  'hits': [{'_index': 'quotes',\n",
       "    '_type': 'quote',\n",
       "    '_id': 'AXCJwWCGjPBjFZvI57Hh',\n",
       "    '_score': 1.4461608,\n",
       "    '_source': {'author': 'Ralph Waldo Emerson',\n",
       "     'numOfWords': 54,\n",
       "     'content': '“Enthusiasm is one of the most powerful engines of success. When you do a thing, do it with all your might. Put your whole soul into it. Stamp it with your own personality. Be active, be energetic, be enthusiastic and faithful, and you will accomplish your object. Nothing great was ever achieved without enthusiasm.”'}},\n",
       "   {'_index': 'quotes',\n",
       "    '_type': 'quote',\n",
       "    '_id': 'AXCJwWCYjPBjFZvI57Hj',\n",
       "    '_score': 0.93629104,\n",
       "    '_source': {'author': 'Jim Rohn',\n",
       "     'numOfWords': 62,\n",
       "     'content': '“Don’t wish it was easier, wish you were better. Don’t wish for less problems, wish for more skills. Don’t wish for less challenges, wish for more wisdom. The major value in life is not what you get. The major value in life is what you become. Success is not to be pursued; it is to be attracted by the person you become.”'}},\n",
       "   {'_index': 'quotes',\n",
       "    '_type': 'quote',\n",
       "    '_id': 'AXCJwWBBjPBjFZvI57Hb',\n",
       "    '_score': 0.22009256,\n",
       "    '_source': {'author': 'John Traver',\n",
       "     'numOfWords': 17,\n",
       "     'content': 'Success is just a never-ending process of trying to become better and better at whatever you want'}}]}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.search(index=\"quotes\", doc_type=\"quote\", body={\"query\": {\"terms\": {\"content\": [\"success\", \"enthusiasm\", \"engines\"]}}})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the rank of the hitting documents has changed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How Elasticsearch Stores Data Distributedly?\n",
    "\n",
    "Elasticsearch would first divide the data and each shard of data will have multiple copies. For a shard and its replicas, we have a primary shard and the others are shards replicas. Our write operations will always first be applied to the primary shard, and then the primary shard will copy the new data into other parallel replica shards in background.\n",
    "\n",
    "![es_shards](files/pics/es_shards.png)\n",
    "\n",
    "In this way, each shard of data for an index is scattered among all nodes in the cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use Cases\n",
    "\n",
    "Elasticsearch has a very fast querying speed, but a relatively slow writing speed due to data merge. And generally, Elasticsearch would cost more memory, storage, CPU resources than HBase and some other NoSQL databases. And sometimes in order to improve the querying speed, we will use SSDs instead of hard drives as the storage for Elasticsearch. Thus it's often more expensive to build up an ES cluster.\n",
    "\n",
    "The main use cases for Elasticsearch are full-text search systems, ELK log analysis systems, etc. These use scenarios all don't require real-time write performance but require fast query and analysis performance.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "1.  Lucene is an IR library written in Java based on the idea of the inverted index. It can provide users with plenty of APIs for indexing and searching.\n",
    "2.  Elasticsearch is a distributed full-text search engine based on Lucene. It can provide users with plenty of RESTful APIs for querying and analyzing.\n",
    "3.  Compared to Lucene, Elasticsearch doesn't require the users to have much knowledge in Information Retrieval.\n",
    "4.  The concepts of **index**, **type**, and **document** in Elasticsearch are important. They are similar to the concepts of database, table, and record in RDBMS.\n",
    "5.  Elasticsearch stores data in a distributed way. Each shard of data will have several paralleled replicas.\n",
    "6.  Elasticsearch has a good performance in querying, but a relatively slow write speed due to data merge and replication.\n",
    "7.  A typical application of Elasticsearch is the ELK log analysis system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\n",
    "1.  [Lucene in Action, Second Edition - Manning](https://www.manning.com/books/lucene-in-action-second-edition)\n",
    "2.  [Lucene Indexing in Liferay Part - 1 : Basic Understanding](http://lifreaystuff.blogspot.com/2015/09/lucene-indexing.html)\n",
    "3.  [zzboy/lucene](https://github.com/zzboy/lucene)\n",
    "4.  [Elasticsearch Official Reference](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/index.html)\n",
    "5.  [终于有人把Elasticsearch原理讲透了！](https://zhuanlan.zhihu.com/p/62892586)\n",
    "6.  [Elasticsearch Python API Doc](https://elasticsearch-py.readthedocs.io/en/5.5.1/api.html#indices)\n",
    "7.  [Elasticsearch 5.x Cookbook - Third Edition](https://www.amazon.com/Elasticsearch-5-x-Cookbook-Alberto-Paro/dp/1786465582)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}