notebooks/ocrpy_with_haystack.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OCR, Index and Semantic Search Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before running this tutorial make sure you've installed haystack and also downloaded an opensearch image from dockerhub. If not you can do that by running the following commands:\n",
    "\n",
    "1. Install haystack via `pip install haystack`.\n",
    "2. Then you can download the image by running the following command: `docker pull opensearchproject/opensearch:1.0.1`\n",
    "3. and then please launch the image by running the following command: `docker run -p 9200:9200 -p 9600:9600 -e \"discovery.type=single-node\" opensearchproject/opensearch:1.0.1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ocrpy import DocumentReader, TextOcrIndexPipeline\n",
    "from haystack.nodes import BM25Retriever, TfidfRetriever\n",
    "from haystack.document_stores import OpenSearchDocumentStore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unzip the data\n",
    "!unzip sample_data/data.zip -d sample_data/data\n",
    "!mkdir sample_data/output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lets create a new pipeline and Index the documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "SOURCE = 'sample_data/data' # s3 bucket or local directory or gcs bucket with your documents.\n",
    "DESTINATION = 'sample_data/output/' # s3 bucket or local directory or gcs bucket to write the processed documents.\n",
    "PARSER = 'pytesseract' # or 'google-cloud-vision' or 'pytesseract'\n",
    "CREDENTIALS = {\"AWS\": \"path/to/aws-credentials.env/file\",\n",
    "               \"GCP\": \"path/to/gcp-credentials.json/file\"} # optional - if you are using any cloud service.\n",
    "\n",
    "DATABASE_BACKEND = \"opensearch\"\n",
    "DATABASE_CONFIG = {\"opensearch\": {\"port\": 9200, \"username\": \"admin\", \"password\": \"admin\"} , \"batch_size\": 100}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = TextOcrIndexPipeline(source_dir= SOURCE,\n",
    "                                destination_dir=DESTINATION,\n",
    "                                parser_backend=PARSER,\n",
    "                                credentials_config=CREDENTIALS,\n",
    "                                database_backend=DATABASE_BACKEND,\n",
    "                                database_config=DATABASE_CONFIG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Pipeline with the following configuration:\n",
      "\n",
      "1. DOCUMENT_SOURCE: data\n",
      "2. DOCUMENT_DESTINATION: output\n",
      "3. SOURCE_STORAGE_TYPE: LOCAL\n",
      "4. DESTINATION_STORAGE_TYPE: LOCAL\n",
      "5. PARSER_BACKEND_TYPE: pytesseract\n",
      "6. TOTAL_DOCUMENT_COUNT: 9\n",
      "7. IMAGE_FILE_COUNT: 3\n",
      "8. PDF_FILE_COUNT: 5\n",
      "9. CREDENTIALS: {'AWS': 'path/to/aws-credentials.env/file', 'GCP': 'path/to/gcp-credentials.json/file'}\n",
      "10. DATABASE_BACKEND: opensearch\n",
      "11. DATABASE_CONFIG: {'opensearch': {'port': 9200, 'username': 'admin', 'password': 'admin'}, 'batch_size': 100}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FILE: .DS_Store - ERROR: 'FileTypeNotSupported' object is not iterable\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "9it [07:14, 48.24s/it]\n"
     ]
    }
   ],
   "source": [
    "pipeline.process()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Semantic Search with Haystack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Connect to open search instance with your credentials and index name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a document store to retrieve;\n",
    "doc_store = OpenSearchDocumentStore(**DATABASE_CONFIG['opensearch'])\n",
    "retriver = BM25Retriever(doc_store)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search the index with your query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File Name: How to improve visualization_pytesseract.json\n",
      "Content: \n",
      " How to Practice Visualization Meditation: 3 Best Scripts 21/07/22, 8:22 PM\n",
      "\n",
      "What Is Visualization\n",
      "Meditation?\n",
      "\n",
      " \n",
      "\n",
      " \n",
      "\n",
      "Visualization meditation focuses on the use of guided\n",
      "imagery to cultivate certain \n",
      "----------\n",
      "File Name: How to improve meditation_pytesseract.json\n",
      "Content: \n",
      " How to Perform Body Scan Meditation: 3 Best Scripts 21/07/22, 8:23 PM\n",
      "\n",
      "Research\n",
      "\n",
      "Nervous\n",
      "system\n",
      "response to\n",
      "body scan\n",
      "meditation\n",
      "\n",
      " \n",
      "\n",
      "Although MBSR has been studied extensively as a program\n",
      "and shown t\n",
      "----------\n",
      "File Name: Humanistic Psychology_pytesseract.json\n",
      "Content: \n",
      " Humanistic Psychology's Approach to Wellbeing: 3 Theories\n",
      "\n",
      "Brief History of Humanistic\n",
      "Psychology\n",
      "\n",
      "The revolution of\n",
      "humanistic\n",
      "psychology first\n",
      "began in the 1960s.\n",
      "\n",
      "At this time, humanistic psycholog\n",
      "----------\n"
     ]
    }
   ],
   "source": [
    "for i in retriver.retrieve(query=\"benefits of meditation and visualization\", top_k=3):    \n",
    "    print(f\"File Name: {i.meta['file_name']}\")\n",
    "    print(\"Content: \\n\",i.content[:200])\n",
    "    print(\"-\"*10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lets do another search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File Name: Know your Self_pytesseract.json\n",
      "Content: \n",
      " Gretchen Rubin 21/07/22, 8:44 PM\n",
      "\n",
      " \n",
      "\n",
      "Spotlight on the Know Yourself\n",
      "\n",
      "Better Journal\n",
      "October 18, 2021\n",
      "\n",
      "People often ask me, \"What's the secret to happiness? If you had to\n",
      "choose one thing, what would y\n",
      "----------\n",
      "File Name: How to improve visualization_pytesseract.json\n",
      "Content: \n",
      " How to Practice Visualization Meditation: 3 Best Scripts 21/07/22, 8:22 PM\n",
      "\n",
      "What Is Visualization\n",
      "Meditation?\n",
      "\n",
      " \n",
      "\n",
      " \n",
      "\n",
      "Visualization meditation focuses on the use of guided\n",
      "imagery to cultivate certain \n",
      "----------\n",
      "File Name: How to Become More Creative | Psychology Today_pages-to-jpg-0002_pytesseract.json\n",
      "Content: \n",
      " How to Become More Creative | Psychology Today 21/07/22, 9:06 PM\n",
      "breath and then gasped for another.\n",
      "| can’t believe this is happening to me again. Why me?!\n",
      "Another inner voice suddenly called out:\n",
      "\n",
      "S\n",
      "----------\n"
     ]
    }
   ],
   "source": [
    "for i in retriver.retrieve(query=\"What are best way to be happy\", top_k=3):    \n",
    "    print(f\"File Name: {i.meta['file_name']}\")\n",
    "    print(\"Content: \\n\",i.content[:200])\n",
    "    print(\"-\"*10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.8 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "df77feada6351b27bbd6d76bad3ff5503c942d24557a7a15d7c3085b4f9c786b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}