BluetickPDF - ebooks/BluetickPDF-ebooks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d1c09417-afb0-4499-b959-82db5104bcae",
   "metadata": {},
   "source": [
    "## BluetickPDF - ebooks Analyzer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c264669-4911-4586-bb4f-8174078e41d2",
   "metadata": {},
   "source": [
    "<img src=\"bluetick-logo.jpg\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e988349-025c-4b2e-aead-30c47475b9d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install ebooklib BeautifulSoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc4f2bfe-be61-4f05-901e-d719586896f4",
   "metadata": {},
   "source": [
    "### Analyzing eBook Contents: \n",
    "To begin our exploration, let's consider an example eBook called \"The Monkey's Paw.\" We start by importing the necessary libraries and loading the eBook file:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "268b4f8f-658a-458d-893b-eff8204115e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = \"The Monkey's Paw.epub\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "384a92b2-fdc0-4ee0-998c-847e49104470",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<ebooklib.epub.EpubHtml object at 0x0000020BF9E541F0>, <ebooklib.epub.EpubHtml object at 0x0000020BF9E54220>, <ebooklib.epub.EpubNav object at 0x0000020BF9E1DC70>, <ebooklib.epub.EpubHtml object at 0x0000020BF9E1D970>]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\akash\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\ebooklib\\epub.py:1395: UserWarning: In the future version we will turn default option ignore_ncx to True.\n",
      "  warnings.warn('In the future version we will turn default option ignore_ncx to True.')\n"
     ]
    }
   ],
   "source": [
    "import ebooklib\n",
    "from ebooklib import epub\n",
    "\n",
    "book = epub.read_epub(file_name)\n",
    "items = list(book.get_items_of_type(ebooklib.ITEM_DOCUMENT))\n",
    "print(items)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fb80350-d4c8-4fa1-a67d-e1db6147df96",
   "metadata": {},
   "source": [
    "### Extracting chapters\n",
    "By accessing the book's items, we can extract individual chapters or sections for further analysis. We collect all the chapters into a list:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "17053027-1b6d-4a8b-9844-cf4650a78224",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n"
     ]
    }
   ],
   "source": [
    "chapters = []    \n",
    "for item in book.get_items():        \n",
    "    if item.get_type() == ebooklib.ITEM_DOCUMENT:            \n",
    "        chapters.append(item.get_content())\n",
    "\n",
    "print(len(chapters))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3346add-01df-4112-822e-0899e8716418",
   "metadata": {},
   "source": [
    "### HTML to Text conversion\n",
    "Next, we convert each chapter's HTML content into plain text for easier processing:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "506b3a88-a9ec-4344-b7b0-35ff72ccb3ee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: The Monkey's Paw Author: W. W. Jacobs Release date: April 1, 2004 [eBook #12122]Most recently updated: December 14, 2020 Language: English Credits: Produced by David Widger                          I. Without, the night was cold and wet, but in the small parlour of LaburnamVilla the blinds were drawn and the fire burned brightly.  Father and sonwere at chess, the former, who possessed ideas about the game involvingradical changes, putting his king into such sharp and unnecessary perilstha\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "def chapter_to_str(chapter):\n",
    "    soup = BeautifulSoup(chapter, 'html.parser')\n",
    "    text = [para.get_text() for para in soup.find_all('p')]\n",
    "    return ' '.join(text)\n",
    "    \n",
    "texts = \"\"  \n",
    "for c in chapters:\n",
    "    raw_text = chapter_to_str(c)\n",
    "    texts += raw_text.replace(\"\\n\",\"\")\n",
    "    \n",
    "\n",
    "print(texts[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4df1cb36-b4e4-49b6-a8b6-ddcc8d9ba234",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21835\n"
     ]
    }
   ],
   "source": [
    "print(len(texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f34abb-f924-4726-bd51-2bf59a68c6b3",
   "metadata": {},
   "source": [
    "### Splitting the Text: \n",
    "Now we have all the text from the eBook concatenated into a single string, ready for analysis.\n",
    "Querying eBook Contents: To enable querying over the eBook's contents, we utilize Langchain, a powerful framework that integrates generative AI models and other tools. Firstly, we split the eBook into smaller documents using Langchain's text splitter:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0ef4bb39-d681-4078-91ed-96efe57e43b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "\n",
    "text_splitter = CharacterTextSplitter(separator=\".\", chunk_size=500, chunk_overlap=10, length_function = len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "84289703-d022-43e5-8e35-51772aa85dda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now our book is split up into 49 documents\n",
      "page_content=\"Title: The Monkey's Paw Author: W. W. Jacobs Release date: April 1, 2004 [eBook #12122]Most recently updated: December 14, 2020 Language: English Credits: Produced by David Widger                          I. Without, the night was cold and wet, but in the small parlour of LaburnamVilla the blinds were drawn and the fire burned brightly\" metadata={}\n"
     ]
    }
   ],
   "source": [
    "pages = text_splitter.create_documents([texts])\n",
    "\n",
    "num_documents = len(pages)\n",
    "print(f\"Now our book is split up into {num_documents} documents\")\n",
    "print(pages[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2fd8b01b-ea3d-4377-9819-49bdf5c2666a",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import os\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"add you openai api key\"\n",
    "os.environ[\"PINECONE_API_KEY\"] = \"add your pinecone api key\"\n",
    "os.environ[\"PINECONE_API_ENV\"] = \"add your pinecone api env\"\n",
    "\n",
    "os.environ[\"index_name\"]       = \"the-monkeys-paw\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2c1acf9-d24a-4b40-9392-ce88bb6345c4",
   "metadata": {},
   "source": [
    "### Generating Embeddings and Creating a Knowledge Base Index:\n",
    "By splitting the eBook into smaller documents, we can perform more efficient and targeted queries.\n",
    "Next, we leverage Langchain's embeddings and vector stores to enable similarity search and question-answering capabilities. We use OpenAI's embeddings and Pinecone as the vector store:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "28a22728-f1e1-4f78-adc6-461ede1b5410",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "embeddings = OpenAIEmbeddings(openai_api_key=os.environ.get(\"OPENAI_API_KEY\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "073f480e-9cb6-4175-9b35-efee7e056d49",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\akash\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\pinecone\\index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm\n"
     ]
    }
   ],
   "source": [
    "from langchain.vectorstores import Pinecone\n",
    "import pinecone\n",
    "\n",
    "# Initialize Pinecone\n",
    "pinecone.init(api_key=os.environ.get(\"PINECONE_API_KEY\"), environment=os.environ.get(\"PINECONE_API_ENV\"))\n",
    "index_name = \"the-monkeys-paw\"\n",
    "\n",
    "# Create the index\n",
    "docsearch = Pinecone.from_texts([t.page_content for t in pages], embeddings, index_name=index_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eabfe36-be7e-4c3a-a777-3bcd0b83ee67",
   "metadata": {},
   "source": [
    "### Querying the Knowledge Base:\n",
    "We have now set up a vector store index, allowing us to perform similarity searches and retrieve relevant documents based on queries.\n",
    "Finally, we can utilize Langchain's generative AI models to answer questions about the eBook's contents. We employ the ChatOpenAI model for this purpose:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3cbee96a-8220-478b-be7b-2f7982958a9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(temperature=0, max_tokens=1000, model_name='gpt-3.5-turbo', openai_api_key=os.environ.get(\"OPENAI_API_KEY\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e8f469cd-6d95-413c-83d2-eed01b024968",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The author of The Monkey's Paw is W. W. Jacobs.\n"
     ]
    }
   ],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "index_name = \"the-monkeys-paw\"\n",
    "text_field = \"text\"\n",
    "index = pinecone.Index(index_name)\n",
    "vectorstore = Pinecone(\n",
    "    index, embeddings.embed_query, text_field\n",
    ")\n",
    "\n",
    "query = \"Who is the author of The Monkey's Paw\"\n",
    "\n",
    "docs = vectorstore.similarity_search(query, k=3)\n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm,\n",
    "    chain_type=\"stuff\",\n",
    "    retriever=vectorstore.as_retriever()\n",
    ")\n",
    "\n",
    "output = qa.run(query)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "682230a9-62b1-4d43-b2ad-270b07eb50c8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}