llama_hub/llama_packs/node_parser/semantic_chunking/semantic_chunking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d617ade9-796f-431f-86ff-6b865e0eb007",
   "metadata": {},
   "source": [
    "# Semantic Chunker\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/run-llama/llama-hub/blob/main/llama_hub/llama_packs/node_parser/semantic_chunking/semantic_chunking.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "\"Semantic chunking\" is a new concept proposed Greg Kamradt in his video tutorial on 5 levels of embedding chunking: https://youtu.be/8OJC21T2SL4?t=1933.\n",
    "\n",
    "Instead of chunking text with a **fixed** chunk size, the semantic splitter adaptively picks the breakpoint in-between sentences using embedding similarity. This ensures that a \"chunk\" contains sentences that are semantically related to each other. \n",
    "\n",
    "We adapted it into a LlamaIndex module.\n",
    "\n",
    "Check out our notebook below!\n",
    "\n",
    "Caveats:\n",
    "\n",
    "- The regex primarily works for English sentences\n",
    "- You may have to tune the breakpoint percentile threshold."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12dcc784-f2c6-4c37-8771-57a921ff2eab",
   "metadata": {},
   "source": [
    "## Setup Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "19321d2d-3567-4f0c-bf8d-08223d59e531",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import SimpleDirectoryReader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "af19dac3-d915-4531-bfc9-a8b0fdf52277",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-01-11 00:04:37--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8003::154, 2606:50c0:8000::154, 2606:50c0:8002::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8003::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 75042 (73K) [text/plain]\n",
      "Saving to: ‘pg_essay.txt’\n",
      "\n",
      "pg_essay.txt        100%[===================>]  73.28K  --.-KB/s    in 0.02s   \n",
      "\n",
      "2024-01-11 00:04:37 (4.29 MB/s) - ‘pg_essay.txt’ saved [75042/75042]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'pg_essay.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "425872ea-e4f1-4a3c-b2de-420ee45f571c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load documents\n",
    "documents = SimpleDirectoryReader(input_files=[\"pg_essay.txt\"]).load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd21470a-a6b4-43aa-94d6-503860706404",
   "metadata": {},
   "source": [
    "## Define Semantic Splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b204c43c-f98a-47fb-b84c-1ed5e07c7f4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# option 1: if importing llama-hub as a package\n",
    "# from llama_hub.llama_packs.node_parser.semantic_chunking.base import SemanticChunker\n",
    "\n",
    "# option 2: if downloading from llama_hub\n",
    "from llama_index.llama_pack import download_llama_pack\n",
    "\n",
    "download_llama_pack(\n",
    "    \"SemanticChunkingQueryEnginePack\",\n",
    "    \"./semantic_chunking_pack\",\n",
    "    skip_load=True,\n",
    "    # leave the below line commented out if using the notebook on main\n",
    "    # llama_hub_url=\"https://raw.githubusercontent.com/run-llama/llama-hub/jerry/add_semantic_chunker/llama_hub\"\n",
    ")\n",
    "from semantic_chunking_pack.base import SemanticChunker\n",
    "\n",
    "\n",
    "from llama_index.node_parser import SentenceSplitter\n",
    "from llama_index.embeddings import OpenAIEmbedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bf398967-74b0-4fe1-a6ee-2da246d33757",
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_model = OpenAIEmbedding()\n",
    "splitter = SemanticChunker(\n",
    "    buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model\n",
    ")\n",
    "\n",
    "# also baseline splitter\n",
    "base_splitter = SentenceSplitter(chunk_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b9cc498a-9d75-4a87-b8a8-fcc995872a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes = splitter.get_nodes_from_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "874ccc0f-7011-47af-89ca-f64ed3b39eb6",
   "metadata": {},
   "source": [
    "### Inspecting the Chunks\n",
    "\n",
    "Let's take a look at chunks produced by the semantic splitter.\n",
    "\n",
    "#### Chunk 1: IBM 1401"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "52c1bdeb-02ff-4d8b-af6a-2196a726b8b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep. The first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights. The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer. I was puzzled by the 1401. I couldn't figure out what to do with it.\n"
     ]
    }
   ],
   "source": [
    "print(nodes[1].get_content())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac0bf8b8-0e30-4d0d-9850-b15100f844f0",
   "metadata": {},
   "source": [
    "#### Chunk 2: Personal Computer + College"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "10492cef-9e5e-468e-a165-a666347cdfa0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear. With microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1]\n",
      "\n",
      "The first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer. Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter. Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored. I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI. AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we'd have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words. There weren't any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar elsewhere. Learning Lisp expanded my concept of a program so fast that it was years before I started to have a sense of where the new limits were. This was more like it; this was what I had expected college to do. It wasn't happening in a class, like it was supposed to, but that was ok. For the next couple years I was on a roll.\n"
     ]
    }
   ],
   "source": [
    "print(nodes[2].get_content())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "883f08f5-f90f-43db-8218-eb533128656b",
   "metadata": {},
   "source": [
    "#### Chunk 3: Finishing up College + Grad School"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "57ef3df6-7bf8-49c2-8b85-46f0dc8ac51d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I knew what I was going to do. For my undergraduate thesis, I reverse-engineered SHRDLU. My God did I love working on that program. It was a pleasing bit of code, but what made it even more exciting was my belief — hard to imagine now, but not unique in 1985 — that it was already climbing the lower slopes of intelligence. I had gotten into a program at Cornell that didn't make you choose a major. You could take whatever classes you liked, and choose whatever you liked to put on your degree. I of course chose \"Artificial Intelligence.\" When I got the actual physical diploma, I was dismayed to find that the quotes had been included, which made them read as scare-quotes. At the time this bothered me, but now it seems amusingly accurate, for reasons I was about to discover. I applied to 3 grad schools: MIT and Yale, which were renowned for AI at the time, and Harvard, which I'd visited because Rich Draves went there, and was also home to Bill Woods, who'd invented the type of parser I used in my SHRDLU clone. Only Harvard accepted me, so that was where I went. I don't remember the moment it happened, or if there even was a specific moment, but during the first year of grad school I realized that AI, as practiced at the time, was a hoax. By which I mean the sort of AI in which a program that's told \"the dog is sitting on the chair\" translates this into some formal representation and adds it to the list of things it knows. What these programs really showed was that there's a subset of natural language that's a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to work. Its brokenness did, as so often happens, generate a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get us Mike. So I looked around to see what I could salvage from the wreckage of my plans, and there was Lisp. I knew from experience that Lisp was interesting for its own sake and not just for its association with AI, even though that was the main reason people cared about it at the time. So I decided to focus on Lisp. In fact, I decided to write a book about Lisp hacking. It's scary to think how little I knew about Lisp hacking when I started writing that book. But there's nothing like writing a book about something to help you learn it. The book, On Lisp, wasn't published till 1993, but I wrote much of it in grad school. Computer Science is an uneasy alliance between two halves, theory and systems. The theory people prove things, and the systems people build things. I wanted to build things. I had plenty of respect for theory — indeed, a sneaking suspicion that it was the more admirable of the two halves — but building things seemed so much more exciting. The problem with systems work, though, was that it didn't last. Any program you wrote today, no matter how good, would be obsolete in a couple decades at best. People might mention your software in footnotes, but no one would actually use it. And indeed, it would seem very feeble work. Only people with a sense of the history of the field would even realize that, in its time, it had been good. There were some surplus Xerox Dandelions floating around the computer lab at one point. Anyone who wanted one to play around with could have one. I was briefly tempted, but they were so slow by present standards; what was the point? No one else wanted one either, so off they went. That was what happened to systems work. I wanted not just to build things, but to build things that would last. In this dissatisfied state I went in 1988 to visit Rich Draves at CMU, where he was in grad school. One day I went to visit the Carnegie Institute, where I'd spent a lot of time as a kid. While looking at a painting there I realized something that might seem obvious, but was a big surprise to me. There, right on the wall, was something you could make that would last. Paintings didn't become obsolete. Some of the best ones were hundreds of years old. And moreover this was something you could make a living doing. Not as easily as you could by writing software, of course, but I thought if you were really industrious and lived really cheaply, it had to be possible to make enough to survive. And as an artist you could be truly independent. You wouldn't have a boss, or even need to get research funding. I had always liked looking at paintings. Could I make them? I had no idea.\n"
     ]
    }
   ],
   "source": [
    "print(nodes[3].get_content())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e0dcab0-92cd-4c3b-b69f-52e1a81bc388",
   "metadata": {},
   "source": [
    "### Compare against Baseline\n",
    "\n",
    "In contrast let's compare against the baseline with a fixed chunk size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9a8b6e27-dd5d-44d0-a13c-05149b624f19",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_nodes = base_splitter.get_nodes_from_documents(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c796f9c3-e9fd-40d1-ba27-8e6cc358bd9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter.\n",
      "\n",
      "Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
      "\n",
      "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.\n",
      "\n",
      "AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we'd have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words.\n",
      "\n",
      "There weren't any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar elsewhere.\n"
     ]
    }
   ],
   "source": [
    "print(base_nodes[2].get_content())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a2c4cd2-15b6-4feb-a01c-611e90385887",
   "metadata": {},
   "source": [
    "## Setup Query Engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2a62b4fa-025c-4ef8-b35b-32b69e86428e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import VectorStoreIndex\n",
    "from llama_index.response.notebook_utils import display_source_node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d20f2a94-a965-409e-acad-79c6bd6799c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_index = VectorStoreIndex(nodes)\n",
    "query_engine = vector_index.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c8553e9b-1889-46d4-8620-4807bd55b0d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_vector_index = VectorStoreIndex(base_nodes)\n",
    "base_query_engine = base_vector_index.as_query_engine()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dc13b0c-8a1f-47f8-860d-e94e42efbf30",
   "metadata": {},
   "source": [
    "### Run some Queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3b6c3276-9d31-4071-b829-5b5d0c4eccf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = query_engine.query(\n",
    "    \"Tell me about the author's programming journey through childhood to college\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "44f01fff-2454-484a-8476-8edfae1b08dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The author's programming journey began in childhood when the only form of input to programs was data stored on punched cards. However, the author did not have any data stored on punched cards, so they were limited in what they could do with programming. They did not know enough math to calculate interesting approximations of pi either. The author's clearest memory of programming during this time was when they learned that programs could fail to terminate. In college, the author initially planned to study philosophy but found it boring and switched to studying AI. They taught themselves Lisp, which was regarded as the language of AI at the time. Learning Lisp expanded their concept of a program and they became passionate about programming.\n"
     ]
    }
   ],
   "source": [
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e93c6eea-aa81-4bb6-9a6b-fb31d57bb685",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Node ID:** f730d1e3-88a2-436b-b094-47413d61caf8<br>**Similarity:** 0.8522101977249293<br>**Text:** And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear. With microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1]\n",
       "\n",
       "The first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer. Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter. Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored. I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI. AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we'd have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words. There weren't any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar elsewhere. Learning Lisp expanded my concept of a program so fast that it was years before I started to have a sense of where the new limits were. This was more like it; this was what I had expected college to do. It wasn't happening in a class, like it was supposed to, but that was ok. For the next couple years I was on a roll.<br>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Node ID:** 8355601a-1622-47ff-bcae-7944b6cb4e64<br>**Similarity:** 0.8465724903756918<br>**Text:** I'd never imagined it was even possible. I knew intellectually that people made art — that it didn't just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous. That fall I started taking art classes at Harvard. Grad students could take classes in any department, and my advisor, Tom Cheatham, was very easy going. If he even knew about the strange classes I was taking, he never said anything. So now I was in a PhD program in computer science, yet planning to be an artist, yet also genuinely in love with Lisp hacking and working away at On Lisp. In other words, like many a grad student, I was working energetically on multiple projects that were not my thesis. I didn't see a way out of this situation. I didn't want to drop out of grad school, but how else was I going to get out? I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he'd found such a spectacular way to get out of grad school. Then one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn't have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing parts of On Lisp where I could, and I was able to respond, with no perceptible delay \"Yes, I think so. I'll give you something to read in a few days.\"\n",
       "\n",
       "I picked applications of continuations as the topic. In retrospect I should have written about macros and embedded languages. There's a whole world there that's barely been explored. But all I wanted was to get out of grad school, and my rapidly written dissertation sufficed, just barely. Meanwhile I was applying to art schools. I applied to two: RISD in the US, and the Accademia di Belli Arti in Florence, which, because it was the oldest art school, I imagined would be good. RISD accepted me, and I never heard back from the Accademia, so off to Providence I went. I'd applied for the BFA program at RISD, which meant in effect that I had to go to college again. This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to take in fundamental subjects like drawing, color, and design. Toward the end of the summer I got a big surprise: a letter from the Accademia, which had been delayed because they'd sent it to Cambridge England instead of Cambridge Massachusetts, inviting me to take the entrance exam in Florence that fall. This was now only weeks away.<br>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for n in response.source_nodes:\n",
    "    display_source_node(n, source_length=20000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f991e373-b5ce-4c1d-ac03-a7071152ffba",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_response = base_query_engine.query(\n",
    "    \"Tell me about the author's programming journey through childhood to college\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4cdf636b-b3d2-4b58-ab8d-c37a3b5c248f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The author's programming journey began in childhood when they started writing simple games and programs to predict the flight of model rockets. They also developed a word processor that their father used to write a book. Despite their interest in programming, the author initially planned to study philosophy in college. However, they found philosophy courses to be boring and decided to switch to studying AI. At that time, there were no AI classes at Cornell, so the author taught themselves by learning Lisp, which was considered the language of AI. The author's programming journey continued to evolve as they encountered new technologies, such as microcomputers, which allowed for more interactive and accessible programming experiences.\n"
     ]
    }
   ],
   "source": [
    "print(str(base_response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fd6e2b03-6802-4d13-abf0-58929b953226",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Node ID:** a650364c-4a35-48e0-9e85-5db063e6009e<br>**Similarity:** 0.8636948911786474<br>**Text:** This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter.\n",
       "\n",
       "Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
       "\n",
       "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.\n",
       "\n",
       "AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we'd have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words.\n",
       "\n",
       "There weren't any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar elsewhere.<br>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Node ID:** b4cedca0-45db-4f61-8e65-7dd371006a6c<br>**Similarity:** 0.8571409465192146<br>**Text:** What I Worked On\n",
       "\n",
       "February 2021\n",
       "\n",
       "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
       "\n",
       "The first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\n",
       "\n",
       "The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\n",
       "\n",
       "I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear.\n",
       "\n",
       "With microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping.<br>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for n in base_response.source_nodes:\n",
    "    display_source_node(n, source_length=20000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d6cec3b0-a3e0-4887-b0df-3db1110721da",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = query_engine.query(\"Tell me about the author's experience in YC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "7481f039-098f-4f23-8f9a-3f69f2144e01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The author had a significant experience in YC. They initially intended for YC to be a part-time job, but as it grew, it started to take up more of their attention. They found it engaging and enjoyed working with the startups in each batch, as it allowed them to learn a lot about startups in a short amount of time. However, there were also parts of the job that they didn't like, such as disputes between cofounders and dealing with people who maltreated the startups. Despite the challenges, the author worked hard and wanted YC to be successful. They received unsolicited advice from Robert Morris, suggesting that they should make sure YC isn't the last cool thing they do, which made them consider the possibility of leaving YC in the future.\n"
     ]
    }
   ],
   "source": [
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c3f553c3-5c94-4f0a-92db-5e2bfb62f7a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_response = base_query_engine.query(\"Tell me about the author's experience in YC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "08a2dcd5-2b8b-4e03-86ee-2c113c1e53bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The author's experience in YC was different from other kinds of work they have done. Instead of deciding for themselves what to work on, the problems came to them. Every 6 months, there was a new batch of startups, and their problems became the author's problems. This work was engaging because the problems were varied, and the good founders were very effective. However, there were parts of the job that the author didn't like, such as disputes between cofounders and dealing with people who maltreated the startups. Despite this, the author worked hard even at the parts they didn't like because they wanted YC to be good.\n"
     ]
    }
   ],
   "source": [
    "print(str(base_response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c5e2784-b439-44ab-826a-5be1eb37a1d8",
   "metadata": {},
   "source": [
    "## Run Pack End-to-End \n",
    "\n",
    "We can also run the LlamaPack e2e."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "227e819a-6845-4027-87a5-6e79587673b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# option 1: if importing llama-hub as a package\n",
    "from llama_hub.llama_packs.node_parser.semantic_chunking.base import (\n",
    "    SemanticChunkingQueryEnginePack,\n",
    ")\n",
    "\n",
    "# # option 2: if downloading from llama_hub\n",
    "# from llama_index.llama_pack import download_llama_pack\n",
    "# download_llama_pack(\n",
    "#     \"SemanticChunkingQueryEnginePack\",\n",
    "#     \"./semantic_chunking_pack\",\n",
    "#     skip_load=True,\n",
    "#     # leave the below line commented out if using the notebook on main\n",
    "#     llama_hub_url=\"https://raw.githubusercontent.com/run-llama/llama-hub/jerry/add_semantic_chunking/llama_hub\"\n",
    "# )\n",
    "# from semantic_chunking.base import SemanticChunkingQueryEnginePack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d96ce450-248e-4c16-aa55-8b4ea2fc204f",
   "metadata": {},
   "outputs": [],
   "source": [
    "pack = SemanticChunkingQueryEnginePack(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "95bfccbb-3989-491a-af74-c037ab395617",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The author's programming experience throughout childhood and college started with limited options for input to programs, which were data stored on punched cards. However, the author did not have any data stored on punched cards, so they couldn't do much with programming at that time. They did not have enough math knowledge to do interesting calculations like approximations of pi. The author's clearest memory of programming was when they learned that programs could fail to terminate. In college, the author initially planned to study philosophy but found it boring and switched to studying AI. They taught themselves Lisp, which was considered the language of AI at that time. The author's undergraduate thesis involved reverse-engineering a program called SHRDLU. During the first year of grad school, the author realized that the way AI was practiced at the time was not effective and decided to focus on Lisp instead. They wrote a book about Lisp hacking called \"On Lisp.\"\n"
     ]
    }
   ],
   "source": [
    "response = pack.run(\n",
    "    \"Tell me about the author's programming experience throughout childhood and college\"\n",
    ")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bf46236-1bb8-4b71-9ca1-17e057a24223",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama_hub",
   "language": "python",
   "name": "llama_hub"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}