quickstart_end_to_end.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weaviate quickstart guide (as a notebook!)\n",
    "\n",
    "This notebook will guide you through the basics of Weaviate. You can find the full documentation [on our site here](https://weaviate.io/developers/weaviate/quickstart).\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/weaviate-tutorials/quickstart/blob/main/quickstart_end_to_end.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need the Weaviate Python client. If you don't yet have it installed - you can do so with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -U weaviate-client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weaviate instance\n",
    "\n",
    "For this, you will need a working instance of Weaviate somewhere. We recommend either:\n",
    "- Creating a free sandbox instance on Weaviate Cloud Services (https://console.weaviate.cloud/), or\n",
    "- Using [Embedded Weaviate](https://weaviate.io/developers/weaviate/installation/embedded).\n",
    "\n",
    "Instantiate the client using **one** of the following code examples:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For using WCS\n",
    "\n",
    "NOTE: Before you do this, you need to create the instance in WCS and get the credentials. Please refer to the [WCS Quickstart guide](https://weaviate.io/developers/wcs/quickstart)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # For using WCS\n",
    "# import weaviate\n",
    "# import json\n",
    "# import os\n",
    "\n",
    "# client = weaviate.Client(\n",
    "#     url = \"https://some-endpoint.weaviate.network\",  # Replace with your endpoint\n",
    "#     auth_client_secret=weaviate.AuthApiKey(api_key=\"YOUR-WEAVIATE-API-KEY\"),  # Replace w/ your Weaviate instance API key\n",
    "#     additional_headers = {\n",
    "#         \"X-OpenAI-Api-Key\": os.environ[\"OPENAI_APIKEY\"]  # Replace with your inference API key\n",
    "#     }\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For using Embedded Weaviate\n",
    "\n",
    "This will spin up a Weaviate instance in the background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started /Users/jphwang/.cache/weaviate-embedded: process ID 21668\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{\"action\":\"startup\",\"default_vectorizer_module\":\"none\",\"level\":\"info\",\"msg\":\"the default vectorizer modules is set to \\\"none\\\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer\",\"time\":\"2023-08-18T01:52:22+01:00\"}\n",
      "{\"action\":\"startup\",\"auto_schema_enabled\":true,\"level\":\"info\",\"msg\":\"auto schema enabled setting is set to \\\"true\\\"\",\"time\":\"2023-08-18T01:52:22+01:00\"}\n",
      "{\"action\":\"hnsw_vector_cache_prefill\",\"count\":3000,\"index_id\":\"knowledgeblock_1q1b9BLntORd\",\"level\":\"info\",\"limit\":1000000000000,\"msg\":\"prefilled vector cache\",\"time\":\"2023-08-18T01:52:22+01:00\",\"took\":567417}\n",
      "{\"action\":\"hnsw_vector_cache_prefill\",\"count\":3000,\"index_id\":\"docchunk_ZzfkiDQ02JIO\",\"level\":\"info\",\"limit\":1000000000000,\"msg\":\"prefilled vector cache\",\"time\":\"2023-08-18T01:52:22+01:00\",\"took\":464208}\n",
      "{\"level\":\"warning\",\"msg\":\"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.\",\"time\":\"2023-08-18T01:52:22+01:00\"}\n",
      "{\"action\":\"grpc_startup\",\"level\":\"info\",\"msg\":\"grpc server listening at [::]:50051\",\"time\":\"2023-08-18T01:52:22+01:00\"}\n",
      "{\"action\":\"hnsw_vector_cache_prefill\",\"count\":3000,\"index_id\":\"chunk_Ga9xZlRrrGhH\",\"level\":\"info\",\"limit\":1000000000000,\"msg\":\"prefilled vector cache\",\"time\":\"2023-08-18T01:52:22+01:00\",\"took\":15272542}\n",
      "{\"action\":\"restapi_management\",\"level\":\"info\",\"msg\":\"Serving weaviate at http://127.0.0.1:6666\",\"time\":\"2023-08-18T01:52:22+01:00\"}\n",
      "{\"action\":\"hnsw_vector_cache_prefill\",\"count\":3000,\"index_id\":\"question_cNPobrLkKipx\",\"level\":\"info\",\"limit\":1000000000000,\"msg\":\"prefilled vector cache\",\"time\":\"2023-08-18T01:52:22+01:00\",\"took\":33429500}\n"
     ]
    }
   ],
   "source": [
    "# For using embedded\n",
    "import weaviate\n",
    "from weaviate.embedded import EmbeddedOptions\n",
    "import json\n",
    "import os\n",
    "\n",
    "client = weaviate.Client(\n",
    "    embedded_options=EmbeddedOptions(),\n",
    "    additional_headers = {\n",
    "        \"X-OpenAI-Api-Key\": os.environ[\"OPENAI_APIKEY\"]  # Replace with your inference API key\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "if client.schema.exists(\"Question\"):\n",
    "    client.schema.delete_class(\"Question\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{\"action\":\"hnsw_vector_cache_prefill\",\"count\":1000,\"index_id\":\"question_DAjXSD8bgF1b\",\"level\":\"info\",\"limit\":1000000000000,\"msg\":\"prefilled vector cache\",\"time\":\"2023-08-18T01:52:22+01:00\",\"took\":51959}\n"
     ]
    }
   ],
   "source": [
    "class_obj = {\n",
    "    \"class\": \"Question\",\n",
    "    \"vectorizer\": \"text2vec-openai\",  # If set to \"none\" you must always provide vectors yourself. Could be any other \"text2vec-*\" also.\n",
    "    \"moduleConfig\": {\n",
    "        \"text2vec-openai\": {},\n",
    "        \"generative-openai\": {}  # Ensure the `generative-openai` module is used for generative queries\n",
    "    }\n",
    "}\n",
    "\n",
    "client.schema.create_class(class_obj)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add objects\n",
    "\n",
    "We'll add objects to our Weaviate instance using a batch import process.\n",
    "\n",
    "We shows you two options, where you can either:\n",
    "- Have Weaviate create vectors, or\n",
    "- Specify custom vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Have Weaviate create vectors (with `text2vec-openai`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "importing question: 1\n",
      "importing question: 2\n",
      "importing question: 3\n",
      "importing question: 4\n",
      "importing question: 5\n",
      "importing question: 6\n",
      "importing question: 7\n",
      "importing question: 8\n",
      "importing question: 9\n",
      "importing question: 10\n"
     ]
    }
   ],
   "source": [
    "# Load data\n",
    "import requests\n",
    "url = 'https://raw.githubusercontent.com/weaviate-tutorials/quickstart/main/data/jeopardy_tiny.json'\n",
    "resp = requests.get(url)\n",
    "data = json.loads(resp.text)\n",
    "\n",
    "# Configure a batch process\n",
    "with client.batch(\n",
    "    batch_size=100\n",
    ") as batch:\n",
    "    # Batch import all Questions\n",
    "    for i, d in enumerate(data):\n",
    "        print(f\"importing question: {i+1}\")\n",
    "\n",
    "        properties = {\n",
    "            \"answer\": d[\"Answer\"],\n",
    "            \"question\": d[\"Question\"],\n",
    "            \"category\": d[\"Category\"],\n",
    "        }\n",
    "\n",
    "        client.batch.add_data_object(\n",
    "            properties,\n",
    "            \"Question\",\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Specify \"custom\" vectors (i.e. generated outside of Weaviate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Load data\n",
    "# import requests\n",
    "# fname = \"jeopardy_tiny_with_vectors_all-OpenAI-ada-002.json\"  # This file includes pre-generated vectors\n",
    "# url = f'https://raw.githubusercontent.com/weaviate-tutorials/quickstart/main/data/{fname}'\n",
    "# resp = requests.get(url)\n",
    "# data = json.loads(resp.text)\n",
    "\n",
    "# # Configure a batch process\n",
    "# with client.batch(\n",
    "#     batch_size=100\n",
    "# ) as batch:\n",
    "#     # Batch import all Questions\n",
    "#     for i, d in enumerate(data):\n",
    "#         print(f\"importing question: {i+1}\")\n",
    "\n",
    "#         properties = {\n",
    "#             \"answer\": d[\"Answer\"],\n",
    "#             \"question\": d[\"Question\"],\n",
    "#             \"category\": d[\"Category\"],\n",
    "#         }\n",
    "\n",
    "#         custom_vector = d[\"vector\"]\n",
    "#         client.batch.add_data_object(\n",
    "#             properties,\n",
    "#             \"Question\",\n",
    "#             vector=custom_vector  # Add custom vector\n",
    "#         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Semantic search\n",
    "\n",
    "Let's try a similarity search. We'll use nearText search to look for quiz objects most similar to biology."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"data\": {\n",
      "        \"Get\": {\n",
      "            \"Question\": [\n",
      "                {\n",
      "                    \"answer\": \"DNA\",\n",
      "                    \"category\": \"SCIENCE\",\n",
      "                    \"question\": \"In 1953 Watson & Crick built a model of the molecular structure of this, the gene-carrying substance\"\n",
      "                },\n",
      "                {\n",
      "                    \"answer\": \"species\",\n",
      "                    \"category\": \"SCIENCE\",\n",
      "                    \"question\": \"2000 news: the Gunnison sage grouse isn't just another northern sage grouse, but a new one of this classification\"\n",
      "                }\n",
      "            ]\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "nearText = {\"concepts\": [\"biology\"]}\n",
    "\n",
    "response = (\n",
    "    client.query\n",
    "    .get(\"Question\", [\"question\", \"answer\", \"category\"])\n",
    "    .with_near_text(nearText)\n",
    "    .with_limit(2)\n",
    "    .do()\n",
    ")\n",
    "\n",
    "print(json.dumps(response, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The response includes a list of top 2 (due to the limit set) objects whose vectors are most similar to the word biology.\n",
    "\n",
    "Notice that even though the word biology does not appear anywhere, Weaviate returns biology-related entries.\n",
    "\n",
    "This example shows why vector searches are powerful. Vectorized data objects allow for searches based on degrees of similarity, as shown here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Semantic search with a filter\n",
    "You can add a Boolean filter to your example. For example, let's run the same search, but only look in objects that have a \"category\" value of \"ANIMALS\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"data\": {\n",
      "        \"Get\": {\n",
      "            \"Question\": [\n",
      "                {\n",
      "                    \"answer\": \"the nose or snout\",\n",
      "                    \"category\": \"ANIMALS\",\n",
      "                    \"question\": \"The gavial looks very much like a crocodile except for this bodily feature\"\n",
      "                },\n",
      "                {\n",
      "                    \"answer\": \"Elephant\",\n",
      "                    \"category\": \"ANIMALS\",\n",
      "                    \"question\": \"It's the only living mammal in the order Proboseidea\"\n",
      "                }\n",
      "            ]\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "nearText = {\"concepts\": [\"biology\"]}\n",
    "\n",
    "response = (\n",
    "    client.query\n",
    "    .get(\"Question\", [\"question\", \"answer\", \"category\"])\n",
    "    .with_near_text(nearText)\n",
    "    .with_where({\n",
    "        \"path\": [\"category\"],\n",
    "        \"operator\": \"Equal\",\n",
    "        \"valueText\": \"ANIMALS\"\n",
    "    })\n",
    "    .with_limit(2)\n",
    "    .do()\n",
    ")\n",
    "\n",
    "print(json.dumps(response, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The response includes a list of top 2 (due to the limit set) objects whose vectors are most similar to the word biology - but only from the \"ANIMALS\" category.\n",
    "\n",
    "Using a Boolean filter allows you to combine the flexibility of vector search with the precision of where filters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generative search (single prompt)\n",
    "\n",
    "Next, let's try a generative search, where search results are processed with a large language model (LLM).\n",
    "\n",
    "Here, we use a `single prompt` query, and the model to explain each answer in plain terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"data\": {\n",
      "        \"Get\": {\n",
      "            \"Question\": [\n",
      "                {\n",
      "                    \"_additional\": {\n",
      "                        \"generate\": {\n",
      "                            \"error\": null,\n",
      "                            \"singleResult\": \"DNA is like a special code that tells our bodies how to grow and work. It's like a recipe book that has all the instructions for making you who you are. Just like a recipe book has different recipes for different foods, DNA has different instructions for making different parts of your body, like your eyes, hair, and even your personality! It's really amazing because it's what makes you unique and special.\"\n",
      "                        }\n",
      "                    },\n",
      "                    \"answer\": \"DNA\",\n",
      "                    \"category\": \"SCIENCE\",\n",
      "                    \"question\": \"In 1953 Watson & Crick built a model of the molecular structure of this, the gene-carrying substance\"\n",
      "                },\n",
      "                {\n",
      "                    \"_additional\": {\n",
      "                        \"generate\": {\n",
      "                            \"error\": null,\n",
      "                            \"singleResult\": \"Well, a species is a group of living things that are similar to each other in many ways. They have the same kind of body parts, like legs or wings, and they can have babies with other members of their species. For example, dogs are a species, and so are cats. They look different and act differently, but all dogs can have puppies with other dogs, and all cats can have kittens with other cats. So, a species is like a big family of animals or plants that are all related to each other in a special way.\"\n",
      "                        }\n",
      "                    },\n",
      "                    \"answer\": \"species\",\n",
      "                    \"category\": \"SCIENCE\",\n",
      "                    \"question\": \"2000 news: the Gunnison sage grouse isn't just another northern sage grouse, but a new one of this classification\"\n",
      "                }\n",
      "            ]\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "nearText = {\"concepts\": [\"biology\"]}\n",
    "\n",
    "response = (\n",
    "    client.query\n",
    "    .get(\"Question\", [\"question\", \"answer\", \"category\"])\n",
    "    .with_near_text(nearText)\n",
    "    .with_generate(single_prompt=\"Explain {answer} as you might to a five-year-old.\")\n",
    "    .with_limit(2)\n",
    "    .do()\n",
    ")\n",
    "\n",
    "print(json.dumps(response, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that Weaviate has retrieved the same results as before. But now it includes an additional, generated text with a plain-language explanation of each answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generative search (grouped task)\n",
    "\n",
    "In the next example, we will use a grouped task prompt instead to combine all search results and send them to the LLM with a prompt. We ask the LLM to write a tweet about all of these search results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧬 Did you know? In 1953, Watson & Crick 🧪 built a model of the molecular structure of DNA, the gene-carrying substance! 🧬 #ScienceFacts\n",
      "\n",
      "🐦🌿 Exciting news! In 2000, a new species 🆕 of sage grouse, the Gunnison sage grouse, was discovered. It's not just another northern sage grouse, but a unique classification! 🦆🌿 #ScienceDiscoveries\n"
     ]
    }
   ],
   "source": [
    "response = (\n",
    "    client.query\n",
    "    .get(\"Question\", [\"question\", \"answer\", \"category\"])\n",
    "    .with_near_text({\"concepts\": [\"biology\"]})\n",
    "    .with_generate(grouped_task=\"Write a tweet with emojis about these facts.\")\n",
    "    .with_limit(2)\n",
    "    .do()\n",
    ")\n",
    "\n",
    "print(response[\"data\"][\"Get\"][\"Question\"][0][\"_additional\"][\"generate\"][\"groupedResult\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generative search sends retrieved data from Weaviate to a large language model, or LLM. This allows you to go beyond simple data retrieval, but transform the data into a more useful form, without ever leaving Weaviate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well done! In just a few short minutes, you have:\n",
    "\n",
    "- Created your own cloud-based vector database with Weaviate,\n",
    "- Populated it with data objects,\n",
    "    - Using an inference API, or\n",
    "    - Using custom vectors,\n",
    "- Performed searches, including:\n",
    "    - Semantic search,\n",
    "    - Sementic search with a filter and\n",
    "    - Generative search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next\n",
    "\n",
    "You can do much more with Weaviate. We suggest trying:\n",
    "\n",
    "- Examples from our [search how-to](https://weaviate.io/developers/weaviate/search) guides for [keyword](https://weaviate.io/developers/weaviate/search/bm25), [similarity](https://weaviate.io/developers/weaviate/search/similarity), [hybrid](https://weaviate.io/developers/weaviate/search/hybrid), [generative](https://weaviate.io/developers/weaviate/search/generative) searches and [filters](https://weaviate.io/developers/weaviate/search/filters) or\n",
    "- Learning [how to manage data](https://weaviate.io/developers/weaviate/manage-data), like [reading](https://weaviate.io/developers/weaviate/manage-data/read), [batch importing](https://weaviate.io/developers/weaviate/manage-data/import), [updating](https://weaviate.io/developers/weaviate/manage-data/update), [deleting](https://weaviate.io/developers/weaviate/manage-data/delete) objects or [bulk exporting](https://weaviate.io/developers/weaviate/manage-data/read-all-objects) data.\n",
    "\n",
    "For more holistic learning, try <i class=\"fa-solid fa-graduation-cap\"></i> [Weaviate Academy](https://weaviate.io/developers/academy). We have built free courses for you to learn about Weaviate and the world of vector search.\n",
    "\n",
    "You can also try a larger, [1,000 row](https://raw.githubusercontent.com/databyjp/wv_demo_uploader/main/weaviate_datasets/data/jeopardy_1k.json) version of the Jeopardy! dataset, or [this tiny set of 50 wine reviews](https://raw.githubusercontent.com/databyjp/wv_demo_uploader/main/weaviate_datasets/data/winemag_tiny.csv)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}