![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
+ "\n",
+ "This interactive notebook uses Langchain to split fictional workplace documents into passages and uses OpenAI to transform these passages into embeddings and store them into Elasticsearch.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Then when we ask a question, we retrieve the relevant passages from the vector store and use langchain and OpenAI to provide a summary for the question."
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "GyAst2W-VpHb"
+ },
+ "source": [
+ "## Install packages and import modules\n",
+ "\n",
+ "To get started, we'll need to connect to our Elastic deployment using the Python client.\n",
+ "Because we're using an Elastic Cloud deployment, we'll use the **Cloud ID** to identify our deployment.\n",
+ "\n",
+ "First we need to install the `elasticsearch` Python client."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "33A-cP-XvFCr"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -qU langchain jq openai elasticsearch tiktoken"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "id": "J8-93TiJsNyK"
+ },
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "from urllib.request import urlopen\n",
+ "\n",
+ "url = \"https://raw.githubusercontent.com/elastic/elasticsearch-labs/main/esre-examples/python-flask-example/example-data/data.json\"\n",
+ "\n",
+ "response = urlopen(url)\n",
+ "\n",
+ "workplace_docs = json.loads(response.read())\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qtEOCsCLWCZp"
+ },
+ "source": [
+ "## Create Elastic Cloud deployment\n",
+ "\n",
+ "If you don't have an Elastic Cloud deployment, sign up [here](https://cloud.elastic.co/registration?fromURI=%2Fhome) for a free trial.\n",
+ "\n",
+ "- Go to the [Create deployment](https://cloud.elastic.co/deployments/create) page\n",
+ "- Select **Create deployment**\n",
+ "\n",
+ "Now we can instantiate the [Elasticsearch python client](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html), providing the cloud id and password in your deployment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "id": "a-t1mglib54F"
+ },
+ "outputs": [],
+ "source": [
+ "from elasticsearch import Elasticsearch\n",
+ "\n",
+ "client = Elasticsearch(\n",
+ " cloud_id=\"CLOUD_ID\",\n",
+ " basic_auth=(\"elastic\", \"PASSWORD\")\n",
+ ")\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "p0cQFDl1b9v4"
+ },
+ "source": [
+ "### Split Documents into Passages\n",
+ "\n",
+ "We’ll chunk documents into passages in order to improve the retrieval specificity and to ensure that we can provide multiple passages within the context window of the final question answering prompt.\n",
+ "\n",
+ "Here we are chunking documents into 800 token passages with an overlap of 400 tokens.\n",
+ "\n",
+ "Here we are using a simple splitter but Langchain offers more advanced splitters to reduce the chace of context being lost."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "dbHEoTF6vBXE"
+ },
+ "outputs": [],
+ "source": [
+ "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+ "from langchain.text_splitter import CharacterTextSplitter\n",
+ "from langchain.embeddings import OpenAIEmbeddings\n",
+ "\n",
+ "metadata = []\n",
+ "content = []\n",
+ "\n",
+ "for doc in workplace_docs:\n",
+ " content.append(doc[\"content\"])\n",
+ " metadata.append({\n",
+ " \"name\": doc[\"name\"],\n",
+ " \"summary\": doc[\"summary\"]\n",
+ " })\n",
+ "\n",
+ "text_splitter = CharacterTextSplitter(chunk_size=800, chunk_overlap=400)\n",
+ "docs = text_splitter.create_documents(content, metadatas=metadata)\n",
+ "\n",
+ "embeddings = OpenAIEmbeddings(openai_api_key=\"OPENAI_KEY\")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kRPxrJnXWfMD"
+ },
+ "source": [
+ "### Setup the Index\n",
+ "Next define the mapping for the passages. Langchain relies on two fields: text and vector."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "BfhBEjLsOr0l"
+ },
+ "outputs": [],
+ "source": [
+ "# Define the mapping\n",
+ "mapping = {\n",
+ " \"mappings\": {\n",
+ " \"properties\": {\n",
+ " \"text\": { \"type\": \"keyword\" },\n",
+ " \"vector\": {\n",
+ " \"type\": \"dense_vector\",\n",
+ " \"dims\": 1536,\n",
+ " \"index\": True,\n",
+ " \"similarity\": \"cosine\"\n",
+ " }\n",
+ " }\n",
+ " }\n",
+ "}\n",
+ "\n",
+ "# Create the index\n",
+ "client.indices.create(index='workplace_index', body=mapping)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "RmCUl0hxW4lG"
+ },
+ "source": [
+ "### Enrich Passages with OpenAI Model\n",
+ "\n",
+ "Next we are going to enrich each passage with an embedding from OpenAI."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Qc1LXk-rOzNR"
+ },
+ "outputs": [],
+ "source": [
+ "# get the embeddings from openAI\n",
+ "\n",
+ "texts = []\n",
+ "for passage in docs:\n",
+ " texts.append(passage.page_content)\n",
+ "\n",
+ "textEmbeddings = embeddings.embed_documents(texts)\n",
+ "\n",
+ "# persist the passage documents into elasticsearch\n",
+ "\n",
+ "actions = []\n",
+ "for i, passage in enumerate(docs):\n",
+ " actions.append({\"index\": {\"_index\": \"workplace_index\"}})\n",
+ " actions.append({\n",
+ " \"text\": passage.page_content,\n",
+ " \"vector\": textEmbeddings[i],\n",
+ " \"metadata\": passage.metadata\n",
+ " })\n",
+ "\n",
+ "client.bulk(operations=actions)\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rXJH_MiWejv7"
+ },
+ "source": [
+ "## Asking a question\n",
+ "Now that we have the passages stored in Elasticsearch, we can now ask a question to get the relevant passages."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "OobeBT6rek7Q",
+ "outputId": "ba7b3a7a-253e-4e7f-83b9-cec07ebdac09"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "---- answer ----\n",
+ " The NASA Sales Team is a regional sales team consisting of dedicated account managers, sales representatives, and support staff, led by their respective Area Vice-Presidents (Laura Martinez and Gary Johnson). They are responsible for identifying and pursuing new business opportunities, nurturing existing client relationships, and ensuring customer satisfaction.\n",
+ "---- sources ----\n",
+ "Sales Organization Overview\n",
+ "Our sales organization is structured to effectively serve our customers and achieve our business objectives across multiple regions. The organization is divided into the following main regions:\n",
+ "\n",
+ "The Americas: This region includes the United States, Canada, Mexico, as well as Central and South America. The North America South America region (NASA) has two Area Vice-Presidents: Laura Martinez is the Area Vice-President of North America, and Gary Johnson is the Area Vice-President of South America.\n",
+ "Sales Organization Overview\n",
+ "Each regional sales team consists of dedicated account managers, sales representatives, and support staff, led by their respective Area Vice-Presidents. They are responsible for identifying and pursuing new business opportunities, nurturing existing client relationships, and ensuring customer satisfaction. The teams collaborate closely with other departments, such as marketing, product development, and customer support, to ensure we consistently deliver high-quality products and services to our clients.\n",
+ "Sales Engineering Collaboration\n",
+ "Title: Working with the Sales Team as an Engineer in a Tech Company\n",
+ "Sales Engineering Collaboration\n",
+ "Introduction:\n",
+ "As an engineer in a tech company, collaboration with the sales team is essential to ensure the success of the company's products and services. This guidance document aims to provide an overview of how engineers can effectively work with the sales team, fostering a positive and productive working environment.\n",
+ "Understanding the Sales Team's Role:\n",
+ "The sales team is responsible for promoting and selling the company's products and services to potential clients. Their role involves establishing relationships with customers, understanding their needs, and ensuring that the offered solutions align with their requirements.\n"
+ ]
+ }
+ ],
+ "source": [
+ "from langchain.vectorstores.elastic_vector_search import ElasticKnnSearch\n",
+ "from langchain.llms import OpenAI\n",
+ "from langchain.chains import RetrievalQA\n",
+ "\n",
+ "db = ElasticKnnSearch(\n",
+ " es_connection=client, index_name=\"workplace_index\", embedding=embeddings\n",
+ ")\n",
+ "\n",
+ "retriever = db.as_retriever()\n",
+ "\n",
+ "llm = OpenAI(openai_api_key=\"OPENAI_KEY\")\n",
+ "\n",
+ "qa = RetrievalQA.from_chain_type(\n",
+ " llm=llm,\n",
+ " chain_type=\"stuff\",\n",
+ " retriever=retriever,\n",
+ " return_source_documents=True\n",
+ ")\n",
+ "\n",
+ "ans = qa({\"query\": \"what is the nasa sales team?\"})\n",
+ "\n",
+ "print(\"---- answer ----\")\n",
+ "print(ans[\"result\"])\n",
+ "print(\"---- sources ----\")\n",
+ "for doc in ans[\"source_documents\"]:\n",
+ " print(doc.metadata[\"name\"])\n",
+ " print(doc.page_content)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "SOeP4-DLnwDB",
+ "outputId": "acff2056-eec6-46e5-cddc-9af9050c046e"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "ObjectApiResponse({'acknowledged': True})"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "client.indices.delete(index=\"workplace_index\")"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/colab-notebooks-examples/index.md b/colab-notebooks-examples/index.md
new file mode 100644
index 00000000..e69de29b
diff --git a/colab-notebooks-examples/integrations/hugging-face/hugging-face-endpoints.ipynb b/colab-notebooks-examples/integrations/hugging-face/hugging-face-endpoints.ipynb
new file mode 100644
index 00000000..e69de29b
diff --git a/colab-notebooks-examples/integrations/hugging-face/loading-model-from-hugging-face.ipynb b/colab-notebooks-examples/integrations/hugging-face/loading-model-from-hugging-face.ipynb
new file mode 100644
index 00000000..e69de29b
diff --git a/colab-notebooks-examples/integrations/llama-index/intro.ipynb b/colab-notebooks-examples/integrations/llama-index/intro.ipynb
new file mode 100644
index 00000000..e69de29b
diff --git a/colab-notebooks-examples/integrations/openai/intro.ipynb b/colab-notebooks-examples/integrations/openai/intro.ipynb
new file mode 100644
index 00000000..e69de29b
diff --git a/colab-notebooks-examples/langchain/langchain-elasticsearch-embeddings.ipynb b/colab-notebooks-examples/langchain/langchain-elasticsearch-embeddings.ipynb
new file mode 100644
index 00000000..264fe34f
--- /dev/null
+++ b/colab-notebooks-examples/langchain/langchain-elasticsearch-embeddings.ipynb
@@ -0,0 +1,18 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": []
+ }
+ ],
+ "metadata": {
+ "language_info": {
+ "name": "python"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/colab-notebooks-examples/langchain/langchain-vector-store.ipynb b/colab-notebooks-examples/langchain/langchain-vector-store.ipynb
new file mode 100644
index 00000000..e69de29b
diff --git a/colab-notebooks-examples/search/00-quick-start.ipynb b/colab-notebooks-examples/search/00-quick-start.ipynb
new file mode 100644
index 00000000..daccb2fe
--- /dev/null
+++ b/colab-notebooks-examples/search/00-quick-start.ipynb
@@ -0,0 +1,783 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "87773ce7",
+ "metadata": {
+ "id": "87773ce7"
+ },
+ "source": [
+ "# Elasticsearch Quick Start\n",
+ "\n",
+ "\n",
+ "\n",
+ "This interactive notebook will introduce you to the very basics of getting started with simple Elasticsearch queries, using the official [Elasticsearch Python client](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html).\n",
+ "We'll run through getting the client up and running, indexing a small data set into Elasticsearch, and performing basic searches against your data."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a32202e2",
+ "metadata": {
+ "id": "a32202e2"
+ },
+ "source": [
+ "## Create Elastic Cloud deployment\n",
+ "\n",
+ "If you don't have an Elastic Cloud deployment, sign up [here](https://cloud.elastic.co/registration?fromURI=%2Fhome) for a free trial.\n",
+ "\n",
+ "- Go to the [Create deployment](https://cloud.elastic.co/deployments/create) page\n",
+ " - Select **Create deployment**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "52a6a607",
+ "metadata": {
+ "id": "52a6a607"
+ },
+ "source": [
+ "## Install packages and import modules\n",
+ "\n",
+ "To get started, we'll need to connect to our Elastic deployment using the Python client.\n",
+ "Because we're using an Elastic Cloud deployment, we'll use the **Cloud ID** to identify our deployment.\n",
+ "\n",
+ "First we need to install the `elasticsearch` Python client."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ffc5fa6f",
+ "metadata": {
+ "id": "ffc5fa6f",
+ "scrolled": false
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -qU elasticsearch sentence-transformers==2.2.2"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "28AH8LhI-0UD",
+ "metadata": {
+ "id": "28AH8LhI-0UD"
+ },
+ "source": [
+ "# Setup the Embedding Model\n",
+ "For this example, we're using the all-MiniLM-L6-v2, part of the sentence_transformers library. You can read more about this model in [hugging face](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "WHC3hHGW-wbI",
+ "metadata": {
+ "id": "WHC3hHGW-wbI"
+ },
+ "outputs": [],
+ "source": [
+ "from sentence_transformers import SentenceTransformer\n",
+ "import torch\n",
+ "\n",
+ "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
+ "\n",
+ "model = SentenceTransformer('all-MiniLM-L6-v2', device=device)\n",
+ "model"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0241694c",
+ "metadata": {
+ "id": "0241694c"
+ },
+ "source": [
+ "## Initialize the Elasticsearch client\n",
+ "\n",
+ "Now we can instantiate the [Elasticsearch python client](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html), providing the cloud id and password in your deployment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "id": "f38e0397",
+ "metadata": {
+ "id": "f38e0397",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "ad6df489-d242-4229-a42a-39c5ca19d124"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Elastic Cloud ID··········\n",
+ "Elastic Password··········\n"
+ ]
+ }
+ ],
+ "source": [
+ "from elasticsearch import Elasticsearch\n",
+ "from getpass import getpass\n",
+ "\n",
+ "CLOUD_ID = getpass(\"Elastic Cloud ID\")\n",
+ "CLOUD_PASSWORD = getpass(\"Elastic Password\")\n",
+ "\n",
+ "# Create the client instance\n",
+ "client = Elasticsearch(\n",
+ " cloud_id=CLOUD_ID,\n",
+ " basic_auth=(\"elastic\", CLOUD_PASSWORD)\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fcd165fa",
+ "metadata": {
+ "id": "fcd165fa"
+ },
+ "source": [
+ "If you're running Elasticsearch locally or self-managed, you can pass in the Elasticsearch host instead. [Read more](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#_verifying_https_with_certificate_fingerprints_python_3_10_or_later) on how to connect to Elasticsearch locally"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1462ebd8",
+ "metadata": {
+ "id": "1462ebd8"
+ },
+ "source": [
+ "Confirm that the client has connected with this test."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "id": "25c618eb",
+ "metadata": {
+ "id": "25c618eb",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "30a6ba5b-5109-4457-ddfe-5633a077ca9b"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "{'name': 'instance-0000000000', 'cluster_name': '1a56ad21587c44d3930932eb9fa1d8e8', 'cluster_uuid': 'gX4zlwtlR4qhZpp1SPm4Yg', 'version': {'number': '8.8.2', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': '98e1271edf932a480e4262a471281f1ee295ce6b', 'build_date': '2023-06-26T05:16:16.196344851Z', 'build_snapshot': False, 'lucene_version': '9.6.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(client.info())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "61e1e6d8",
+ "metadata": {
+ "id": "61e1e6d8"
+ },
+ "source": [
+ "## Index some test data\n",
+ "\n",
+ "Our client is set up and connected to our Elastic deployment.\n",
+ "Now we need some data to test out the basics of Elasticsearch queries.\n",
+ "We'll use a small index of books with the following fields:\n",
+ "\n",
+ "- `title`\n",
+ "- `authors`\n",
+ "- `publish_date`\n",
+ "- `num_reviews`\n",
+ "- `publisher`\n",
+ "\n",
+ "### Create index\n",
+ "\n",
+ "Let's create an Elasticsearch index with the correct mappings for our test data."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6bc95238",
+ "metadata": {
+ "id": "6bc95238"
+ },
+ "outputs": [],
+ "source": [
+ "# Define the mapping\n",
+ "mapping = {\n",
+ " \"mappings\": {\n",
+ " \"properties\": {\n",
+ " \"title\": {\"type\": \"text\"},\n",
+ " \"authors\": {\"type\": \"keyword\"},\n",
+ " \"summary\": {\"type\": \"text\"},\n",
+ " \"publish_date\": {\"type\": \"date\"},\n",
+ " \"num_reviews\": {\"type\": \"integer\"},\n",
+ " \"publisher\": {\"type\": \"keyword\"},\n",
+ " \"title_vector\": {\n",
+ " \"type\": \"dense_vector\",\n",
+ " \"dims\": 384,\n",
+ " \"index\": \"true\",\n",
+ " \"similarity\": \"cosine\"\n",
+ " }\n",
+ " }\n",
+ " }\n",
+ "}\n",
+ "\n",
+ "# Create the index\n",
+ "client.indices.create(index='book_index', body=mapping)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "075f5eb6",
+ "metadata": {
+ "id": "075f5eb6"
+ },
+ "source": [
+ "### Index test data\n",
+ "\n",
+ "Run the following command to upload some test data, containing information about 10 popular programming books from this [dataset](https://raw.githubusercontent.com/joemcelroy/elasticsearch-labs/notebooks-guides/colab-notebooks-examples/search/data.json)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "008d723e",
+ "metadata": {
+ "id": "008d723e"
+ },
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "from urllib.request import urlopen\n",
+ "\n",
+ "url = \"https://raw.githubusercontent.com/joemcelroy/elasticsearch-labs/notebooks-guides/colab-notebooks-examples/search/data.json\"\n",
+ "response = urlopen(url)\n",
+ "books = json.loads(response.read())\n",
+ "\n",
+ "actions = []\n",
+ "for book in books:\n",
+ " actions.append({\"index\": {\"_index\": \"book_index\"}})\n",
+ " # Transforming the title into an embedding using the model\n",
+ " book[\"title_vector\"] = model.encode(book[\"title\"]).tolist()\n",
+ " actions.append(book)\n",
+ "client.bulk(index=\"book_index\", operations=actions)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cd8b03e0",
+ "metadata": {
+ "id": "cd8b03e0"
+ },
+ "source": [
+ "## Aside: Pretty printing Elasticsearch responses\n",
+ "\n",
+ "Your API calls will return hard-to-read nested JSON.\n",
+ "We'll create a little function called `pretty_response` to return nice, human-readable outputs from our examples."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "id": "f12ce2c9",
+ "metadata": {
+ "id": "f12ce2c9"
+ },
+ "outputs": [],
+ "source": [
+ "def pretty_response(response):\n",
+ " for hit in response['hits']['hits']:\n",
+ " id = hit['_id']\n",
+ " publication_date = hit['_source']['publish_date']\n",
+ " score = hit['_score']\n",
+ " title = hit['_source']['title']\n",
+ " summary = hit['_source']['summary']\n",
+ " publisher = hit[\"_source\"][\"publisher\"]\n",
+ " num_reviews = hit[\"_source\"][\"num_reviews\"]\n",
+ " authors = hit[\"_source\"][\"authors\"]\n",
+ " pretty_output = (f\"\\nID: {id}\\nPublication date: {publication_date}\\nTitle: {title}\\nSummary: {summary}\\nPublisher: {publisher}\\nReviews: {num_reviews}\\nAuthors: {authors}\\nScore: {score}\")\n",
+ " print(pretty_output)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "39bdefe0",
+ "metadata": {
+ "id": "39bdefe0"
+ },
+ "source": [
+ "##Making Queries\n",
+ "\n",
+ "Now that we have indexed the books, we want to perform a semantic search for books that similarly match the query. We embed the query and perform a search."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
+ "id": "Df7hwcIjYwMT",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "Df7hwcIjYwMT",
+ "outputId": "e63884d7-d4a5-4f5d-ea43-fc2f0793f040"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\n",
+ "ID: OOlWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2008-05-15\n",
+ "Title: JavaScript: The Good Parts\n",
+ "Summary: A deep dive into the parts of JavaScript that are essential to writing maintainable code\n",
+ "Publisher: oreilly\n",
+ "Reviews: 51\n",
+ "Authors: ['douglas crockford']\n",
+ "Score: 0.8075247\n",
+ "\n",
+ "ID: NOlWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2015-03-27\n",
+ "Title: You Don't Know JS: Up & Going\n",
+ "Summary: Introduction to JavaScript and programming as a whole\n",
+ "Publisher: oreilly\n",
+ "Reviews: 36\n",
+ "Authors: ['kyle simpson']\n",
+ "Score: 0.6946182\n",
+ "\n",
+ "ID: NelWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2018-12-04\n",
+ "Title: Eloquent JavaScript\n",
+ "Summary: A modern introduction to programming\n",
+ "Publisher: no starch press\n",
+ "Reviews: 38\n",
+ "Authors: ['marijn haverbeke']\n",
+ "Score: 0.66179085\n",
+ "\n",
+ "ID: MOlWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2019-10-29\n",
+ "Title: The Pragmatic Programmer: Your Journey to Mastery\n",
+ "Summary: A guide to pragmatic programming for software engineers and developers\n",
+ "Publisher: addison-wesley\n",
+ "Reviews: 30\n",
+ "Authors: ['andrew hunt', 'david thomas']\n",
+ "Score: 0.61159486\n",
+ "\n",
+ "ID: OelWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2012-06-27\n",
+ "Title: Introduction to the Theory of Computation\n",
+ "Summary: Introduction to the theory of computation and complexity theory\n",
+ "Publisher: cengage learning\n",
+ "Reviews: 33\n",
+ "Authors: ['michael sipser']\n",
+ "Score: 0.58697784\n",
+ "\n",
+ "ID: N-lWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2011-05-13\n",
+ "Title: The Clean Coder: A Code of Conduct for Professional Programmers\n",
+ "Summary: A guide to professional conduct in the field of software engineering\n",
+ "Publisher: prentice hall\n",
+ "Reviews: 20\n",
+ "Authors: ['robert c. martin']\n",
+ "Score: 0.57042736\n",
+ "\n",
+ "ID: NulWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 1994-10-31\n",
+ "Title: Design Patterns: Elements of Reusable Object-Oriented Software\n",
+ "Summary: Guide to design patterns that can be used in any object-oriented language\n",
+ "Publisher: addison-wesley\n",
+ "Reviews: 45\n",
+ "Authors: ['erich gamma', 'richard helm', 'ralph johnson', 'john vlissides']\n",
+ "Score: 0.56175697\n",
+ "\n",
+ "ID: M-lWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2008-08-11\n",
+ "Title: Clean Code: A Handbook of Agile Software Craftsmanship\n",
+ "Summary: A guide to writing code that is easy to read, understand and maintain\n",
+ "Publisher: prentice hall\n",
+ "Reviews: 55\n",
+ "Authors: ['robert c. martin']\n",
+ "Score: 0.55407417\n",
+ "\n",
+ "ID: MulWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2020-04-06\n",
+ "Title: Artificial Intelligence: A Modern Approach\n",
+ "Summary: Comprehensive introduction to the theory and practice of artificial intelligence\n",
+ "Publisher: pearson\n",
+ "Reviews: 39\n",
+ "Authors: ['stuart russell', 'peter norvig']\n",
+ "Score: 0.5461982\n",
+ "\n",
+ "ID: MelWP4kB-GB5Evg6zHVx\n",
+ "Publication date: 2019-05-03\n",
+ "Title: Python Crash Course\n",
+ "Summary: A fast-paced, no-nonsense guide to programming in Python\n",
+ "Publisher: no starch press\n",
+ "Reviews: 42\n",
+ "Authors: ['eric matthes']\n",
+ "Score: 0.536102\n"
+ ]
+ },
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "\n",
+ "\n",
+ "This interactive notebook will introduce you to the basic Elasticsearch queries, using the official Elasticsearch Python client. Before getting start this section we recommend working through our [quick start](https://github.com/yansavitski/elasticsearch-labs/blob/notebooks-guides/colab-notebooks-examples/search/00-quick-start.ipynb)."
+ ],
+ "metadata": {
+ "id": "83LdOUCwwHzs"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Querying\n",
+ "In the query context, a query clause answers the question _“How well does this document match this query clause?”_. In addition to deciding whether or not the document matches, the query clause also calculates a relevance score in the `_score `metadata field.\n",
+ "\n",
+ "### Full text queries\n",
+ "\n",
+ "Full text queries enable you to search analyzed text fields such as the body of an email. The query string is processed using the same analyzer that was applied to the field during indexing.\n",
+ "\n",
+ "* **match**.\n",
+ " The standard query for performing full text queries, including fuzzy matching and phrase or proximity queries.\n",
+ "* **multi-match**.\n",
+ " The multi-field version of the match query."
+ ],
+ "metadata": {
+ "id": "22onltbgxxGm"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Match query\n",
+ "Returns documents that `match` a provided text, number, date or boolean value. The provided text is analyzed before matching.\n",
+ "\n",
+ "The `match` query is the standard query for performing a full-text search, including options for fuzzy matching.\n",
+ "\n",
+ "[Read more](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html#match-query-ex-request).\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "clXQwoFQ6x61"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "response = client.search(index=\"book_index\", query={\n",
+ " \"match\": {\n",
+ " \"summary\": {\n",
+ " \"query\": \"guide\"\n",
+ " }\n",
+ " }\n",
+ " })\n",
+ "\n",
+ "pd.DataFrame.from_records(pretty_response_transform(response))"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 197
+ },
+ "id": "q_OE0XVx6_qX",
+ "outputId": "6a1d7760-5fb9-4809-e060-e35a398ed3c4"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " id publication_date score \\\n",
+ "0 3cXgIYkBfxlbyhU5Krfc 2019-10-29 0.704228 \n",
+ "1 3sXgIYkBfxlbyhU5Krfc 2019-05-03 0.704228 \n",
+ "2 5MXgIYkBfxlbyhU5Krfd 2011-05-13 0.677165 \n",
+ "3 4MXgIYkBfxlbyhU5Krfc 2008-08-11 0.628835 \n",
+ "4 48XgIYkBfxlbyhU5Krfd 1994-10-31 0.628835 \n",
+ "\n",
+ " title \\\n",
+ "0 The Pragmatic Programmer: Your Journey to Mastery \n",
+ "1 Python Crash Course \n",
+ "2 The Clean Coder: A Code of Conduct for Profess... \n",
+ "3 Clean Code: A Handbook of Agile Software Craft... \n",
+ "4 Design Patterns: Elements of Reusable Object-O... \n",
+ "\n",
+ " summary \n",
+ "0 A guide to pragmatic programming for software ... \n",
+ "1 A fast-paced, no-nonsense guide to programming... \n",
+ "2 A guide to professional conduct in the field o... \n",
+ "3 A guide to writing code that is easy to read, ... \n",
+ "4 Guide to design patterns that can be used in a... "
+ ],
+ "text/html": [
+ "\n",
+ " | \n", + " | id | \n", + "publication_date | \n", + "score | \n", + "title | \n", + "summary | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "3cXgIYkBfxlbyhU5Krfc | \n", + "2019-10-29 | \n", + "0.704228 | \n", + "The Pragmatic Programmer: Your Journey to Mastery | \n", + "A guide to pragmatic programming for software ... | \n", + "
| 1 | \n", + "3sXgIYkBfxlbyhU5Krfc | \n", + "2019-05-03 | \n", + "0.704228 | \n", + "Python Crash Course | \n", + "A fast-paced, no-nonsense guide to programming... | \n", + "
| 2 | \n", + "5MXgIYkBfxlbyhU5Krfd | \n", + "2011-05-13 | \n", + "0.677165 | \n", + "The Clean Coder: A Code of Conduct for Profess... | \n", + "A guide to professional conduct in the field o... | \n", + "
| 3 | \n", + "4MXgIYkBfxlbyhU5Krfc | \n", + "2008-08-11 | \n", + "0.628835 | \n", + "Clean Code: A Handbook of Agile Software Craft... | \n", + "A guide to writing code that is easy to read, ... | \n", + "
| 4 | \n", + "48XgIYkBfxlbyhU5Krfd | \n", + "1994-10-31 | \n", + "0.628835 | \n", + "Design Patterns: Elements of Reusable Object-O... | \n", + "Guide to design patterns that can be used in a... | \n", + "
| \n", + " | id | \n", + "publication_date | \n", + "score | \n", + "title | \n", + "summary | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "4sXgIYkBfxlbyhU5Krfc | \n", + "2018-12-04 | \n", + "2.030753 | \n", + "Eloquent JavaScript | \n", + "A modern introduction to programming | \n", + "
| 1 | \n", + "5cXgIYkBfxlbyhU5Krfd | \n", + "2008-05-15 | \n", + "1.706409 | \n", + "JavaScript: The Good Parts | \n", + "A deep dive into the parts of JavaScript that ... | \n", + "
| 2 | \n", + "4cXgIYkBfxlbyhU5Krfc | \n", + "2015-03-27 | \n", + "1.636058 | \n", + "You Don't Know JS: Up & Going | \n", + "Introduction to JavaScript and programming as ... | \n", + "
| \n", + " | id | \n", + "publication_date | \n", + "score | \n", + "title | \n", + "summary | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "4sXgIYkBfxlbyhU5Krfc | \n", + "2018-12-04 | \n", + "6.092258 | \n", + "Eloquent JavaScript | \n", + "A modern introduction to programming | \n", + "
| 1 | \n", + "5cXgIYkBfxlbyhU5Krfd | \n", + "2008-05-15 | \n", + "5.119226 | \n", + "JavaScript: The Good Parts | \n", + "A deep dive into the parts of JavaScript that ... | \n", + "
| 2 | \n", + "4cXgIYkBfxlbyhU5Krfc | \n", + "2015-03-27 | \n", + "1.636058 | \n", + "You Don't Know JS: Up & Going | \n", + "Introduction to JavaScript and programming as ... | \n", + "
| \n", + " | id | \n", + "publication_date | \n", + "score | \n", + "title | \n", + "summary | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "4sXgIYkBfxlbyhU5Krfc | \n", + "2018-12-04 | \n", + "1.0 | \n", + "Eloquent JavaScript | \n", + "A modern introduction to programming | \n", + "
| 1 | \n", + "5cXgIYkBfxlbyhU5Krfd | \n", + "2008-05-15 | \n", + "1.0 | \n", + "JavaScript: The Good Parts | \n", + "A deep dive into the parts of JavaScript that ... | \n", + "
| \n", + " | id | \n", + "publication_date | \n", + "score | \n", + "title | \n", + "summary | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "4sXgIYkBfxlbyhU5Krfc | \n", + "2018-12-04 | \n", + "1.624602 | \n", + "Eloquent JavaScript | \n", + "A modern introduction to programming | \n", + "
| 1 | \n", + "5cXgIYkBfxlbyhU5Krfd | \n", + "2008-05-15 | \n", + "1.365127 | \n", + "JavaScript: The Good Parts | \n", + "A deep dive into the parts of JavaScript that ... | \n", + "
| \n", + " | id | \n", + "publication_date | \n", + "score | \n", + "title | \n", + "summary | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "4MXgIYkBfxlbyhU5Krfc | \n", + "2008-08-11 | \n", + "1.97297 | \n", + "Clean Code: A Handbook of Agile Software Craft... | \n", + "A guide to writing code that is easy to read, ... | \n", + "
| \n", + " | id | \n", + "publication_date | \n", + "score | \n", + "title | \n", + "summary | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "4MXgIYkBfxlbyhU5Krfc | \n", + "2008-08-11 | \n", + "1.972970 | \n", + "Clean Code: A Handbook of Agile Software Craft... | \n", + "A guide to writing code that is easy to read, ... | \n", + "
| 1 | \n", + "5cXgIYkBfxlbyhU5Krfd | \n", + "2008-05-15 | \n", + "1.254593 | \n", + "JavaScript: The Good Parts | \n", + "A deep dive into the parts of JavaScript that ... | \n", + "
| 2 | \n", + "3cXgIYkBfxlbyhU5Krfc | \n", + "2019-10-29 | \n", + "0.704228 | \n", + "The Pragmatic Programmer: Your Journey to Mastery | \n", + "A guide to pragmatic programming for software ... | \n", + "
| 3 | \n", + "3sXgIYkBfxlbyhU5Krfc | \n", + "2019-05-03 | \n", + "0.704228 | \n", + "Python Crash Course | \n", + "A fast-paced, no-nonsense guide to programming... | \n", + "
| 4 | \n", + "5MXgIYkBfxlbyhU5Krfd | \n", + "2011-05-13 | \n", + "0.677165 | \n", + "The Clean Coder: A Code of Conduct for Profess... | \n", + "A guide to professional conduct in the field o... | \n", + "
| 5 | \n", + "48XgIYkBfxlbyhU5Krfd | \n", + "1994-10-31 | \n", + "0.628835 | \n", + "Design Patterns: Elements of Reusable Object-O... | \n", + "Guide to design patterns that can be used in a... | \n", + "
| \n", + " | id | \n", + "publication_date | \n", + "score | \n", + "title | \n", + "summary | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "3cXgIYkBfxlbyhU5Krfc | \n", + "2019-10-29 | \n", + "0.0 | \n", + "The Pragmatic Programmer: Your Journey to Mastery | \n", + "A guide to pragmatic programming for software ... | \n", + "
| 1 | \n", + "3sXgIYkBfxlbyhU5Krfc | \n", + "2019-05-03 | \n", + "0.0 | \n", + "Python Crash Course | \n", + "A fast-paced, no-nonsense guide to programming... | \n", + "
| 2 | \n", + "4MXgIYkBfxlbyhU5Krfc | \n", + "2008-08-11 | \n", + "0.0 | \n", + "Clean Code: A Handbook of Agile Software Craft... | \n", + "A guide to writing code that is easy to read, ... | \n", + "
| 3 | \n", + "48XgIYkBfxlbyhU5Krfd | \n", + "1994-10-31 | \n", + "0.0 | \n", + "Design Patterns: Elements of Reusable Object-O... | \n", + "Guide to design patterns that can be used in a... | \n", + "
| 4 | \n", + "5MXgIYkBfxlbyhU5Krfd | \n", + "2011-05-13 | \n", + "0.0 | \n", + "The Clean Coder: A Code of Conduct for Profess... | \n", + "A guide to professional conduct in the field o... | \n", + "