getting-started-with-python-client-colab/Getting_Started_With_Weaviate_Python_Client.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.12"
    },
    "colab": {
      "name": "Getting-Started-With-Weaviate-Python-Client.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rapid-harris"
      },
      "source": [
        "# Getting Started with Weaviate Python Library"
      ],
      "id": "rapid-harris"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "enhanced-cabin"
      },
      "source": [
        "* 0. New updates.\n",
        "* 1. What is Weaviate?\n",
        "* 2. Where can it be used?\n",
        "* 3. What are the advantages?\n",
        "* 4. What is Weaviate Python Client?\n",
        "* 5. How to use the Weaviate Python Client with a weaviate cluster?\n",
        "  * 5.0. Create a Weaviate instance/cluster.\n",
        "  * 5.1. Connect to the cluster.\n",
        "  * 5.2. Get Data and Analyze it.\n",
        "  * 5.3. Create appropriate data types.\n",
        "  * 5.4. Load data.\n",
        "  * 5.5. Query data."
      ],
      "id": "enhanced-cabin"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3zUh1JbdOAlj"
      },
      "source": [
        "## 0. New updates."
      ],
      "id": "3zUh1JbdOAlj"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nI3cRsLpOH36"
      },
      "source": [
        "Weaviate-client version 3.0.0 brings some new changes, check the official documentation for all the information here: https://weaviate-python-client.readthedocs.io/en/latest/changelog.html#version-3-0-0 \n",
        "\n",
        "The whole article now contains examples with the old version and the new one (only the ones that are changed)."
      ],
      "id": "nI3cRsLpOH36"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "executive-zimbabwe"
      },
      "source": [
        "## 1. What is Weaviate?"
      ],
      "id": "executive-zimbabwe"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "satisfactory-multiple"
      },
      "source": [
        "Weaviate is an open-source, cloud-native, modular, real-time vector search engine. It is build to scale your machine learning models. Because Weaviate is modular, you can use it with any machine learning model that does data encoding. Weaviate comes with optional modules for text, image, and other other media types, that can be chose based on your task and data. Also, one could use more than one module, depending on the variety of data. More information [here](https://www.semi.technology/developers/weaviate/current/).\n",
        "\n",
        "In this articles we are going to use the _text_ module to see the most important functionalities and capabilities of Weaviate. The text module, also called _text2vec-contextionary_, captures the semantic meaning of the text objects and places it a concept hyper-space. This allows to do semantic search, in contrast to 'word matching search' that other search engines do.\n",
        "\n",
        "For more information about the Weaviate and SeMI Technology - the company that builds Weaviate, visit the official [website](https://www.semi.technology/)."
      ],
      "id": "satisfactory-multiple"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "separate-parameter"
      },
      "source": [
        "## 2. Where can it be used?"
      ],
      "id": "separate-parameter"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "frank-concern"
      },
      "source": [
        "At the moment weaviate is used in such cases as: \n",
        "  - semantic search,\n",
        "  - similarity search,\n",
        "  - image search,\n",
        "  - power recommendation engines,\n",
        "  - e-commerce search,\n",
        "  - cybersecurity threat analysis,\n",
        "  - automated data harmonization,\n",
        "  - anomaly detection,\n",
        "  - data classification in ERP systems,\n",
        "  \n",
        ", and many many more cases."
      ],
      "id": "frank-concern"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "competitive-coffee"
      },
      "source": [
        "## 3. What are the advantages?"
      ],
      "id": "competitive-coffee"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "signed-passing"
      },
      "source": [
        "To understand what are the Weaviate advantages, you should ask yourself these questions:\n",
        "\n",
        " - Is the quality of results, that your current search engine gives you, good enough for you?\n",
        " - Is it is too much work to bring your machine learning models to scale?\n",
        " - Do you need to classify large datasets fast and near-real time?\n",
        " - Do you need to scale your machine learning models to production size?\n",
        " \n",
        "Weaviate is the solution to all these questions."
      ],
      "id": "signed-passing"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dressed-click"
      },
      "source": [
        "## 4. What is Weaviate Python Client?"
      ],
      "id": "dressed-click"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "copyrighted-reach"
      },
      "source": [
        "The Weaviate Python Client is a python package that allows you to connect and interact with a Weaviate instance. The python client is NOT a Weaviate instance but you can use it to create one on the [Weaviate Cloud Service](https://console.semi.technology/). It provides API for importing data, creating schemas, do classification, query data, ... We are going to go through most of them and explain how and when one could use them.\n",
        "\n",
        "The package is published to PyPI ([link](https://pypi.org/project/weaviate-client/)). Also, a CLI tool is available on PyPI ([link](https://pypi.org/project/weaviate-cli/))."
      ],
      "id": "copyrighted-reach"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "attempted-punishment"
      },
      "source": [
        "## 5. How to use the python-client with a weaviate cluster?"
      ],
      "id": "attempted-punishment"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "difficult-martin"
      },
      "source": [
        "In this section we are going to go through the process of creating a weaviate instance, connecting to it and explore some functionalities."
      ],
      "id": "difficult-martin"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "adjacent-purple"
      },
      "source": [
        "### 5.0. Create an weaviate instance/cluster."
      ],
      "id": "adjacent-purple"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brief-beaver"
      },
      "source": [
        "Creating a Weaviate instance can be done in multiple ways. It can be done using a `docker-compose.yaml` file that can be generated [here](https://www.semi.technology/developers/weaviate/current/getting-started/installation.html#customize-your-weaviate-setup). For this option you have to have `docker` and `docker-compose` installed, and space on your drive.\n",
        "\n",
        "Another option is to create an account on [Weaviate Cloud Service](https://console.semi.technology/) (WCS) and create a cluster there. There are different options for clusters you can choose from. If you do not have an account go ahead and create one.\n",
        "\n",
        "In this tutorial we are going to create a cluster on WCS directly from python (you will only need your WCS credentials)."
      ],
      "id": "brief-beaver"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "several-energy"
      },
      "source": [
        "The first thing we have to do now, is to install the Weaviate Python Client. It can be done using pip command. "
      ],
      "id": "several-energy"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "compliant-separate",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "86e12e50-5da5-4242-80de-8308c5a947c9"
      },
      "source": [
        "import sys\n",
        "!{sys.executable} -m pip install weaviate-client==2.5.0 "
      ],
      "id": "compliant-separate",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: weaviate-client==3.0.0 in /usr/local/lib/python3.7/dist-packages (3.0.0)\n",
            "Requirement already satisfied: tqdm<4.62.0,>=4.59.0 in /usr/local/lib/python3.7/dist-packages (from weaviate-client==3.0.0) (4.61.2)\n",
            "Requirement already satisfied: validators<0.19.0,>=0.18.2 in /usr/local/lib/python3.7/dist-packages (from weaviate-client==3.0.0) (0.18.2)\n",
            "Requirement already satisfied: requests<2.26.0,>=2.23.0 in /usr/local/lib/python3.7/dist-packages (from weaviate-client==3.0.0) (2.23.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<2.26.0,>=2.23.0->weaviate-client==3.0.0) (2021.5.30)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<2.26.0,>=2.23.0->weaviate-client==3.0.0) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<2.26.0,>=2.23.0->weaviate-client==3.0.0) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<2.26.0,>=2.23.0->weaviate-client==3.0.0) (2.10)\n",
            "Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from validators<0.19.0,>=0.18.2->weaviate-client==3.0.0) (1.15.0)\n",
            "Requirement already satisfied: decorator>=3.4.0 in /usr/local/lib/python3.7/dist-packages (from validators<0.19.0,>=0.18.2->weaviate-client==3.0.0) (4.4.2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "byy9rNydO_x7"
      },
      "source": [
        "__UPDATE__ for version __3.0.0__.\n",
        "\n",
        "```python\n",
        "import sys\n",
        "!{sys.executable} -m pip install weaviate-client==3.0.0\n",
        "```"
      ],
      "id": "byy9rNydO_x7"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "possible-continent"
      },
      "source": [
        "Now lets import the package and create a cluster on WCS."
      ],
      "id": "possible-continent"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "toxic-cooler"
      },
      "source": [
        "from getpass import getpass # hide password\n",
        "import weaviate # to communicate to the Weaviate instance\n",
        "from weaviate.tools import WCS"
      ],
      "id": "toxic-cooler",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MpVyM1C9PSwX"
      },
      "source": [
        "__UPDATE__ for version __3.0.0__.\n",
        "\n",
        "```python\n",
        "from getpass import getpass # hide password\n",
        "import weaviate # to communicate to the Weaviate instance\n",
        "from weaviate.wcs import WCS\n",
        "```"
      ],
      "id": "MpVyM1C9PSwX"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "polar-miniature"
      },
      "source": [
        "In order to authenticate to WCS or Weaviate instance (if Weaviate instance has Authentication enable) we need to create an Authentication object. At the moment it supports two types of authentication credentials: \n",
        "* Password credentials: `weaviate.auth.AuthClientPassword(username='WCS_ACCOUNT_EMAIL', password='WCS_ACCOUNT_PASSWORD')`\n",
        "* Token credentials `weaviate.auth.AuthClientCredentials(client_secret=YOUR_SECRET_TOKEN)`\n",
        "\n",
        "For WCS we will use the Password credentials.  "
      ],
      "id": "polar-miniature"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "seasonal-wallace",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "379de29d-6b9c-455b-d482-f0994b29fef3"
      },
      "source": [
        "my_credentials = weaviate.auth.AuthClientPassword(username=input(\"User name: \"), password=getpass('Password: '))"
      ],
      "id": "seasonal-wallace",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "User name: stefan@semi.technology\n",
            "Password: ··········\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "backed-foundation"
      },
      "source": [
        "The `my_credentials` object contains your credentials so be careful not make it public."
      ],
      "id": "backed-foundation"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "elect-wellington"
      },
      "source": [
        "my_wcs = WCS(my_credentials)"
      ],
      "id": "elect-wellington",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "logical-beatles"
      },
      "source": [
        "Now that we connected to WCS, we can `create`, `delete`, `get_clusters`, `get_cluster_config` and check the status of a cluster with `is_ready` method.\n",
        "\n",
        "Here is the prototype of the `create` method:\n",
        "```python\n",
        "my_wcs.create(cluster_name:str=None, \n",
        "    cluster_type:str='sandbox',\n",
        "    config:dict=None,\n",
        "    wait_for_completion:bool=True) -> str\n",
        "```\n",
        "The return value is the URL of the created cluster.\n",
        "\n",
        "*If you want to check the prototype and docstring of any methods in a notebook, run this command: `object.method?`. You can also use the `help()` function.*<br>\n",
        "Ex: `WCS.is_ready?` or `my_wcs.is_ready?` or `help(WCS.is_ready)`."
      ],
      "id": "logical-beatles"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mathematical-clear",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "392ad5a3-87e8-4c39-a6f7-ea53482e7def"
      },
      "source": [
        "cluster_name = 'my-first-weaviate-instance'\n",
        "weaviate_url = my_wcs.create(cluster_name=cluster_name)\n",
        "weaviate_url"
      ],
      "id": "mathematical-clear",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 100.0/100 [00:46<00:00,  2.17it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'https://my-first-weaviate-instance.semi.network'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "loving-conference",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "014bc428-67b5-474a-dfff-2960a5fb9187"
      },
      "source": [
        "my_wcs.is_ready(cluster_name)"
      ],
      "id": "loving-conference",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "suspended-portfolio"
      },
      "source": [
        "### 5.1. Connect to the cluster."
      ],
      "id": "suspended-portfolio"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "copyrighted-document"
      },
      "source": [
        "Now we can connect to the created weaviate instance with the `Client` object. The constructor looks like this:\n",
        "```python\n",
        "weaviate.Client(\n",
        "    url:str,\n",
        "    auth_client_secret:weaviate.auth.AuthCredentials=None,\n",
        "    timeout_config:Union[Tuple[int, int], NoneType]=None,\n",
        ")\n",
        "```\n",
        "\n",
        "The constructor has only one required argument, `url`, and two optional ones: `auth_client_secret` - used if weaviate instance has authentication enabled and `timeout_config` - that sets REST time out configuration and is a tuple (retries, time out seconds). For more information about the arguments look at the docstring."
      ],
      "id": "copyrighted-document"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "working-brief"
      },
      "source": [
        "client = weaviate.Client(weaviate_url)"
      ],
      "id": "working-brief",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "chinese-lighter"
      },
      "source": [
        "Now that we connected to Weavite, it does not necessary mean that it is all set up. It might still do some setup processes in the background. We can check the health of the Weaviate instance by calling the `.is_live` method, and check if Weaviate is ready for requests by calling the `.is_ready`."
      ],
      "id": "chinese-lighter"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "suffering-mineral",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ecb8e063-c706-4d9d-a3f4-0e1ec7a9582d"
      },
      "source": [
        "client.is_ready()"
      ],
      "id": "suffering-mineral",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ongoing-factor"
      },
      "source": [
        "### 5.2. Get Data and Analyze it."
      ],
      "id": "ongoing-factor"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "impressive-musical"
      },
      "source": [
        "We set up the Weaviate instance, connected to it and have it ready for requests, now we can take a step back and get some data and analyze it.\n",
        "\n",
        "This step, as for all the machine learning models, is the most important one. Here we have to decide what is relevant, what is important and what data structures/types to use.\n",
        "\n",
        "In this example we are going to use news articles to construct weaviate data. For this we are going to need the `newspaper3k` package."
      ],
      "id": "impressive-musical"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "requested-fancy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cc96de2e-effb-450a-dbd2-a4cadbbfb103"
      },
      "source": [
        "!{sys.executable} -m pip install newspaper3k"
      ],
      "id": "requested-fancy",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting newspaper3k\n",
            "  Downloading newspaper3k-0.2.8-py3-none-any.whl (211 kB)\n",
            "\u001b[K     |████████████████████████████████| 211 kB 5.0 MB/s \n",
            "\u001b[?25hCollecting tldextract>=2.0.1\n",
            "  Downloading tldextract-3.1.0-py2.py3-none-any.whl (87 kB)\n",
            "\u001b[K     |████████████████████████████████| 87 kB 5.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: Pillow>=3.3.0 in /usr/local/lib/python3.7/dist-packages (from newspaper3k) (7.1.2)\n",
            "Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.7/dist-packages (from newspaper3k) (2.8.1)\n",
            "Collecting feedparser>=5.2.1\n",
            "  Downloading feedparser-6.0.8-py3-none-any.whl (81 kB)\n",
            "\u001b[K     |████████████████████████████████| 81 kB 8.1 MB/s \n",
            "\u001b[?25hCollecting jieba3k>=0.35.1\n",
            "  Downloading jieba3k-0.35.1.zip (7.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 7.4 MB 23.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: lxml>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from newspaper3k) (4.2.6)\n",
            "Collecting cssselect>=0.9.2\n",
            "  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: requests>=2.10.0 in /usr/local/lib/python3.7/dist-packages (from newspaper3k) (2.23.0)\n",
            "Requirement already satisfied: beautifulsoup4>=4.4.1 in /usr/local/lib/python3.7/dist-packages (from newspaper3k) (4.6.3)\n",
            "Collecting feedfinder2>=0.0.4\n",
            "  Downloading feedfinder2-0.0.4.tar.gz (3.3 kB)\n",
            "Collecting tinysegmenter==0.3\n",
            "  Downloading tinysegmenter-0.3.tar.gz (16 kB)\n",
            "Requirement already satisfied: PyYAML>=3.11 in /usr/local/lib/python3.7/dist-packages (from newspaper3k) (3.13)\n",
            "Requirement already satisfied: nltk>=3.2.1 in /usr/local/lib/python3.7/dist-packages (from newspaper3k) (3.2.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from feedfinder2>=0.0.4->newspaper3k) (1.15.0)\n",
            "Collecting sgmllib3k\n",
            "  Downloading sgmllib3k-1.0.0.tar.gz (5.8 kB)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.10.0->newspaper3k) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.10.0->newspaper3k) (2021.5.30)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.10.0->newspaper3k) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.10.0->newspaper3k) (3.0.4)\n",
            "Collecting requests-file>=1.4\n",
            "  Downloading requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB)\n",
            "Requirement already satisfied: filelock>=3.0.8 in /usr/local/lib/python3.7/dist-packages (from tldextract>=2.0.1->newspaper3k) (3.0.12)\n",
            "Building wheels for collected packages: tinysegmenter, feedfinder2, jieba3k, sgmllib3k\n",
            "  Building wheel for tinysegmenter (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for tinysegmenter: filename=tinysegmenter-0.3-py3-none-any.whl size=13551 sha256=38308c94d29ed89f0a8ebe2714f227fdd08060d355b2a837bfe68ff593c0fa9f\n",
            "  Stored in directory: /root/.cache/pip/wheels/df/67/41/faca10fa501ca010be41b49d40360c2959e1c4f09bcbfa37fa\n",
            "  Building wheel for feedfinder2 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for feedfinder2: filename=feedfinder2-0.0.4-py3-none-any.whl size=3354 sha256=9553bea542aa5d62d3a2300e5212bdde61c029737615a978453042fdb2e2f1b6\n",
            "  Stored in directory: /root/.cache/pip/wheels/7f/d4/8f/6e2ca54744c9d7292d88ddb8d42876bcdab5e6d84a21c10346\n",
            "  Building wheel for jieba3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for jieba3k: filename=jieba3k-0.35.1-py3-none-any.whl size=7398404 sha256=f466207f13a1383a3906e3ef6d989cb122d10582cb55340695210beb5419243b\n",
            "  Stored in directory: /root/.cache/pip/wheels/4c/91/46/3c208287b726df325a5979574324878b679116e4baae1af3c3\n",
            "  Building wheel for sgmllib3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sgmllib3k: filename=sgmllib3k-1.0.0-py3-none-any.whl size=6065 sha256=e3d7ca957a03496fe3b73d936401045e0c8d899bc18fe521702e1d644b546cdb\n",
            "  Stored in directory: /root/.cache/pip/wheels/73/ad/a4/0dff4a6ef231fc0dfa12ffbac2a36cebfdddfe059f50e019aa\n",
            "Successfully built tinysegmenter feedfinder2 jieba3k sgmllib3k\n",
            "Installing collected packages: sgmllib3k, requests-file, tldextract, tinysegmenter, jieba3k, feedparser, feedfinder2, cssselect, newspaper3k\n",
            "Successfully installed cssselect-1.1.0 feedfinder2-0.0.4 feedparser-6.0.8 jieba3k-0.35.1 newspaper3k-0.2.8 requests-file-1.5.1 sgmllib3k-1.0.0 tinysegmenter-0.3 tldextract-3.1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ziiBvPgwNDK7"
      },
      "source": [
        "**UPDATE:** If none of the articles were downloaded, it might be because nltk punkt tools were not downloaded. To fix it please run the cell below."
      ],
      "id": "ziiBvPgwNDK7"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "whJTLOrzNBi2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8252bf4e-db01-4795-ecb4-1a35f27375d5"
      },
      "source": [
        "import nltk # it is a dependency of newspaper3k\n",
        "nltk.download('punkt')"
      ],
      "id": "whJTLOrzNBi2",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G_MSEE_LNYDk"
      },
      "source": [
        "Thanks to the GitHub user @gosha1128 for finding this bug that is suppressed in the `try`/`except` block in the `get_articles_from_newspaper` function below."
      ],
      "id": "G_MSEE_LNYDk"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "communist-correction"
      },
      "source": [
        "import newspaper\n",
        "import uuid\n",
        "import json\n",
        "from tqdm import tqdm\n",
        "\n",
        "def get_articles_from_newspaper(\n",
        "        news_url: str, \n",
        "        max_articles: int=100\n",
        "    ) -> None:\n",
        "    \"\"\"\n",
        "    Download and save newspaper articles as weaviate schemas.\n",
        "    Parameters\n",
        "    ----------\n",
        "    newspaper_url : str\n",
        "        Newspaper title.\n",
        "    \"\"\"\n",
        "    \n",
        "    objects = []\n",
        "    \n",
        "    # Build the actual newspaper    \n",
        "    news_builder = newspaper.build(news_url, memoize_articles=False)\n",
        "    \n",
        "    if max_articles > news_builder.size():\n",
        "        max_articles = news_builder.size()\n",
        "    pbar = tqdm(total=max_articles)\n",
        "    pbar.set_description(f\"{news_url}\")\n",
        "    i = 0\n",
        "    while len(objects) < max_articles and i < news_builder.size():\n",
        "        article = news_builder.articles[i]\n",
        "        try:\n",
        "            article.download()\n",
        "            article.parse()\n",
        "            article.nlp()\n",
        "\n",
        "            if (article.title != '' and \\\n",
        "                article.title is not None and \\\n",
        "                article.summary != '' and \\\n",
        "                article.summary is not None and\\\n",
        "                article.authors):\n",
        "\n",
        "                # create an UUID for the article using its URL\n",
        "                article_id = uuid.uuid3(uuid.NAMESPACE_DNS, article.url)\n",
        "\n",
        "                # create the object\n",
        "                objects.append({\n",
        "                    'id': str(article_id),\n",
        "                    'title': article.title,\n",
        "                    'summary': article.summary,\n",
        "                    'authors': article.authors\n",
        "                })\n",
        "                \n",
        "                pbar.update(1)\n",
        "\n",
        "        except:\n",
        "            # something went wrong with getting the article, ignore it\n",
        "            pass\n",
        "        i += 1\n",
        "    pbar.close()\n",
        "    return objects"
      ],
      "id": "communist-correction",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "apart-density",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "330ff94a-b9ad-4661-bceb-41b266654f75"
      },
      "source": [
        "data = []\n",
        "data += get_articles_from_newspaper('https://www.theguardian.com/international')\n",
        "data += get_articles_from_newspaper('http://cnn.com')"
      ],
      "id": "apart-density",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "https://www.theguardian.com/international: 100%|██████████| 100/100 [00:27<00:00,  3.66it/s]\n",
            "http://cnn.com: 100%|██████████| 100/100 [00:53<00:00,  1.87it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "front-consideration"
      },
      "source": [
        "### 5.3. Create appropriate data types."
      ],
      "id": "front-consideration"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "involved-granny"
      },
      "source": [
        "In the function `get_articles_from_newspaper` we keep the _title, summary_ and _authors_ of the article. We also compute an UUID (Universally Unique IDentifier) for each article. All of these fields can be seen in the cell above.\n",
        "\n",
        "With this information at hand we already can define a schema, that is a data structure for each object type and how they are related. The schema is a nested dictionary.\n",
        "\n",
        "So lets create the `Article` class schema. We know that the article has a ___title, summary___ and ___authors___.\n",
        "\n",
        "More about schemas and how to create them can be found [here](https://www.semi.technology/developers/weaviate/current/data-schema/schema-configuration.html) and [here](https://www.semi.technology/developers/weaviate/current/restful-api-references/schema.html#parameters)."
      ],
      "id": "involved-granny"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "armed-facility"
      },
      "source": [
        "article_class_schema = {\n",
        "    # name of the class\n",
        "    \"class\": \"Article\",\n",
        "    # a description of what this class represents\n",
        "    \"description\": \"An Article class to store the article summary and its authors\",\n",
        "    # class properties\n",
        "    \"properties\": [\n",
        "        {\n",
        "            \"name\": \"title\",\n",
        "            \"dataType\": [\"string\"],\n",
        "            \"description\": \"The title of the article\", \n",
        "        },\n",
        "        {\n",
        "            \"name\": \"summary\",\n",
        "            \"dataType\": [\"text\"],\n",
        "            \"description\": \"The summary of the article\",\n",
        "        },\n",
        "        {\n",
        "            \"name\": \"hasAuthors\",\n",
        "            \"dataType\": [\"Author\"],\n",
        "            \"description\": \"The authors this article has\",\n",
        "        }\n",
        "    ]\n",
        "}"
      ],
      "id": "armed-facility",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "liquid-habitat"
      },
      "source": [
        "In the class schema above we create a class named `Article`and with the description `An Article class to store the article summary and its authors`. The description is there to explain the user what this class is about.\n",
        "\n",
        "Also we define 3 properties: `title` - The title of the article, of type `string` (case sensitive), `summary` - The summary of the article, of data type `text` (case insensitive), `hasAuthor` - The authors of the article, of data type `Author`. The `Author` is NOT a primitive data type, it is another class that we should define. The list of primitive data types can be found [here](https://www.semi.technology/developers/weaviate/current/data-schema/datatypes.html).\n",
        "\n",
        "**NOTE 1:** The properties should always be in cameCase format and starts with a lowercased word.<br>\n",
        "**NOTE 2:** The property data type is always a list because it can accept more than one data type. \n",
        "\n",
        "Specifying another class as a data type is called cross-referencing. This way you can link your data objects in-between them and create a relation graph."
      ],
      "id": "liquid-habitat"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "flying-corrections"
      },
      "source": [
        "Now lets create the `Author` class schema in the same manner, but with properties `name` and `wroteArticles`."
      ],
      "id": "flying-corrections"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "precious-scene"
      },
      "source": [
        "author_class_schema = {\n",
        "    \"class\": \"Author\",\n",
        "    \"description\": \"An Author class to store the author information\",\n",
        "    \"properties\": [\n",
        "        {\n",
        "            \"name\": \"name\",\n",
        "            \"dataType\": [\"string\"],\n",
        "            \"description\": \"The name of the author\", \n",
        "        },\n",
        "        {\n",
        "            \"name\": \"wroteArticles\",\n",
        "            \"dataType\": [\"Article\"],\n",
        "            \"description\": \"The articles of the author\", \n",
        "        }\n",
        "    ]\n",
        "}"
      ],
      "id": "precious-scene",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "apart-mount"
      },
      "source": [
        "Now that we decided on the data structure, we can tell Weaviate what kind of data we will import. This can be done by accessing the `schema` attribute of the client. \n",
        "\n",
        "Schema can be created in two different ways:\n",
        "1. using the `.create_class()` method, this option creates only one class per call.\n",
        "2. using the `.create()` method, this option creates multiple classes at once (useful if you have the whole schema)\n",
        "\n",
        "Also we can check if a schema is present or if a particular class schema is present with the `.contains()` method.\n",
        "\n",
        "More about schema methods, click [here](https://www.semi.technology/developers/weaviate/current/restful-api-references/schema.html)."
      ],
      "id": "apart-mount"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "creative-wellington"
      },
      "source": [
        "Because we defined each class separately we should use the `.create_class()` method.\n",
        "\n",
        "```python\n",
        "client.schema.create_class(schema_class:Union[dict, str]) -> None\n",
        "```\n",
        "It accepts also file paths or URLs to a class definition file."
      ],
      "id": "creative-wellington"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "elder-disclosure",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 324
        },
        "outputId": "0b3b06f2-bd50-4cf6-f96a-fa6dee5c3647"
      },
      "source": [
        "client.schema.create_class(article_class_schema)"
      ],
      "id": "elder-disclosure",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "UnexpectedStatusCodeException",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mUnexpectedStatusCodeException\u001b[0m             Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-17-6d56a74d9293>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mclient\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mschema\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marticle_class_schema\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/weaviate/schema/crud_schema.py\u001b[0m in \u001b[0;36mcreate_class\u001b[0;34m(self, schema_class)\u001b[0m\n\u001b[1;32m    145\u001b[0m         \u001b[0mcheck_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloaded_schema_class\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    146\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_class_with_premitives\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloaded_schema_class\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 147\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_complex_properties_from_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloaded_schema_class\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    148\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    149\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mdelete_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclass_name\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/weaviate/schema/crud_schema.py\u001b[0m in \u001b[0;36m_create_complex_properties_from_class\u001b[0;34m(self, schema_class)\u001b[0m\n\u001b[1;32m    431\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconn_err\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    432\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstatus_code\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m200\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 433\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mUnexpectedStatusCodeException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Add properties to classes\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    434\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    435\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_create_complex_properties_from_classes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mschema_classes_list\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mUnexpectedStatusCodeException\u001b[0m: Add properties to classes! Unexpected status code: 422, with response body: {'error': [{'message': \"Data type of property 'hasAuthors' is invalid; SingleRef class name 'Author' does not exist\"}]}"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "metropolitan-gibraltar"
      },
      "source": [
        "As we can see, we cannot create the class property that reference a non-existing data type. This does not mean that the class `Article` was not created at all. Lets get the schema from weaviate and look what was created."
      ],
      "id": "metropolitan-gibraltar"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "charitable-location"
      },
      "source": [
        "# helper function\n",
        "def prettify(json_dict): \n",
        "    print(json.dumps(json_dict, indent=2))"
      ],
      "id": "charitable-location",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aging-headset",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "159c5ee9-d393-4013-b649-856e0eae7e21"
      },
      "source": [
        "prettify(client.schema.get())"
      ],
      "id": "aging-headset",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"classes\": [\n",
            "    {\n",
            "      \"class\": \"Article\",\n",
            "      \"description\": \"An Article class to store the article summary and its authors\",\n",
            "      \"invertedIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 60\n",
            "      },\n",
            "      \"moduleConfig\": {\n",
            "        \"text2vec-contextionary\": {\n",
            "          \"vectorizeClassName\": true\n",
            "        }\n",
            "      },\n",
            "      \"properties\": [\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"string\"\n",
            "          ],\n",
            "          \"description\": \"The title of the article\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"title\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"text\"\n",
            "          ],\n",
            "          \"description\": \"The summary of the article\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"summary\"\n",
            "        }\n",
            "      ],\n",
            "      \"vectorIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 300,\n",
            "        \"maxConnections\": 64,\n",
            "        \"efConstruction\": 128,\n",
            "        \"vectorCacheMaxObjects\": 2000000\n",
            "      },\n",
            "      \"vectorIndexType\": \"hnsw\",\n",
            "      \"vectorizer\": \"text2vec-contextionary\"\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "extended-writer"
      },
      "source": [
        "The configurations we did not specify are not mandatory and were set to the default values.\n",
        "\n",
        "As we can see, only the `hasAuthor` property was not created. So lets then create the `Author` class."
      ],
      "id": "extended-writer"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "appreciated-multiple",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3d2530d7-c31b-491f-ae6d-7daf2b89067c"
      },
      "source": [
        "client.schema.create_class(author_class_schema)\n",
        "prettify(client.schema.get())"
      ],
      "id": "appreciated-multiple",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"classes\": [\n",
            "    {\n",
            "      \"class\": \"Article\",\n",
            "      \"description\": \"An Article class to store the article summary and its authors\",\n",
            "      \"invertedIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 60\n",
            "      },\n",
            "      \"moduleConfig\": {\n",
            "        \"text2vec-contextionary\": {\n",
            "          \"vectorizeClassName\": true\n",
            "        }\n",
            "      },\n",
            "      \"properties\": [\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"string\"\n",
            "          ],\n",
            "          \"description\": \"The title of the article\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"title\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"text\"\n",
            "          ],\n",
            "          \"description\": \"The summary of the article\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"summary\"\n",
            "        }\n",
            "      ],\n",
            "      \"vectorIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 300,\n",
            "        \"maxConnections\": 64,\n",
            "        \"efConstruction\": 128,\n",
            "        \"vectorCacheMaxObjects\": 2000000\n",
            "      },\n",
            "      \"vectorIndexType\": \"hnsw\",\n",
            "      \"vectorizer\": \"text2vec-contextionary\"\n",
            "    },\n",
            "    {\n",
            "      \"class\": \"Author\",\n",
            "      \"description\": \"An Author class to store the author information\",\n",
            "      \"invertedIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 60\n",
            "      },\n",
            "      \"moduleConfig\": {\n",
            "        \"text2vec-contextionary\": {\n",
            "          \"vectorizeClassName\": true\n",
            "        }\n",
            "      },\n",
            "      \"properties\": [\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"string\"\n",
            "          ],\n",
            "          \"description\": \"The name of the author\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"name\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"Article\"\n",
            "          ],\n",
            "          \"description\": \"The articles of the author\",\n",
            "          \"name\": \"wroteArticles\"\n",
            "        }\n",
            "      ],\n",
            "      \"vectorIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 300,\n",
            "        \"maxConnections\": 64,\n",
            "        \"efConstruction\": 128,\n",
            "        \"vectorCacheMaxObjects\": 2000000\n",
            "      },\n",
            "      \"vectorIndexType\": \"hnsw\",\n",
            "      \"vectorizer\": \"text2vec-contextionary\"\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cardiac-generic"
      },
      "source": [
        "Now we have both classes created but still we do not have the `hasAuthor` property. No worries, it can be created at any time, using the schema's attribute `property` and its method `create`.\n",
        "```python\n",
        "client.schema.property.create(schema_class_name:str, schema_property:dict) -> None\n",
        "```"
      ],
      "id": "cardiac-generic"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "liquid-merchant"
      },
      "source": [
        "client.schema.property.create('Article', article_class_schema['properties'][2])"
      ],
      "id": "liquid-merchant",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "opposed-penalty"
      },
      "source": [
        "Now lets get the schema and see if it is what we expect it to be."
      ],
      "id": "opposed-penalty"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "starting-russian",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0b779e39-ce78-4221-ea21-263e8da93f3e"
      },
      "source": [
        "prettify(client.schema.get())"
      ],
      "id": "starting-russian",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"classes\": [\n",
            "    {\n",
            "      \"class\": \"Article\",\n",
            "      \"description\": \"An Article class to store the article summary and its authors\",\n",
            "      \"invertedIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 60\n",
            "      },\n",
            "      \"moduleConfig\": {\n",
            "        \"text2vec-contextionary\": {\n",
            "          \"vectorizeClassName\": true\n",
            "        }\n",
            "      },\n",
            "      \"properties\": [\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"string\"\n",
            "          ],\n",
            "          \"description\": \"The title of the article\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"title\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"text\"\n",
            "          ],\n",
            "          \"description\": \"The summary of the article\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"summary\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"Author\"\n",
            "          ],\n",
            "          \"description\": \"The authors this article has\",\n",
            "          \"name\": \"hasAuthors\"\n",
            "        }\n",
            "      ],\n",
            "      \"vectorIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 300,\n",
            "        \"maxConnections\": 64,\n",
            "        \"efConstruction\": 128,\n",
            "        \"vectorCacheMaxObjects\": 2000000\n",
            "      },\n",
            "      \"vectorIndexType\": \"hnsw\",\n",
            "      \"vectorizer\": \"text2vec-contextionary\"\n",
            "    },\n",
            "    {\n",
            "      \"class\": \"Author\",\n",
            "      \"description\": \"An Author class to store the author information\",\n",
            "      \"invertedIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 60\n",
            "      },\n",
            "      \"moduleConfig\": {\n",
            "        \"text2vec-contextionary\": {\n",
            "          \"vectorizeClassName\": true\n",
            "        }\n",
            "      },\n",
            "      \"properties\": [\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"string\"\n",
            "          ],\n",
            "          \"description\": \"The name of the author\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"name\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"Article\"\n",
            "          ],\n",
            "          \"description\": \"The articles of the author\",\n",
            "          \"name\": \"wroteArticles\"\n",
            "        }\n",
            "      ],\n",
            "      \"vectorIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 300,\n",
            "        \"maxConnections\": 64,\n",
            "        \"efConstruction\": 128,\n",
            "        \"vectorCacheMaxObjects\": 2000000\n",
            "      },\n",
            "      \"vectorIndexType\": \"hnsw\",\n",
            "      \"vectorizer\": \"text2vec-contextionary\"\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "respected-carolina"
      },
      "source": [
        "Everything is exactly as we intended."
      ],
      "id": "respected-carolina"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "second-timeline"
      },
      "source": [
        "If you do not want to think about which class was created when and what properties might fail or not (due to yet non-existing classes), there is a solution for it. The solution is to create the whole schema with the `create` method. So lets delete the schema from weaviate and see how it works."
      ],
      "id": "second-timeline"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "better-madness",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "98d911f1-238e-4e98-d8b0-2826b8042c0a"
      },
      "source": [
        "schema = client.schema.get() # save schema\n",
        "client.schema.delete_all() # delete all classes\n",
        "prettify(client.schema.get())"
      ],
      "id": "better-madness",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"classes\": []\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "chief-seven"
      },
      "source": [
        "*Note that if we delete the schema or a class we delete all the objects associated with it.*\n",
        "\n",
        "Now lets create it from the saved schema."
      ],
      "id": "chief-seven"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "above-notice",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3247b31a-840d-44d3-ff06-584400b5ff7a"
      },
      "source": [
        "client.schema.create(schema)\n",
        "prettify(client.schema.get())"
      ],
      "id": "above-notice",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"classes\": [\n",
            "    {\n",
            "      \"class\": \"Article\",\n",
            "      \"description\": \"An Article class to store the article summary and its authors\",\n",
            "      \"invertedIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 60\n",
            "      },\n",
            "      \"moduleConfig\": {\n",
            "        \"text2vec-contextionary\": {\n",
            "          \"vectorizeClassName\": true\n",
            "        }\n",
            "      },\n",
            "      \"properties\": [\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"string\"\n",
            "          ],\n",
            "          \"description\": \"The title of the article\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"title\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"text\"\n",
            "          ],\n",
            "          \"description\": \"The summary of the article\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"summary\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"Author\"\n",
            "          ],\n",
            "          \"description\": \"The authors this article has\",\n",
            "          \"name\": \"hasAuthors\"\n",
            "        }\n",
            "      ],\n",
            "      \"vectorIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 300,\n",
            "        \"maxConnections\": 64,\n",
            "        \"efConstruction\": 128,\n",
            "        \"vectorCacheMaxObjects\": 2000000\n",
            "      },\n",
            "      \"vectorIndexType\": \"hnsw\",\n",
            "      \"vectorizer\": \"text2vec-contextionary\"\n",
            "    },\n",
            "    {\n",
            "      \"class\": \"Author\",\n",
            "      \"description\": \"An Author class to store the author information\",\n",
            "      \"invertedIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 60\n",
            "      },\n",
            "      \"moduleConfig\": {\n",
            "        \"text2vec-contextionary\": {\n",
            "          \"vectorizeClassName\": true\n",
            "        }\n",
            "      },\n",
            "      \"properties\": [\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"string\"\n",
            "          ],\n",
            "          \"description\": \"The name of the author\",\n",
            "          \"moduleConfig\": {\n",
            "            \"text2vec-contextionary\": {\n",
            "              \"skip\": false,\n",
            "              \"vectorizePropertyName\": false\n",
            "            }\n",
            "          },\n",
            "          \"name\": \"name\"\n",
            "        },\n",
            "        {\n",
            "          \"dataType\": [\n",
            "            \"Article\"\n",
            "          ],\n",
            "          \"description\": \"The articles of the author\",\n",
            "          \"name\": \"wroteArticles\"\n",
            "        }\n",
            "      ],\n",
            "      \"vectorIndexConfig\": {\n",
            "        \"cleanupIntervalSeconds\": 300,\n",
            "        \"maxConnections\": 64,\n",
            "        \"efConstruction\": 128,\n",
            "        \"vectorCacheMaxObjects\": 2000000\n",
            "      },\n",
            "      \"vectorIndexType\": \"hnsw\",\n",
            "      \"vectorizer\": \"text2vec-contextionary\"\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "controlled-airfare"
      },
      "source": [
        "This looks exactly as the schema we created class by class and property by property. This way we can save now the schema in a file and in the next session just directly import it by providing the file path.\n",
        "```python\n",
        "# save schema to file\n",
        "with open('schema.json', 'w') as outfile: \n",
        "    json.dump(schema, outfile)\n",
        "# remove current schema from Weaviate, removes all the data too\n",
        "client.schema.delete_all()\n",
        "# import schema using file path\n",
        "client.schema.create('schema.json')\n",
        "# print schema\n",
        "print(json.dumps(client.schema.get(), indent=2))\n",
        "```"
      ],
      "id": "controlled-airfare"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "backed-clark"
      },
      "source": [
        "### 5.4. Load data."
      ],
      "id": "backed-clark"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "competent-stockholm"
      },
      "source": [
        "Now that we have our data ready, and Weaviate is aware of what kind of data we have, we can add the `Articles` and `Authors` to the Weaviate instance.\n",
        "\n",
        "Importing data to weaviate can be done in 3 different ways.\n",
        "\n",
        "1. Adding object by object iteratively. This ca be done using the `data_object` object attribute of the client.\n",
        "2. In batches. This can be done by creating an appropriate batch request object and submitting it using the `batch` object attribute of the client. (__Only in__ `weaviate-client` __version <3.0.0.__)\n",
        "3. Using a `Batcher` object from the `weaviate.tools` module. (__Only in__ `weaviate-client` __version <3.0.0.__)\n",
        "4. __New__ `Batch` __class introduced in weaviate-client version 3.0.0.__\n",
        "\n",
        "We are going to see all of them in action, but first lets underline the differences between them.\n",
        "\n",
        "- Option 1. is the safest method to add data objects and creating references because it does object validation before creating it, whereas importing data in batches skips most of the validation in favor for speed. This option requires one REST request per object, thus is slower than importing data in batches. It is recommended to use this option if you are not sure if the your data is valid.\n",
        "\n",
        "\n",
        "- Option 2. as mentioned above skips most of data validation and requires only one REST request per BATCH. For this method you just add as much data as you want to a batch request (there are 2 types: `ReferenceBatchRequest` and `ObjectsBatchRequest`) then you submit it using the `batch` object attribute of the client. This option requires you to first import data objects, and then references (make sure that the objects used in the reference are already imported before creating a reference). (__Only in__ `weaviate-client` __version <3.0.0.__)\n",
        "\n",
        "\n",
        "- Option 3. relies on the batch requests from 2. but for a `Batcher` you do not have to submit any batch requests it does it automatically for you when it is full. (__Only in__ `weaviate-client` __version <3.0.0.__)\n",
        "\n",
        "- Option 4: __New__ `Batch` __class introduced in weaviate-client version 3.0.0.__ The new `Batch` object does not need the `BatchRequests` from 2. but uses them internally. The new class also supports 3 different cases of loading data in batches: a) Manually - the user has the absolute control when and how to add and create batches; b) Auto-create batches when full; c) Auto-create batches using dynamic batching, i.e. the batch size is adjusted every time it is created to avoid any `Timeout` errors."
      ],
      "id": "competent-stockholm"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "intensive-granny"
      },
      "source": [
        "### 5.4.1 Load data using `data_object` attribute"
      ],
      "id": "intensive-granny"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "excess-treasury"
      },
      "source": [
        "For this case lets take only one article (`data[0]`) and import it to Weaviate using the `data_object` attribute.\n",
        "\n",
        "The way to do it, is by creating first the objects and then the reference that links them.\n",
        "\n",
        "Run `client.data_object.create?` in a notebook to get more info about the method. Or `help(client.data_object.create)` in the IDLE."
      ],
      "id": "excess-treasury"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "governing-highlight"
      },
      "source": [
        "Each data object should have the same format as defined in schema."
      ],
      "id": "governing-highlight"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "animated-garage",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8253462d-5dbd-45ea-accd-6f44fdd49037"
      },
      "source": [
        "prettify(data[0])"
      ],
      "id": "animated-garage",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"id\": \"d6f8b228-a2c6-3b67-a191-17f1c72983f4\",\n",
            "  \"title\": \"Simone Biles to compete in balance beam final, USA Gymnastics says\",\n",
            "  \"summary\": \"Simone Biles will compete again at the Tokyo Olympics.\\nThe six-time Olympic medallist will contest the balance beam final on Tuesday, the final day of artistic gymnastics in Tokyo.\\nIts tweet read: \\u201cWe are so excited to confirm that you will see two US athletes in the balance beam final tomorrow - Suni Lee AND Simone Biles!!\\nWe are so excited to confirm that you will see two U.S. athletes in the balance beam final tomorrow - Suni Lee AND Simone Biles!!\\nOn the balance beam, Biles normally attempts fewer twists than all other events and so she may feel most comfortable in that event.\",\n",
            "  \"authors\": [\n",
            "    \"Tumaini Carayol\"\n",
            "  ]\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "postal-camcorder",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c452a88a-6d2c-4716-ca71-8025e5280861"
      },
      "source": [
        "article_object = {\n",
        "    'title': data[0]['title'],\n",
        "    'summary': data[0]['summary'].replace('\\n', '') # remove newline character\n",
        "    # we leave out the `hasAuthors` because it is a reference and will be created after we create the Authors\n",
        "}\n",
        "article_id = data[0]['id']\n",
        "\n",
        "# validated the object\n",
        "result = client.data_object.validate(\n",
        "    data_object=article_object,\n",
        "    class_name='Article',\n",
        "    uuid=article_id\n",
        ")\n",
        "\n",
        "prettify(result)"
      ],
      "id": "postal-camcorder",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"error\": null,\n",
            "  \"valid\": true\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "loaded-parts"
      },
      "source": [
        "Object passed the validation test, now it is safe to create/import it."
      ],
      "id": "loaded-parts"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "civil-treasurer",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "d9f2631c-3594-483e-a2d9-db0d94950d96"
      },
      "source": [
        "# create the object\n",
        "client.data_object.create(\n",
        "    data_object=article_object,\n",
        "    class_name='Article',\n",
        "    uuid=article_id # if not specified, weaviate is going to create an UUID for you.\n",
        ")"
      ],
      "id": "civil-treasurer",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'d6f8b228-a2c6-3b67-a191-17f1c72983f4'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "stuck-syndrome"
      },
      "source": [
        "The `client.data_object.create` return the UUID of the object, if you specified one it is going to be returned too. If you do not specify one, Weaviate is going to generate one for you and return it.\n",
        "\n",
        "Congratulations we have added our first object to weaviate!!!\n",
        "\n",
        "Now we can actually \"get\" this object from Weaviate by its UUID using `get_by_id` or `get` method. (`get` without specifying and UUID return first 100 objects)"
      ],
      "id": "stuck-syndrome"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "constitutional-condition",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2f0465ba-87c1-4355-f8d3-0a7cedbbb6d9"
      },
      "source": [
        "prettify(client.data_object.get(article_id, with_vector=False))"
      ],
      "id": "constitutional-condition",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"class\": \"Article\",\n",
            "  \"creationTimeUnix\": 1627901256976,\n",
            "  \"id\": \"d6f8b228-a2c6-3b67-a191-17f1c72983f4\",\n",
            "  \"lastUpdateTimeUnix\": 1627901256976,\n",
            "  \"properties\": {\n",
            "    \"summary\": \"Simone Biles will compete again at the Tokyo Olympics.The six-time Olympic medallist will contest the balance beam final on Tuesday, the final day of artistic gymnastics in Tokyo.Its tweet read: \\u201cWe are so excited to confirm that you will see two US athletes in the balance beam final tomorrow - Suni Lee AND Simone Biles!!We are so excited to confirm that you will see two U.S. athletes in the balance beam final tomorrow - Suni Lee AND Simone Biles!!On the balance beam, Biles normally attempts fewer twists than all other events and so she may feel most comfortable in that event.\",\n",
            "    \"title\": \"Simone Biles to compete in balance beam final, USA Gymnastics says\"\n",
            "  },\n",
            "  \"vectorWeights\": null\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ancient-nerve"
      },
      "source": [
        "Now lets create the authors and the cross references between the `Article` and the `Authors`.\n",
        "\n",
        "The reference addition is in the same manner, but to add references use the `client.data_object.reference.add` method."
      ],
      "id": "ancient-nerve"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "toxic-nicaragua"
      },
      "source": [
        "# keep track of the authors already imported/created and their respective UUID\n",
        "# because same author can write more than one paper.\n",
        "created_authors = {}\n",
        "\n",
        "for author in data[0]['authors']:\n",
        "    # create Author\n",
        "    author_object = {\n",
        "        'name': author,\n",
        "        # we leave out the `wroteArticles` because it is a reference and will be created after we create the Author\n",
        "    }\n",
        "    author_id = client.data_object.create(\n",
        "        data_object=author_object,\n",
        "        class_name='Author'\n",
        "    )\n",
        "    \n",
        "    # add author to the created_authors\n",
        "    created_authors[author] = author_id\n",
        "    \n",
        "    # add references\n",
        "    ## Author -> Article\n",
        "    client.data_object.reference.add(\n",
        "        from_uuid=author_id,\n",
        "        from_property_name='wroteArticles',\n",
        "        to_uuid=article_id\n",
        "    )\n",
        "    ## Article -> Author \n",
        "    client.data_object.reference.add(\n",
        "        from_uuid=article_id,\n",
        "        from_property_name='hasAuthors',\n",
        "        to_uuid=author_id\n",
        "    )\n",
        "    "
      ],
      "id": "toxic-nicaragua",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "legislative-major"
      },
      "source": [
        "In the cell above we iterate through all authors of the article. For each iteration we first create the `Author` then we add the references: the reference from `Author` to `Article` - linked via the `wroteArticles` property of the `Author`, and reference from `Article` to `Author` - through the `hasAuthors` property of the `Article`.\n",
        "\n",
        "Note that it is not required to have bi-directional references.\n",
        "\n",
        "Now lets get the object and take a look at it."
      ],
      "id": "legislative-major"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fleet-tucson",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "647c2c0d-c21f-475e-954b-820ed7c97049"
      },
      "source": [
        "prettify(client.data_object.get(article_id, with_vector=False))"
      ],
      "id": "fleet-tucson",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"class\": \"Article\",\n",
            "  \"creationTimeUnix\": 1627901256976,\n",
            "  \"id\": \"d6f8b228-a2c6-3b67-a191-17f1c72983f4\",\n",
            "  \"lastUpdateTimeUnix\": 1627901256976,\n",
            "  \"properties\": {\n",
            "    \"hasAuthors\": [\n",
            "      {\n",
            "        \"beacon\": \"weaviate://localhost/748a1515-69e1-411c-bbef-8fb91dcc94d7\",\n",
            "        \"href\": \"/v1/objects/748a1515-69e1-411c-bbef-8fb91dcc94d7\"\n",
            "      }\n",
            "    ],\n",
            "    \"summary\": \"Simone Biles will compete again at the Tokyo Olympics.The six-time Olympic medallist will contest the balance beam final on Tuesday, the final day of artistic gymnastics in Tokyo.Its tweet read: \\u201cWe are so excited to confirm that you will see two US athletes in the balance beam final tomorrow - Suni Lee AND Simone Biles!!We are so excited to confirm that you will see two U.S. athletes in the balance beam final tomorrow - Suni Lee AND Simone Biles!!On the balance beam, Biles normally attempts fewer twists than all other events and so she may feel most comfortable in that event.\",\n",
            "    \"title\": \"Simone Biles to compete in balance beam final, USA Gymnastics says\"\n",
            "  },\n",
            "  \"vectorWeights\": null\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "several-professor"
      },
      "source": [
        "As we can see we have the reference set as a `beacon` and a `href`. We cannot see the `Author`s data by getting the objects from weaviate. We can do it by _querying_ data (see section **5.5 Query data.**) or by getting the the object by the UUID (or `beacon`, or `href`)."
      ],
      "id": "several-professor"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "surprised-alliance",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b541cce2-4b17-451b-825f-cb3b6d3ef307"
      },
      "source": [
        "from weaviate.util import get_valid_uuid # extract UUID from URL (beacon or href)\n",
        "\n",
        "# extract authors references, lets take only the first one as an example (the article might have only one)\n",
        "author = client.data_object.get(article_id, with_vector=False)['properties']['hasAuthors'][0]\n",
        "\n",
        "# get and print data object by providing the 'beacon'\n",
        "author_uuid = get_valid_uuid(author['beacon']) # can be 'href' too\n",
        "prettify(client.data_object.get(author_uuid, with_vector=False))"
      ],
      "id": "surprised-alliance",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"class\": \"Author\",\n",
            "  \"creationTimeUnix\": 1627901264041,\n",
            "  \"id\": \"748a1515-69e1-411c-bbef-8fb91dcc94d7\",\n",
            "  \"lastUpdateTimeUnix\": 1627901264041,\n",
            "  \"properties\": {\n",
            "    \"name\": \"Tumaini Carayol\",\n",
            "    \"wroteArticles\": [\n",
            "      {\n",
            "        \"beacon\": \"weaviate://localhost/d6f8b228-a2c6-3b67-a191-17f1c72983f4\",\n",
            "        \"href\": \"/v1/objects/d6f8b228-a2c6-3b67-a191-17f1c72983f4\"\n",
            "      }\n",
            "    ]\n",
            "  },\n",
            "  \"vectorWeights\": null\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "protected-vacuum"
      },
      "source": [
        "So Far, so Good (... So What!)\n",
        "\n",
        "There are more methods for Data Objects (`client.data_object`): `.delete`, `.exists`, `.replace` and `.update`.\n",
        "\n",
        "Also there are some methods for references too (`client.data_object.reference`): `.add`, `.delete` and `.update`."
      ],
      "id": "protected-vacuum"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "numerical-direction"
      },
      "source": [
        "### 5.4.2 Load data using batches\n",
        "(__Only in__ `weaviate-client` __version < 3.0.0.__)"
      ],
      "id": "numerical-direction"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "protecting-passion"
      },
      "source": [
        "Importing data in batches is very similar to adding object by object.\n",
        "\n",
        "The first thing we have to do is to create a `BatchRequest` object for each object type: `DataObject` and `Reference`. They are named accordingly: `ObjectsBatchRequest` and `ReferenceBatchRequest`.\n",
        "\n",
        "Lets create a object of each batch and import the next 99 articles to Weaviate.\n",
        "\n",
        "**NOTE:** I want to bring to your attention again that importing/creating data in batches skips some validation steps and might lead to a corrupted graph."
      ],
      "id": "protecting-passion"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nominated-southwest"
      },
      "source": [
        "from weaviate import ObjectsBatchRequest, ReferenceBatchRequest"
      ],
      "id": "nominated-southwest",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "essential-ideal"
      },
      "source": [
        "Lets create a function that adds a single article to the batch request."
      ],
      "id": "essential-ideal"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sound-austin"
      },
      "source": [
        "def add_article(batch: ObjectsBatchRequest, article_data: dict) -> str:\n",
        "    \n",
        "    article_object = {\n",
        "        'title': article_data['title'],\n",
        "        'summary': article_data['summary'].replace('\\n', '') # remove newline character\n",
        "    }\n",
        "    article_id = article_data['id']\n",
        "    \n",
        "    # add article to the object batch request\n",
        "    batch.add(\n",
        "        data_object=article_object,\n",
        "        class_name='Article',\n",
        "        uuid=article_id\n",
        "    )\n",
        "    \n",
        "    return article_id"
      ],
      "id": "sound-austin",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "seven-circus"
      },
      "source": [
        "Lets now create a function add a single author to the batch request, if the author was not already created."
      ],
      "id": "seven-circus"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "marked-brass"
      },
      "source": [
        "def add_author(batch: ObjectsBatchRequest, author_name: str, created_authors: dict) -> str:\n",
        "    \n",
        "    if author_name in created_authors:\n",
        "        # return author UUID\n",
        "        return created_authors[author_name]\n",
        "    \n",
        "    # generate an UUID for the Author\n",
        "    author_id = generate_uuid(author)\n",
        "    \n",
        "    # add author to the object batch request\n",
        "    batch.add(\n",
        "        data_object={'name': author_name},\n",
        "        class_name='Author',\n",
        "        uuid=author_id\n",
        "    )\n",
        "    \n",
        "    created_authors[author_name] = author_id\n",
        "    return author_id"
      ],
      "id": "marked-brass",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "stylish-overview"
      },
      "source": [
        "And the last function for adding cross references."
      ],
      "id": "stylish-overview"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "continued-glance"
      },
      "source": [
        "def add_references(batch: ReferenceBatchRequest, article_id: str, author_id: str)-> None:\n",
        "    # add references to the reference batch request\n",
        "    ## Author -> Article\n",
        "    batch.add(\n",
        "        from_object_uuid=author_id,\n",
        "        from_object_class_name='Author',\n",
        "        from_property_name='wroteArticles',\n",
        "        to_object_uuid=article_id\n",
        "    )\n",
        "    \n",
        "    ## Article -> Author \n",
        "    batch.add(\n",
        "        from_object_uuid=article_id,\n",
        "        from_object_class_name='Article',\n",
        "        from_property_name='hasAuthors',\n",
        "        to_object_uuid=author_id\n",
        "    )"
      ],
      "id": "continued-glance",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "adequate-holly"
      },
      "source": [
        "Now we can iterate through the data and import data using batches."
      ],
      "id": "adequate-holly"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "revised-trustee"
      },
      "source": [
        "from weaviate.tools import generate_uuid # in version 3.0.0 it is weaviate.util.generate_uuid5\n",
        "from tqdm.notebook import trange\n",
        "\n",
        "objects_batch = ObjectsBatchRequest()\n",
        "reference_batch = ReferenceBatchRequest()\n",
        "\n",
        "for i in trange(1, 100):\n",
        "    \n",
        "    # add article to batch request\n",
        "    article_id = add_article(objects_batch, data[i])\n",
        "    \n",
        "    for author in data[i]['authors']:\n",
        "        \n",
        "        # add author to batch request\n",
        "        author_id = add_author(objects_batch, author, created_authors)\n",
        "        \n",
        "        # add cross references to the reference batch\n",
        "        add_references(reference_batch, article_id=article_id, author_id=author_id)\n",
        "    \n",
        "    if i % 20 == 0:\n",
        "        # submit the object batch request to weaviate, can be done with method '.create_objects'\n",
        "        client.batch.create(objects_batch)\n",
        "        \n",
        "        # submit the reference batch request to weaviate, can be done with method '.create_references'\n",
        "        client.batch.create(reference_batch)\n",
        "        \n",
        "        # batch requests are not reusable, so we create new ones\n",
        "        objects_batch = ObjectsBatchRequest()\n",
        "        reference_batch = ReferenceBatchRequest()\n",
        "\n",
        "\n",
        "# submit the any object that are left\n",
        "status_objects = client.batch.create(objects_batch)\n",
        "status_references = client.batch.create(reference_batch)"
      ],
      "id": "revised-trustee",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lyric-belief"
      },
      "source": [
        "In order to import data in batches we should create a `BatchRequest` object for the data object type we want to import. A batch request object does not have a size limit so you should submit it when there are as many objects as you want. (Keep in mind that if you will use a batch with too many objects it might result in an TimeOut error so keep it to a reasonable size so your Weaviate instance can process it.) Also we keep track of the authors we already created so we do not create the same author over and over again.\n",
        "\n",
        "The call of the `client.batch.create` returns the status of each object that was created. Check it if you want to be sure that everything worked just fine. Also **NOTE** that even if Weaviate failed to create objects it does not mean that the batch submission failed too, for more information read the documentation of the `client.batch.create`."
      ],
      "id": "lyric-belief"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "advisory-newman"
      },
      "source": [
        "### 5.4.3 Load data using a Batcher object.\n",
        "(__Only in__ `weaviate-client` __version <3.0.0.__)"
      ],
      "id": "advisory-newman"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "raising-adams"
      },
      "source": [
        "The `Batcher` is a class that automatically submits objects to weaviate, both `DataObject`s and `Reference`s. The `Batcher` can be found in the `weaviate.tools` module, and has the following constructor prototype:\n",
        "```python\n",
        "Batcher(\n",
        "    client : weaviate.client.Client,\n",
        "    batch_size : int=512,\n",
        "    verbose : bool=False,\n",
        "    auto_commit_timeout : float=-1.0,\n",
        "    max_backoff_time : int=300,\n",
        "    max_request_retries : int=4,\n",
        "    return_values_callback : Callable=None,\n",
        ")\n",
        "```\n",
        "\n",
        "See the documentation for an explanation of each argument.\n",
        "\n",
        "Lets see how it works in action for the rest of the objects from `data` we extracted."
      ],
      "id": "raising-adams"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "formal-laugh"
      },
      "source": [
        "from weaviate.tools import Batcher"
      ],
      "id": "formal-laugh",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "looking-fireplace"
      },
      "source": [
        "For a `Batcher` we only need to add the objects we want to import to Weaviate. The `Batcher` has a special method to add `objects` (`batcher.add_data_object`) and a special method to add `references` (`batcher.add_reference`). Also it provides a `batcher.add` method that has keywords arguments, which detects what kind of data you are trying to add. The `batcher.add` method makes it possible to reuse the `add_article`, `add_author` and `add_references` functions we defined above.\n",
        "\n",
        "**NOTE:** The `Batcher.add` was introduced in `weaviate-client` version 2.3.0. "
      ],
      "id": "looking-fireplace"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "earlier-family"
      },
      "source": [
        "Lets use the batcher to add the remaining articles and authors from `data`. Because the `Batcher` automatically submits objects to weaviate, we need to ALWAYS `.close()` it after we are done to make sure we are submitting what remains in the `Batcher`.\n",
        "\n",
        "If you are like me, and sometimes forget to close objects, `Bather` can be used in a context manager, i.e. used with `with`. Lets see how it works with the context manager."
      ],
      "id": "earlier-family"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fossil-accident"
      },
      "source": [
        "# we still need the 'created_authors' so we do not add the same author twice\n",
        "with Batcher(client, 30, True) as batcher:\n",
        "    for i in trange(100, 200):\n",
        "        \n",
        "        # add article to batcher\n",
        "        article_id = add_article(batcher, data[i]) # NOTE the 'bather' object instead of 'objects_batch'\n",
        "\n",
        "        for author in data[i]['authors']:\n",
        "\n",
        "            # add author to batcher\n",
        "            author_id = add_author(batcher, author, created_authors) # NOTE the 'bather' object instead of 'objects_batch'\n",
        "\n",
        "            # add cross references to the batcher\n",
        "            add_references(batcher, article_id=article_id, author_id=author_id) # NOTE the 'bather' object instead of 'reference_batch'"
      ],
      "id": "fossil-accident",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "specified-article"
      },
      "source": [
        "That is it for a Batcher.\n",
        "\n",
        "These are the 3 ways to import data to Weaviate. Choose the one that is appropriate for you and your project."
      ],
      "id": "specified-article"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Djo_aUbaRyoh"
      },
      "source": [
        "### 5.4.4 New `Batch` object\n",
        "(__Only in__ `weaviate-client` __version >=3.0.0.__)"
      ],
      "id": "Djo_aUbaRyoh"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qbaSEv0BSLHA"
      },
      "source": [
        "The new `Batch` object is accessible the same way: `client.batch`. As described in 5.4 section this class can be used in 3 different ways, so lets see how exactly we can do it and what is the difference between them."
      ],
      "id": "qbaSEv0BSLHA"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2b3ifASOSt_c"
      },
      "source": [
        "The first thing we need to do is to re-define the following functions: `add_article`, `add_author` and `add_references`."
      ],
      "id": "2b3ifASOSt_c"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tueIIGFLVHh0"
      },
      "source": [
        "from weaviate.batch import Batch # for the typing purposes\n",
        "from weaviate.util import generate_uuid5 # old way was from weaviate.tools import generate_uuid\n",
        "\n",
        "\n",
        "def add_article(batch: Batch, article_data: dict) -> str:\n",
        "    \n",
        "    article_object = {\n",
        "        'title': article_data['title'],\n",
        "        'summary': article_data['summary'].replace('\\n', '') # remove newline character\n",
        "    }\n",
        "    article_id = article_data['id']\n",
        "    \n",
        "    # add article to the batch\n",
        "    batch.add_data_object(  # old way was batch.add(...)\n",
        "        data_object=article_object,\n",
        "        class_name='Article',\n",
        "        uuid=article_id\n",
        "    )\n",
        "    \n",
        "    return article_id\n",
        "\n",
        "def add_author(batch: Batch, author_name: str, created_authors: dict) -> str:\n",
        "    \n",
        "    if author_name in created_authors:\n",
        "        # return author UUID\n",
        "        return created_authors[author_name]\n",
        "    \n",
        "    # generate an UUID for the Author\n",
        "    author_id = generate_uuid5(author)\n",
        "    \n",
        "    # add author to the batch\n",
        "    batch.add_data_object(  # old way was batch.add(...)\n",
        "        data_object={'name': author_name},\n",
        "        class_name='Author',\n",
        "        uuid=author_id\n",
        "    )\n",
        "    \n",
        "    created_authors[author_name] = author_id\n",
        "    return author_id\n",
        "\n",
        "def add_references(batch: Batch, article_id: str, author_id: str)-> None:\n",
        "    # add references to the batch\n",
        "    ## Author -> Article\n",
        "    batch.add_reference(  # old way was batch.add(...)\n",
        "        from_object_uuid=author_id,\n",
        "        from_object_class_name='Author',\n",
        "        from_property_name='wroteArticles',\n",
        "        to_object_uuid=article_id\n",
        "    )\n",
        "    \n",
        "    ## Article -> Author \n",
        "    batch.add_reference(  # old way was batch.add(...)\n",
        "        from_object_uuid=article_id,\n",
        "        from_object_class_name='Article',\n",
        "        from_property_name='hasAuthors',\n",
        "        to_object_uuid=author_id\n",
        "    )"
      ],
      "id": "tueIIGFLVHh0",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PC4BPLqqTiVN"
      },
      "source": [
        "Now that we changed the above functions to be compatible with the new `Batch` object, lets see them in action."
      ],
      "id": "PC4BPLqqTiVN"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iv766_9U01es"
      },
      "source": [
        "#### a) Manually"
      ],
      "id": "iv766_9U01es"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7pfbRm-2Xb19"
      },
      "source": [
        "This method gives the user the absolute control of when and how to add and create batches. It is very similar to the `BatchRequests` method of the `weaviate-client` version < 3.0.0, so lets take a look how exactly we can use it (or migrate to it from the old version)."
      ],
      "id": "7pfbRm-2Xb19"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SQBkHOcuZvLy"
      },
      "source": [
        "See the context manager way of the same method below the following code cell.\n",
        "(__Do not run both cells!__)"
      ],
      "id": "SQBkHOcuZvLy"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Fd06g8uUl65"
      },
      "source": [
        "from tqdm import trange\n",
        "\n",
        "for i in trange(1, 100):\n",
        "    \n",
        "    # add article to the batch\n",
        "    article_id = add_article(client.batch, data[i])\n",
        "    \n",
        "    for author in data[i]['authors']:\n",
        "        \n",
        "        # add author to the batch\n",
        "        author_id = add_author(client.batch, author, created_authors)\n",
        "        \n",
        "        # add cross references to the batch\n",
        "        add_references(client.batch, article_id=article_id, author_id=author_id)\n",
        "    \n",
        "    if i % 20 == 0:\n",
        "        # submit the objects from the batch to weaviate\n",
        "        client.batch.create_objects()\n",
        "        \n",
        "        # submit the references from the batch to weaviate\n",
        "        client.batch.create_references()\n",
        "\n",
        "\n",
        "# submit any objects that are left\n",
        "status_objects = client.batch.create_objects()\n",
        "status_references = client.batch.create_references()\n",
        "# if there is no need for the output from batch creation, one could flush both\n",
        "# object and references with one call\n",
        "client.batch.flush()"
      ],
      "id": "0Fd06g8uUl65",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FxxDNbDcZDDS"
      },
      "source": [
        "Alternatively, we could use the `Batch` instance with a context manager, that would call the `flush()` method when it exists the context. "
      ],
      "id": "FxxDNbDcZDDS"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o8xmN0qwZcv1",
        "outputId": "f4da75aa-ed21-43eb-81f6-c2ee07367885"
      },
      "source": [
        "from tqdm import trange\n",
        "\n",
        "with client.batch as batch:\n",
        "    for i in trange(1, 100):\n",
        "        \n",
        "        # add article to the batch\n",
        "        article_id = add_article(batch, data[i])\n",
        "        \n",
        "        for author in data[i]['authors']:\n",
        "            \n",
        "            # add author to the batch\n",
        "            author_id = add_author(batch, author, created_authors)\n",
        "            \n",
        "            # add cross references to the batch\n",
        "            add_references(batch, article_id=article_id, author_id=author_id)\n",
        "        \n",
        "        if i % 20 == 0:\n",
        "            # submit the objects from batch to weaviate\n",
        "            batch.create_objects()\n",
        "            \n",
        "            # submit the references from batch to weaviate\n",
        "            batch.create_references()"
      ],
      "id": "o8xmN0qwZcv1",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "  0%|          | 0/99 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 20%|██        | 20/99 [00:00<00:03, 22.41it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 40%|████      | 40/99 [00:01<00:02, 22.52it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 61%|██████    | 60/99 [00:02<00:01, 21.77it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "100%|██████████| 99/99 [00:03<00:00, 26.42it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xxzHNLBb1ga3"
      },
      "source": [
        "#### b) Auto-create batches when full"
      ],
      "id": "xxzHNLBb1ga3"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RhZ0pOorSvJ1"
      },
      "source": [
        "This method is similar to the `weaviate-client`version < 3.0.0 's `Batcher` object. Lets see how it works."
      ],
      "id": "RhZ0pOorSvJ1"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GZ08qNJcaEQS"
      },
      "source": [
        "# we still need the 'created_authors' so we do not add the same author twice\n",
        "client.batch.configure(\n",
        "    batch_size=30,\n",
        "    callback=None, # use this argument to set a callback function on the batch creation results\n",
        ")\n",
        "for i in trange(100, 200):\n",
        "    \n",
        "    # add article to batcher\n",
        "    article_id = add_article(client.batch, data[i])\n",
        "\n",
        "    for author in data[i]['authors']:\n",
        "\n",
        "        # add author to the batch\n",
        "        author_id = add_author(client.batch, author, created_authors)\n",
        "\n",
        "        # add cross references to the batch\n",
        "        add_references(client.batch, article_id=article_id, author_id=author_id)\n",
        "client.batch.flush()"
      ],
      "id": "GZ08qNJcaEQS",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O68jcW3NbG7x"
      },
      "source": [
        "Of course we could use a context manager here too."
      ],
      "id": "O68jcW3NbG7x"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GFCkEFwXbLmV",
        "outputId": "9c0ed2c1-924a-4ff1-eff2-44434b2cf850"
      },
      "source": [
        "# we still need the 'created_authors' so we do not add the same author twice\n",
        "client.batch.configure(\n",
        "    batch_size=30,\n",
        "    callback=None, # use this argument to set a callback function on the batch creation results\n",
        ")\n",
        "with client.batch(batch_size=30) as batch: # the client.batch(batch_size=30) is the same as client.batch.configure(batch_size=30)\n",
        "    for i in trange(100, 200):\n",
        "\n",
        "        # add article to the batch\n",
        "        article_id = add_article(batch, data[i])\n",
        "\n",
        "        for author in data[i]['authors']:\n",
        "\n",
        "            # add author to the batch\n",
        "            author_id = add_author(batch, author, created_authors)\n",
        "\n",
        "            # add cross references to the batch\n",
        "            add_references(batch, article_id=article_id, author_id=author_id)"
      ],
      "id": "GFCkEFwXbLmV",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "  0%|          | 0/100 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "  6%|▌         | 6/100 [00:00<00:07, 12.99it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 13%|█▎        | 13/100 [00:00<00:06, 13.58it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 19%|█▉        | 19/100 [00:01<00:06, 12.96it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 24%|██▍       | 24/100 [00:01<00:06, 11.77it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 25%|██▌       | 25/100 [00:02<00:13,  5.70it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 30%|███       | 30/100 [00:02<00:10,  6.51it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 37%|███▋      | 37/100 [00:03<00:08,  7.31it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 43%|████▎     | 43/100 [00:04<00:07,  7.88it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 48%|████▊     | 48/100 [00:04<00:06,  8.11it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 55%|█████▌    | 55/100 [00:05<00:05,  8.88it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 64%|██████▍   | 64/100 [00:06<00:03,  9.84it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 72%|███████▏  | 72/100 [00:06<00:02, 10.59it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 80%|████████  | 80/100 [00:07<00:01, 11.27it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 85%|████████▌ | 85/100 [00:07<00:01,  9.79it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 89%|████████▉ | 89/100 [00:08<00:01,  8.61it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "100%|██████████| 100/100 [00:09<00:00, 10.90it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HmTrJBQi13cw"
      },
      "source": [
        "#### c) Auto-create batches using dynamic batching, i.e. the batch size is adjusted every time it is created to avoid any `Timeout` errors.\n"
      ],
      "id": "HmTrJBQi13cw"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ORiOhnFYkP-"
      },
      "source": [
        "This method works in the same manner as the method described in b). We are not going to run any cell with it, but I am going to mention that to enable the dynamic batching, all one need to do is to provide another argument to the `configure`/`__call__` method.\n",
        "\n",
        "Example:\n",
        "```python\n",
        "client.batch.configure(\n",
        "    batch_size=30,\n",
        "    dynamic=True\n",
        ")\n",
        "```"
      ],
      "id": "_ORiOhnFYkP-"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NPnc9pbacVbz"
      },
      "source": [
        "To see the full capabilities of this new `Batch` object see the full documentation [here](https://weaviate-python-client.readthedocs.io/en/latest/weaviate.batch.html#weaviate.batch.Batch) or execute the `help` function on `Batch` or/and any `Batch` methods, like this: `help(Batch)`"
      ],
      "id": "NPnc9pbacVbz"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elder-roommate"
      },
      "source": [
        "### 5.5. Query data."
      ],
      "id": "elder-roommate"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "federal-fitting"
      },
      "source": [
        "Now we have the data imported and ready to be queried. Data can be queried by using the `query` attribute of the client object (`client.query`).\n",
        "\n",
        "The data is queried using GraphQL syntax, and can be done in three different ways:\n",
        "- **GET**: query that gets objects and from Weaviate. More information [here](https://www.semi.technology/developers/weaviate/current/graphql-references/get.html)<br>\n",
        "    Use `client.query.get(class_name, properties).OTHER_OPTIONAL_FILTERS.do()`\n",
        "\n",
        "- **AGGREGATE**: query that aggregates data. More information [here](https://www.semi.technology/developers/weaviate/current/graphql-references/aggregate.html) <br>\n",
        "    Use `client.query.aggregate(class_name, properties).OTHER_OPTIONAL_FILTERS.do()`\n",
        "    \n",
        "- Or use a GraphQL query represented as a `str`. <br>\n",
        "    Use `client.query.raw()`\n",
        "    \n",
        "**NOTE:** Both `.get` and `.aggregate` require the call of the `.do()` method to run the query. `.raw()` does NOT."
      ],
      "id": "federal-fitting"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "three-snake"
      },
      "source": [
        "Lets now get the Articles objects and their corresponding title only."
      ],
      "id": "three-snake"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "growing-liquid"
      },
      "source": [
        "### 5.5.1 GET"
      ],
      "id": "growing-liquid"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "obvious-gardening",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "118d8e3d-03c7-43b7-ae5a-054b27e3bf7e"
      },
      "source": [
        "result = client.query.get(class_name='Article', properties=\"title\")\\\n",
        "    .do()\n",
        "print(f\"Number of articles returned: {len(result['data']['Get']['Article'])}\")\n",
        "result"
      ],
      "id": "obvious-gardening",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of articles returned: 100\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'data': {'Get': {'Article': [{'title': 'Incredible before-and-after imagery shows impact of California wildfires and drought'},\n",
              "    {'title': 'Texans march on capitol to protect voting rights – will Washington listen?'},\n",
              "    {'title': \"A checklist of what to do -- and pack -- when a hurricane's coming\"},\n",
              "    {'title': 'Why right to repair matters – according to a farmer, a medical worker, a computer store owner'},\n",
              "    {'title': 'African-born UFC champions the stars of a new era of fighters from continent'},\n",
              "    {'title': 'US launches emergency airlift to rescue Afghan allies at risk of Taliban’s revenge'},\n",
              "    {'title': 'Wildlife flourishing in uninhabited areas around Fukushima'},\n",
              "    {'title': 'Self-driving robotaxis are taking off in China'},\n",
              "    {'title': 'Sadio Mane donates $693,000 to fund hospital in his hometown in Senegal'},\n",
              "    {'title': \"Not all (internet) heroes wear capes. Meet the 'Super Reviewers'\"},\n",
              "    {'title': 'Climate crisis has cost Colorado billions – now it wants oil firms to pick up the bill'},\n",
              "    {'title': 'Fauci backs new masks guidance as Florida reports highest one-day Covid case total'},\n",
              "    {'title': 'Tunisia shows that democracy will struggle if it can’t deliver prosperity'},\n",
              "    {'title': \"UK sanctions Equatorial Guinea leader's son over 'lavish lifestyle' spending\"},\n",
              "    {'title': \"Bob Odenkirk thanks fans for 'outpouring of love' following health scare\"},\n",
              "    {'title': 'HSBC boosts bonus pool by 50% as profits more than quadruple'},\n",
              "    {'title': \"How synesthesia helps artist Prince Gyasi turn his work into 'color therapy'\"},\n",
              "    {'title': 'Australia v Sweden: Tokyo Olympics 2020 women’s football semi-final – live!'},\n",
              "    {'title': 'Resurgent Taliban escalates nationwide offensive in Afghanistan'},\n",
              "    {'title': \"Ethiopia is at war with itself. Here's what you need to know\"},\n",
              "    {'title': 'China debuts train prototype that can hit speeds of 620 kilometers per hour'},\n",
              "    {'title': 'Share a story with the Guardian'},\n",
              "    {'title': \"India's stray dogs are being locked out by a US ban on adoptions\"},\n",
              "    {'title': 'Greek police recover stolen Picasso after nine years, drop it in front of media – video'},\n",
              "    {'title': 'Rojas thanks Facebook algorithm after making history in triple jump'},\n",
              "    {'title': \"'Old' review: Director M. Night Shyamalan is up to the same old tricks\"},\n",
              "    {'title': \"'Selling Sunset' stars Chrishell Stause and Jason Oppenheim are dating\"},\n",
              "    {'title': 'Rishi Sunak pushing Boris Johnson to relax travel rules'},\n",
              "    {'title': 'Fruit baskets from fourth century BC found in ruins of Thonis-Heracleion'},\n",
              "    {'title': \"A 5-story building in Shanghai 'walks' to a new location\"},\n",
              "    {'title': 'Jessie Cave on body image, bereavement and being relentless: ‘I don’t have any secrets’'},\n",
              "    {'title': \"'Ted Lasso:' A great show, or just the right show at the time we needed it?\"},\n",
              "    {'title': 'Rhik Samadder tries … mindful painting: ‘It dawns on me that I’m a very talented artist’'},\n",
              "    {'title': 'China releases footage from its Mars rover – video'},\n",
              "    {'title': 'Ex-President Obama joins NBA Africa as strategic partner and minority owner'},\n",
              "    {'title': 'MPs who sit on parliamentary groups face scrutiny over lobbying'},\n",
              "    {'title': \"'The Green Knight' takes a dream-like trip through the Arthurian legend\"},\n",
              "    {'title': 'Ethiopia: UN running out of food in Tigray region as 400,000 people face famine'},\n",
              "    {'title': \"Meet the South African singer taking on China's pop scene\"},\n",
              "    {'title': 'How climate crisis made my British butterfly hunt a race against time'},\n",
              "    {'title': 'Tell us: have you been affected by the spectator ban at the Tokyo Olympics?'},\n",
              "    {'title': 'Raven Saunders throws up X on podium to represent where the ‘oppressed meet’'},\n",
              "    {'title': 'Belarus sprinter safe under Tokyo police protection after refusing to go home'},\n",
              "    {'title': 'All the butter: how chefs are transforming the world’s greatest spread'},\n",
              "    {'title': 'Brexit and Covid have created the perfect moment for the politics of crackdown | John Harris'},\n",
              "    {'title': 'Tyrone Mings opens up about mental health struggles at Euro 2020'},\n",
              "    {'title': 'Ocon claims Hungarian Grand Prix as Hamilton gets promoted to second'},\n",
              "    {'title': 'All the Black women in us are tired'},\n",
              "    {'title': 'Dominican Republic merengue and music legend Johnny Ventura dies at 81'},\n",
              "    {'title': 'Johnson faces rebellion over ‘intolerable’ hunger and poverty in home counties'},\n",
              "    {'title': \"From the 'C-Stunner' to the 'Black Mamba,' Kenya's Cyrus Kabiru rides wave of success with new creations made from scrap\"},\n",
              "    {'title': \"Better than Schick? Player scores from centre circle in Ghana's top flight – video\"},\n",
              "    {'title': 'Can a motorcycle-hailing company create an African super app?'},\n",
              "    {'title': \"Instagram influencer 'Hushpuppi' pleads guilty to million-dollar fraud in US\"},\n",
              "    {'title': \"South Africa's 'underground astronauts': Exploring dark corridors in cramped conditions\"},\n",
              "    {'title': 'How we met: ‘She gave me strong signs that she wasn’t looking for romance’'},\n",
              "    {'title': 'Why did Unesco drop Liverpool from its heritage list?'},\n",
              "    {'title': 'Team GB strike gold in Olympic team eventing'},\n",
              "    {'title': 'High-elevation forests in the Rockies are burning more now than in the past 2,000 years'},\n",
              "    {'title': 'Temperatures in the West expected to climb to records as drought conditions worsen'},\n",
              "    {'title': 'US vaccinations rise but White House frustrated with media ‘alarmism’'},\n",
              "    {'title': 'UK factories face shortages and cost increases as supply problems deepen – business live'},\n",
              "    {'title': \"California wildfire season: It's coming quickly and it's coming earlier, forecasters warn\"},\n",
              "    {'title': \"Will Smith stars as Williams sisters' father in 'King Richard'\"},\n",
              "    {'title': 'The Greek alphabet will never be used again to name tropical storms'},\n",
              "    {'title': 'Tigray massacre: How an Ethiopian festival turned into a killing spree'},\n",
              "    {'title': 'USA stunned by Jessie Fleming penalty as Canada reach Olympic soccer final'},\n",
              "    {'title': 'Travis Scott drops surprise new Fragment merch'},\n",
              "    {'title': 'Inside the eye of a hurricane from above 1,000 feet'},\n",
              "    {'title': 'Dolly Parton supports Britney Spears and can relate'},\n",
              "    {'title': 'Thomasina Miers’ recipe for cinnamon semifreddo with chocolate drizzle and raspberries'},\n",
              "    {'title': 'Leaders tackle global question of how to persuade people to get Covid jab'},\n",
              "    {'title': 'Tunisian democracy in crisis after president ousts government'},\n",
              "    {'title': \"TV OT: From 'Physical' to 'White Lotus,' quiet rage is all the rage. Plus, more to 'toon' into\"},\n",
              "    {'title': 'Is Zola’s use of a ‘blaccent’ parody or appropriation?'},\n",
              "    {'title': 'Sharks fleeing toxic red tide take refuge in Florida canal'},\n",
              "    {'title': 'Tanzania finally launches Covid-19 vaccine drive after year-long denial'},\n",
              "    {'title': \"Forgotten for half a century, this rare bean could save Sierra Leone's coffee industry\"},\n",
              "    {'title': \"Inside the final resting place of Tutankhamun's treasures\"},\n",
              "    {'title': 'Meet the crossword setter, plant scientist Soup: ‘A successful clue needs pith and, ideally, humour’'},\n",
              "    {'title': 'Sri Lanka reopens borders to international travelers'},\n",
              "    {'title': \"Bob Odenkirk stable condition after collapsing on 'Better Call Saul' set\"},\n",
              "    {'title': 'Premier League 2021-22 preview No 1: Arsenal'},\n",
              "    {'title': 'Tunisia’s Saied moves on economy and Covid-19 after dismissing government'},\n",
              "    {'title': 'India must fix their misfiring top order to trouble England on home turf'},\n",
              "    {'title': 'Equatorial Guinea VP loses appeal against French embezzlement verdict'},\n",
              "    {'title': 'NBC paid $7.75bn for its Olympic rights ... and we got televisual vomit'},\n",
              "    {'title': 'Pfizer-BioNTech to start producing Covid-19 vaccines in South Africa in 2022'},\n",
              "    {'title': 'National treasures: posters celebrating US parks\\u200b – in pictures'},\n",
              "    {'title': 'Scientists are trying to control lightning with a giant laser'},\n",
              "    {'title': \"Japanese man invents 'edible' plastic bag alternative to save Nara's sacred deer\"},\n",
              "    {'title': 'A Malawian farmer visiting the US wants to know: ‘Why not do more on the climate crisis?’'},\n",
              "    {'title': 'Germany floods: stranded residents rescued by helicopter from rooftops – video'},\n",
              "    {'title': \"Climate change is fueling mass-casualty heat waves. Here's why experts say we don't view them as crises\"},\n",
              "    {'title': 'Hurricane season is forecast to be above average. So are the hurricane forecasts'},\n",
              "    {'title': 'Travel to Japan during Covid-19: What you need to know before you go'},\n",
              "    {'title': 'Dawn Raids apology and an injured cat in Turkey: Weekend’s best photos'},\n",
              "    {'title': \"Meet Diarra Bousso: One of Senegal's most promising designers\"},\n",
              "    {'title': 'Searching for a slice of paradise? Mauritius is looking to lure international homebuyers'},\n",
              "    {'title': \"WNBA players Nneka Ogwumike and Elizabeth Williams won't play for Nigeria at Olympics after appeal rejected\"}]}}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cathedral-heart"
      },
      "source": [
        "So as we can see the `result` contains only 100 articles, this is due to the default limit of 100. Lets change it."
      ],
      "id": "cathedral-heart"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "changed-rabbit",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2e0f4938-941f-49c2-9017-ab8ecec2c398"
      },
      "source": [
        "result = client.query.get(class_name='Article', properties=\"title\")\\\n",
        "    .with_limit(200)\\\n",
        "    .do()\n",
        "print(f\"Number of articles returned: {len(result['data']['Get']['Article'])}\")"
      ],
      "id": "changed-rabbit",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of articles returned: 200\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "equipped-gauge"
      },
      "source": [
        "We can do much more by stacking multiple methods. The available methods for `.get` are:\n",
        "- `.with_limit` - set another limit of returned objects.\n",
        "- `.with_near_object` - get objects that are similar to the object passed to this method.\n",
        "- `.with_near_text` - get objects that are similar to the text passed to this method.\n",
        "- `.with_near_vector` - get objects that are similar to the vector passed to this method.\n",
        "- `.with_where` - get objects that are filtered using the `Where` filter, see this [link](https://www.semi.technology/developers/weaviate/current/graphql-references/filters.html#where-filter) for examples and explanation.\n",
        "\n",
        "Also instead of `.do()` one can use the `.build()` method that returns the GraphQL query as a string. This string can be passed to `.raw()` method.\n",
        "\n",
        "**NOTE:** Only one `.with_near_*` can be used per query."
      ],
      "id": "equipped-gauge"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ambient-example",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f01c3d63-a496-4254-f730-6d78f79426ed"
      },
      "source": [
        "client.query.get(class_name='Article', properties=\"title\")\\\n",
        "    .with_limit(5)\\\n",
        "    .with_near_text({'concepts': ['Fashion']})\\\n",
        "    .do()"
      ],
      "id": "ambient-example",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'data': {'Get': {'Article': [{'title': \"Excited for 'House of Gucci'? Here's what to watch to get ready\"},\n",
              "    {'title': \"Meet Diarra Bousso: One of Senegal's most promising designers\"},\n",
              "    {'title': 'Returning office workers seek sweet spot between casual and formal'},\n",
              "    {'title': 'How a Nigerian model used a Russian beauty pageant and a bloodied flag to protest police brutality'},\n",
              "    {'title': \"Meet the South African singer taking on China's pop scene\"}]}}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mental-mississippi"
      },
      "source": [
        "With `Get` we can see the cross references of each object. We are going to use the `.raw()` method for this since it is not possible with any existing `.with_*` method."
      ],
      "id": "mental-mississippi"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "regional-right",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "af8134d7-7250-4b4b-8e43-aad17c437654"
      },
      "source": [
        "query = \"\"\"\n",
        "{\n",
        "  Get {\n",
        "    Article(limit: 2) {\n",
        "      title\n",
        "\n",
        "      hasAuthors {         # the reference\n",
        "        ... on Author {    # you always set the destination class\n",
        "          name             # the property related to target class\n",
        "        }\n",
        "      }\n",
        "    }\n",
        "  }\n",
        "}\n",
        "\"\"\"\n",
        "\n",
        "prettify(client.query.raw(query)['data']['Get']['Article'])"
      ],
      "id": "regional-right",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[\n",
            "  {\n",
            "    \"hasAuthors\": [\n",
            "      {\n",
            "        \"name\": \"Angela Fritz\"\n",
            "      }\n",
            "    ],\n",
            "    \"title\": \"Incredible before-and-after imagery shows impact of California wildfires and drought\"\n",
            "  },\n",
            "  {\n",
            "    \"hasAuthors\": [\n",
            "      {\n",
            "        \"name\": \"Alexandra Villarreal\"\n",
            "      }\n",
            "    ],\n",
            "    \"title\": \"Texans march on capitol to protect voting rights \\u2013 will Washington listen?\"\n",
            "  }\n",
            "]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "northern-oxide"
      },
      "source": [
        "### 5.5.2 AGGREGATE"
      ],
      "id": "northern-oxide"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "later-azerbaijan"
      },
      "source": [
        "We can use the `.aggregate` to count number of objects that satisfy a specific condition."
      ],
      "id": "later-azerbaijan"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "therapeutic-shame",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b554dd61-1174-44fe-a088-b25a294250da"
      },
      "source": [
        "# no filter, count all objects of class Article\n",
        "client.query.aggregate(class_name='Article')\\\n",
        "    .with_meta_count()\\\n",
        "    .do()"
      ],
      "id": "therapeutic-shame",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'data': {'Aggregate': {'Article': [{'meta': {'count': 200}}]}}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "heated-tractor",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b7c820b1-1f2a-4531-98f3-a887fb4ff1ee"
      },
      "source": [
        "# no filter, count all objects of class Author\n",
        "client.query.aggregate(class_name='Author')\\\n",
        "    .with_meta_count()\\\n",
        "    .do()"
      ],
      "id": "heated-tractor",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'data': {'Aggregate': {'Author': [{'meta': {'count': 219}}]}}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "equal-valentine"
      },
      "source": [
        "\n",
        "Here are the methods that are supported by the `.aggregate`.\n",
        "\n",
        "- `.with_meta_count` sets meta count to True. Used to count objects per filtered group.\n",
        "- `.with_fields` - fields to return by the aggregated query.\n",
        "- `.with_group_by_filter` - set a `GroupBy` filter. See this [link](https://www.semi.technology/developers/weaviate/current/graphql-references/aggregate.html#groupby-filter) for more information about the filter.\n",
        "- `.with_where` - aggregate objects using a `Where` filter. See this [link](https://www.semi.technology/developers/weaviate/current/graphql-references/filters.html#where-filter) for examples and explanation.\n"
      ],
      "id": "equal-valentine"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "prescription-scanner"
      },
      "source": [
        "Of course when it comes to querying data, the possibilities are endless. Have fun experimenting with these capabilities.\n",
        "\n",
        "Feel free to check out and contribute to weaviate-client on [GitHub](https://github.com/semi-technologies/weaviate-python-client)."
      ],
      "id": "prescription-scanner"
    }
  ]
}