example_output/Schema_Validation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Schema Validation",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/cw00dw0rd/ArangoNotebooks/blob/master/example_output/Schema_Validation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jo4KZSZqpVEX",
        "colab_type": "text"
      },
      "source": [
        "Being schema-less and allowing for flexible documents is one of big advantages of ArangoDB. But sometimes there are use-cases where we have a fixed schema. With the upcoming release 3.7, ArangoDB provides users with the means to validate the structure of documents using JSON Schema (draft-4).\n",
        "\n",
        "The validation can be configured on collection level and with different strictness levels allowing users to import unclean data and improve later.\n",
        "\n",
        "Let us consider a concrete example which you can directly try interactively below. You can click the \"Open In Colab\" button to try it yourself or see the static output throughout the post. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TyUILq_opxW_",
        "colab_type": "text"
      },
      "source": [
        "First things first, we need to install and import necessary packages. This notebook also creates a temporary database running on ArangoDB Oasis. \n",
        "\n",
        "If you run this notebook you will be provided a fully functional ArangoDB deployment hostname, username, and password. These credentials, along with the deployment itself will be automatically deleted."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HjI338Lani7U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!git clone -b oasisConnector --single-branch https://github.com/cw00dw0rd/ArangoNotebooks.git\n",
        "!rsync -av ArangoNotebooks/ ./ --exclude=.git\n",
        "!pip3 install pyarango\n",
        "!pip3 install \"python-arango>=5.0\""
      ],
      "execution_count": 79,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rm_ntT1joRwG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "60207c61-c99a-4410-e982-27b140eafc70"
      },
      "source": [
        "import json\n",
        "import requests\n",
        "import sys\n",
        "import pprint\n",
        "import oasis\n",
        "\n",
        "from os import path\n",
        "from pyArango.connection import *\n",
        "from pyArango.collection import Collection, Edges, Field\n",
        "from pyArango.graph import Graph, EdgeDefinition\n",
        "from pyArango.collection import BulkOperation as BulkOperation\n",
        "\n",
        "pp = pprint.PrettyPrinter()\n",
        "\n",
        "# Retrieve tmp credentials from ArangoDB Tutorial Service\n",
        "login = oasis.getTempCredentials(tutorialName='schemaValidation37', tempURL='https://tutorials.arangodb.cloud:8529/_db/_system/tutorialDB/tutorialDB')\n",
        "\n",
        "## Connect to the temp database\n",
        "conn = oasis.connect(login)\n",
        "db = conn[login[\"dbName\"]] \n",
        "\n",
        "def cleanupCollections(db):\n",
        "  try:\n",
        "    db['Customers'].delete()\n",
        "  except:\n",
        "    pass\n",
        "  db.reload()\n",
        "\n",
        "# Cleanup (just in case the example is rerun)\n",
        "cleanupCollections(db)\n",
        "\n",
        "# Generating temporary credentials, run the notebook to generate your own\n",
        "pp.pprint(login)"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requesting new temp credentials.\n",
            "Temp database ready to use.\n",
            "{'dbName': 'TUT74fbdhqtzjtp4j3yk293h',\n",
            " 'hostname': 'tutorials.arangodb.cloud',\n",
            " 'password': 'TUT868nebwahwwozqdi75z0eb',\n",
            " 'port': 8529,\n",
            " 'username': 'TUThf37jys3b5d8xxeaepo9hl'}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "heyiUn-WshhA",
        "colab_type": "text"
      },
      "source": [
        "Let us consider a concrete example, assume you have a collection containing customers and leads. For now you have only names and mail addresses for most of the customers similar as shown below. Here we are adding in some customer information, taking note of the missing email for James. We follow this up with an AQL query that returns all of the customers in our newly created Customers collection."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t9WbKDefo0mc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "8bfcf9c6-cf5e-4ca9-f7d0-1ef846d21ff6"
      },
      "source": [
        "# Create Customer Collection\n",
        "collection = db.createCollection(name=\"Customers\")\n",
        "\n",
        "# insert some documents\n",
        "docs = []\n",
        "doc = collection.createDocument()\n",
        "doc[\"firstName\"] = \"James\"\n",
        "doc[\"lastName\"] = \"Cole\"\n",
        "docs.append(doc)\n",
        "\n",
        "doc = collection.createDocument()\n",
        "doc[\"firstName\"] = \"Claudius\"\n",
        "doc[\"lastName\"] = \"Weinberger\"\n",
        "doc[\"email\"] = \"info@arango.com\"\n",
        "docs.append(doc)\n",
        "collection.bulkSave(docs)\n",
        "\n",
        "# Check customers\n",
        "print(\"Check Customers\")\n",
        "aql = \"\"\"\n",
        "  FOR customer in Customers\n",
        "    return customer\n",
        "  \"\"\"\n",
        "queryResult = db.AQLQuery(aql)\n",
        "for customer in queryResult:\n",
        "   print(customer)"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Check Customers\n",
            "ArangoDoc '_id: Customers/312077378, _key: 312077378, _rev: _a5RAMz----': <store: {'firstName': 'James', 'lastName': 'Cole'}>\n",
            "ArangoDoc '_id: Customers/312077379, _key: 312077379, _rev: _a5RAMz---A': <store: {'email': 'info@arango.com', 'firstName': 'Claudius', 'lastName': 'Weinberger'}>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mv-SMjZh4w_P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Drop the collection for the next example.\n",
        "db[\"Customers\"].delete() # drop\n",
        "db.reloadCollections() "
      ],
      "execution_count": 82,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KMwqbES7vy9J",
        "colab_type": "text"
      },
      "source": [
        "Next, we will add validation for this collection. This is done by providing a well formed object, such as the one below, to the validation attribute of the collection properties. \n",
        "You could add this using arangosh with the `db.customers.properties(<object>)` command.\n",
        "\n",
        "The following example validation object sets a message that will be displayed on validation failure. The level \"moderate\" allows us to work with old documents that are not conforming to the schema. Such documents can be updated but no new invalid objects can be added or objects that are valid can be changed to be invalid. The last attribute which is required is the rule attribute containing a valid JSON Schema description."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mcbniAGy60K7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Example schema validation object\n",
        "schema = {\n",
        "    \"rule\" : {\n",
        "      \"type\" : \"object\",\n",
        "      \"properties\": {\n",
        "        \"firstName\": {\n",
        "            \"type\": \"string\",\n",
        "        },\n",
        "        \"lastName\": {\n",
        "            \"type\": \"string\",\n",
        "        },\n",
        "        \"email\": {\n",
        "            \"type\": \"string\",\n",
        "        },\n",
        "    },\n",
        "    \"required\" : [\"firstName\", \"lastName\", \"email\"],\n",
        "  },\n",
        "  \"level\": \"moderate\",\n",
        "  \"message\": \"Customer Schema Validation Failed.\"\n",
        "}\n",
        "\n",
        "\n",
        "# Recreate Customers collection, now with moderate schema validation\n",
        "collection = db.createCollection(\n",
        "        name = \"Customers\",\n",
        "        schema = schema\n",
        "    )\n"
      ],
      "execution_count": 83,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVovMiA667DH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 510
        },
        "outputId": "41ed3954-964e-45f0-f8de-a95ea62f8962"
      },
      "source": [
        "# Confirm validation has been added by checking collection properties\n",
        "collectionProperties = collection.properties()\n",
        "pp.pprint(collectionProperties)"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'cacheEnabled': False,\n",
            " 'code': 200,\n",
            " 'distributeShardsLike': '_graphs',\n",
            " 'error': False,\n",
            " 'globallyUniqueId': 'c312077380/',\n",
            " 'id': '312077380',\n",
            " 'isDisjoint': False,\n",
            " 'isSmart': False,\n",
            " 'isSmartChild': False,\n",
            " 'isSystem': False,\n",
            " 'keyOptions': {'allowUserKeys': True, 'type': 'traditional'},\n",
            " 'minReplicationFactor': 1,\n",
            " 'name': 'Customers',\n",
            " 'numberOfShards': 1,\n",
            " 'replicationFactor': 3,\n",
            " 'schema': {'level': 'moderate',\n",
            "            'message': 'Customer Schema Validation Failed.',\n",
            "            'rule': {'properties': {'email': {'type': 'string'},\n",
            "                                    'firstName': {'type': 'string'},\n",
            "                                    'lastName': {'type': 'string'}},\n",
            "                     'required': ['firstName', 'lastName', 'email'],\n",
            "                     'type': 'object'}},\n",
            " 'shardKeys': ['_key'],\n",
            " 'shardingStrategy': 'hash',\n",
            " 'status': 3,\n",
            " 'statusString': 'loaded',\n",
            " 'type': 2,\n",
            " 'waitForSync': False,\n",
            " 'writeConcern': 1}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8-697FtXMtJC",
        "colab_type": "text"
      },
      "source": [
        "If we now try to insert a document where either the names aren't strings, the email is not a string, or the document doesn't include the required attributes, the insert will fail with the validation failed message specified in the validation object."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uwx2sei2o_KW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "5aff8c42-06d9-4485-98ab-715c271df02c"
      },
      "source": [
        "#  Try to insert the same documents, now that the James document doesn't match the schema\n",
        "docs = []\n",
        "doc = collection.createDocument()\n",
        "doc[\"firstName\"] = \"James\"\n",
        "doc[\"lastName\"] = \"Cole\"\n",
        "docs.append(doc)\n",
        "\n",
        "doc = collection.createDocument()\n",
        "doc[\"firstName\"] = \"Claudius\"\n",
        "doc[\"lastName\"] = \"Weinberger\"\n",
        "doc[\"email\"] = \"info@arango.com\"\n",
        "docs.append(doc)\n",
        "\n",
        "try:\n",
        "  collection.bulkSave(docs)\n",
        "except Exception as exc:\n",
        "  print(\"The following exception is due to attempting to insert a document that doesn't match the schema.\")\n",
        "  print(exc)\n",
        "\n",
        "print()\n",
        "print(\"Here are the documents that were successfully inserted: \")\n",
        "queryResult = db.AQLQuery(aql)\n",
        "for customer in queryResult:\n",
        "  print(customer)"
      ],
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The following exception is due to attempting to insert a document that doesn't match the schema.\n",
            "1 documents could not be created. Errors: {'error': False, 'created': 1, 'errors': 1, 'empty': 0, 'updated': 0, 'ignored': 0}\n",
            "\n",
            "Here are the documents that were successfully inserted: \n",
            "ArangoDoc '_id: Customers/310078789, _key: 310078789, _rev: _a5RAOOu--_': <store: {'email': 'info@arango.com', 'firstName': 'Claudius', 'lastName': 'Weinberger'}>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R-YJI5mf0TfM",
        "colab_type": "text"
      },
      "source": [
        "To make the validation more stringent you can change the validation level to strict and improve the schema with some additional requirements. Then the validation object could look like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XGyc5qgv2hvj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "db[\"Customers\"].delete() # Drop the collection\n",
        "db.reloadCollections() # Driver method to reload the available collections on the database\n",
        "\n",
        "schema = {\n",
        "  \"message\" : \"Customer Validation Failed\",\n",
        "  \"level\" : \"strict\",\n",
        "  \"rule\" : {\n",
        "    \"type\" : \"object\",\n",
        "    \"properties\" : {\n",
        "      \"firstName\" : {\n",
        "        \"type\" : \"string\",\n",
        "        \"minLength\" : 2,\n",
        "        \"maxLength\" : 20\n",
        "      },\n",
        "      \"lastName\" : {\n",
        "        \"type\" : \"string\",\n",
        "        \"minLength\" : 2,\n",
        "        \"maxLength\" : 20\n",
        "      },\n",
        "      \"email\" : {\n",
        "        \"type\" : \"string\",\n",
        "        \"minLength\" : 5,\n",
        "        \"maxLength\" : 20\n",
        "      },\n",
        "      \"type\" : {\n",
        "        \"enum\" : [\n",
        "          \"lead\",\n",
        "          \"customer\",\n",
        "          \"enterprise\"\n",
        "        ]\n",
        "      }\n",
        "    },\n",
        "    \"required\" : [\n",
        "      \"firstName\",\n",
        "      \"lastName\",\n",
        "      \"email\"\n",
        "    ],\n",
        "    \"additionalProperties\" : False\n",
        "  }\n",
        "}"
      ],
      "execution_count": 86,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8C0amd2PEyN",
        "colab_type": "text"
      },
      "source": [
        "By setting the level to \"strict\" all inserted or changed documents have to match the schema and the schema itself has become more rigorous as well. Now the names and mail have length requirements and are required to be present. Furthermore, there is an optional customer type property that must match one of the 3 given values. We do not allow additional properties to be added to the documents which is controlled by the additionalProperties false flag in the schema definition."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fEE_hHcTk-Lw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "9ec8ecda-bc25-40d0-822f-db0e4bc305b5"
      },
      "source": [
        "print()\n",
        "print(\"Recreate Collection with validation rule\")\n",
        "collection = db.createCollection(\n",
        "        name = \"Customers\",\n",
        "        schema = schema\n",
        "    )\n",
        "\n",
        "#  Try to insert same documents\n",
        "docs = []\n",
        "doc = collection.createDocument()\n",
        "doc[\"firstName\"] = \"James\"\n",
        "doc[\"lastName\"] = \"Cole\"\n",
        "# Note we are missing the required email attribute\n",
        "docs.append(doc)\n",
        "\n",
        "doc = collection.createDocument()\n",
        "doc[\"firstName\"] = \"Claudius\"\n",
        "doc[\"lastName\"] = \"Weinberger\"\n",
        "doc[\"email\"] = \"info@arango.com\"\n",
        "docs.append(doc)\n",
        "\n",
        "try:\n",
        "  collection.bulkSave(docs)\n",
        "except Exception as exc:\n",
        "  print(\"Expected exception as only one of docs is confirming the validation rule.\")\n",
        "  print(exc)\n",
        "\n",
        "# Check customers\n",
        "print()\n",
        "print(\"Checking customers added with validation\")\n",
        "aql = \"\"\"\n",
        "  FOR customer in Customers\n",
        "    return customer\n",
        "  \"\"\"\n",
        "queryResult = db.AQLQuery(aql)\n",
        "for customer in queryResult:\n",
        "  print(customer)\n",
        "\n",
        "\n",
        "# Next Steps\n",
        "print()\n",
        "print(\"If you are running this notebook in Google Colab, use these credentials to access the ArangoDB Web UI at:\")\n",
        "print(\"https://\"+login[\"hostname\"]+\":\"+str(login[\"port\"]))\n",
        "print(\"Username: \" + login[\"username\"])\n",
        "print(\"Password: \" + login[\"password\"])"
      ],
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Recreate Collection with validation rule\n",
            "Expected exception as only one of docs is confirming the validation rule.\n",
            "1 documents could not be created. Errors: {'error': False, 'created': 1, 'errors': 1, 'empty': 0, 'updated': 0, 'ignored': 0}\n",
            "\n",
            "Checking customers added with validation\n",
            "ArangoDoc '_id: Customers/314071743, _key: 314071743, _rev: _a5RAPh2--_': <store: {'email': 'info@arango.com', 'firstName': 'Claudius', 'lastName': 'Weinberger'}>\n",
            "\n",
            "If you are running this notebook in Google Colab, use these credentials to access the ArangoDB Web UI at:\n",
            "https://tutorials.arangodb.cloud:8529\n",
            "Username: TUThf37jys3b5d8xxeaepo9hl\n",
            "Password: TUT868nebwahwwozqdi75z0eb\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2DgQBdoKPlRm",
        "colab_type": "text"
      },
      "source": [
        "If you would like to dive deeper into this example, continue exploring with your temporary database credentials printed above to access the ArangoDB Web UI.\n",
        "\n",
        "Otherwise, if you have not run the notebook in Colab yet, click the Open in Colab button at the top of the page.\n",
        "\n",
        "Please, keep in mind that this database is temporary and will be automatically deleted. If you would like to have a permanent deployment to continue exploring 3.7 with ArangoDB Oasis, [sign up for free](https://cloud.arangodb.com)!\n",
        "\n",
        "If you would like to continue exploring ArangoDB and all of the new features of 3.7 locally instead, you can [download the beta here](https://www.arangodb.com/download-technical-preview/)."
      ]
    }
  ]
}