YANG-for-dummies.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# YANG for Dummies!\n",
    "\n",
    "Most people are excited to hear about the new projects from NetworkToCode, but quickly discover that there are a lot of moving parts that present a steep learning curve.  Community contributions will be critical to the success and adoption of these projects.  The groundwork has already been laid for IOS and JunOS drivers, so I want to document the path to onboarding completely new drivers.  This is standalone work that you can replicate without making changes to upstream libraries.  I'll demonstrate this based on David Barroso's great article titled [\"YANG for dummies\"](https://napalm-automation.net/yang-for-dummies/).  Most of the setup files for this tutorial were plagarized from ntc-rosetta.  I am definitely standing on the backs of giants!  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To play along with this notebook, first run:\n",
    "```\n",
    "git clone git@github.com:dgjustice/ntc-dummies.git && cd ntc-dummies\n",
    "make build_test_container\n",
    "make jupyter\n",
    "```\n",
    "Then open your browser to [localhost:8888](localhost:8888).\n",
    "\n",
    "Let's start by creating some file structure...  (This is already done in the container)\n",
    "\n",
    "- Create yang model in `data/yang/dummies/models/star-wars/napalm-star-wars.yang`\n",
    "  - ```\n",
    "    dummies\n",
    "    └── models\n",
    "       ├── napalm-star-wars\n",
    "       │  └── napalm-star-wars.yang\n",
    "       └── napalm-star-wars-library.json\n",
    "    ```\n",
    "    - Copied from [Napalm-Automation](https://napalm-automation.net/yang-for-dummies/)\n",
    "  - Create the library file (**minus the comment**): `cat napalm-star-wars-library.json`\n",
    "    ```\n",
    "    {\n",
    "        \"ietf-yang-library:modules-state\": {\n",
    "            \"module-set-id\": \"6bd894f2-9168-484e-a0bf-f3ed38d864f9\",\n",
    "            \"module\": [\n",
    "                {\n",
    "                    \"name\": \"napalm-star-wars\",\n",
    "                    \"revision\": \"2019-08-31\",  # <- This key *must* be present, see RFC8040\n",
    "                    \"conformance-type\": \"implement\"\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    }\n",
    "    ```\n",
    "    [RFC8040](https://tools.ietf.org/html/rfc8040)\n",
    "  - `head napalm-star-wars.yang`\n",
    "    ```\n",
    "    // module name\n",
    "    module napalm-star-wars {\n",
    "\n",
    "        // boilerplate\n",
    "        yang-version \"1\";\n",
    "        namespace \"https://napalm-yang.readthedocs.io/yang/napalm-star-wars\";\n",
    "\n",
    "        prefix \"napalm-star-wars\";\n",
    "        revision \"2019-08-31\" {  # <- This *must* match the revision above.\n",
    "          description\n",
    "              \"initial version\";\n",
    "          reference \"0.0.1\";\n",
    "        }\n",
    "        ...\n",
    "    ```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The dummy device\n",
    "Our dummy device is going to implement configuration through YAML files.  That's absurd you say!  YAML is a horrible format!  I don't disagree, but it's suitable for pet examples and it lives up to the \"dummy\" device name.  A more practical example might be to parse a text file for device configuration or interface state.  If you are using something like textfsm, you will end up with a dictionary just like the following example.  The end goal regardless of the input is to start our parsing process with some kind of structured data - most commonly JSON/Python dictionaries.  This could be easily replicated with a dictionary returned from a TextFSM process, for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "universe:\n",
      "  individuals:\n",
      "  - affiliation: REBEL_ALLIANCE\n",
      "    age: 57\n",
      "    name: Obi-Wan Kenobi\n",
      "  - affiliation: REBEL_ALLIANCE\n",
      "    age: 19\n",
      "    name: Luke Skywalker\n",
      "  - affiliation: EMPIRE\n",
      "    age: 42\n",
      "    name: Darth Vader\n",
      "  - affiliation: REBEL_ALLIANCE\n",
      "    age: 896\n",
      "    name: Yoda\n"
     ]
    }
   ],
   "source": [
    "def read_file() -> str:\n",
    "    \"\"\"Read a file and return the text.\n",
    "    \n",
    "    This method will be attached to our parser as a helper below.\"\"\"\n",
    "    with open(\"data/star_wars/universe.yml\") as f:\n",
    "        config = f.read()\n",
    "    return config\n",
    "\n",
    "print(read_file())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The parser\n",
    "\n",
    "This is the code that will parse the model.  Please refer to [the official Yangify docs](https://yangify.readthedocs.io/en/latest/tutorials/parsing-quickstart/parsing-quickstart.html) for a complete explanation and walkthrough.  Also see existing implementations at the [ntc-rosetta](https://github.com/networktocode/ntc-rosetta) Github page.  As you are developing and debugging your own parsers, keep in mind that `pdb` is your friend!  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any, Dict, Iterator, Optional, Tuple, cast\n",
    "import json\n",
    "\n",
    "from ruamel.yaml import YAML\n",
    "\n",
    "from ntc_rosetta.helpers import json_helpers as jh\n",
    "\n",
    "from yangify import parser\n",
    "from yangify.parser.text_tree import parse_indented_config\n",
    "from yangify.parser import Parser, ParserData\n",
    "\n",
    "\n",
    "def to_yaml(config: str) -> Dict[str,Any]:\n",
    "    yaml = YAML()\n",
    "    config_data = yaml.load(config)\n",
    "    return config_data\n",
    "\n",
    "\n",
    "class IndividualData(Parser):\n",
    "    class Yangify(ParserData):\n",
    "        path = \"/napalm-star-wars:individual\"\n",
    "\n",
    "        def extract_elements(self) -> Iterator[Tuple[str, Dict[str, Any]]]:\n",
    "            for person in jh.query(\"individuals\", self.native, default=[]):\n",
    "                yield \"individual\", cast(Dict[str, Any], person)\n",
    "\n",
    "    def name(self) -> Optional[str]:\n",
    "        return jh.query(\"name\", self.yy.native)\n",
    "\n",
    "    def age(self) -> int:\n",
    "        return jh.query(\"age\", self.yy.native)\n",
    "\n",
    "    def affiliation(self) -> bool:\n",
    "        return jh.query(\"affiliation\", self.yy.native)\n",
    "\n",
    "\n",
    "class Universe(Parser):\n",
    "    class Yangify(ParserData):\n",
    "        path = \"/napalm-star-wars:universe\"\n",
    "\n",
    "    individual = IndividualData\n",
    "\n",
    "\n",
    "class DummyParser(parser.RootParser):\n",
    "    \"\"\"\n",
    "    DummyParser expects as native data a dictionary where the `universe`\n",
    "    key is reserved for the device configuration.\n",
    "    \"\"\"\n",
    "\n",
    "    class Yangify(parser.ParserData):\n",
    "        def init(self) -> None:\n",
    "            self.root_native = to_yaml(self.root_native)\n",
    "            self.native = self.root_native[\"universe\"]\n",
    "\n",
    "    universe = Universe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The driver\n",
    "We need to register the driver with rosetta so we can make use of some its utility methods.  **We'll need to overload the `get_datamodel` function so we can load our custom models.**  \n",
    "\n",
    "## YANG models\n",
    "ntc-rosetta is a framework, so we are able to bring-our-own-yang models to the party.  Simply tell yangson where to find the data, and we're off to the races (or the next step of debugging)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ntc_rosetta.drivers.base import Driver\n",
    "from yangson.datamodel import DataModel\n",
    "import pathlib\n",
    "\n",
    "# This part looks just like the ios and junos drivers in ntc-rosetta\n",
    "class DummyDriverNapalmStarWars(Driver):\n",
    "    parser = DummyParser\n",
    "    translator = None\n",
    "    datamodel_name = \"napalm_star_wars\"\n",
    "    \n",
    "    # These are the overloads.  Please see the file structure and notes\n",
    "    # in the introduction\n",
    "    @classmethod\n",
    "    def get_data_model(cls) -> DataModel:\n",
    "        base = pathlib.Path(\"/ntc_dummies/data/yang\")\n",
    "        lib = base.joinpath(\"dummies/models/napalm-star-wars-library.json\")\n",
    "        path = [\n",
    "            base.joinpath(\"dummies/models/napalm-star-wars\"),\n",
    "        ]\n",
    "        return DataModel.from_file(lib, path)\n",
    "    \n",
    "    @classmethod\n",
    "    def get_datamodel(cls) -> DataModel:\n",
    "        if cls._datamodel is None:\n",
    "            cls._datamodel = cls.get_data_model()\n",
    "        return cls._datamodel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the datamodel\n",
    "Let's take a peek at the YANG model to make sure the above worked correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--rw napalm-star-wars:universe\n",
      "   +--rw individual* [name]\n",
      "      +--rw affiliation? <identityref>\n",
      "      +--rw age? <age(uint16)>\n",
      "      +--rw name <string>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# If everything worked\n",
    "dummy_driver = DummyDriverNapalmStarWars()\n",
    "# Let's see if we properly loaded the DataModel from the new YANG file\n",
    "print(dummy_driver.get_datamodel().ascii_tree())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse it\n",
    "Finally, we should be able to parse our \"configuration\" file and get a yang datamodel of our star-wars characters!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"napalm-star-wars:universe\": {\n",
      "        \"individual\": [\n",
      "            {\n",
      "                \"name\": \"Obi-Wan Kenobi\",\n",
      "                \"age\": 57,\n",
      "                \"affiliation\": \"napalm-star-wars:REBEL_ALLIANCE\"\n",
      "            },\n",
      "            {\n",
      "                \"name\": \"Luke Skywalker\",\n",
      "                \"age\": 19,\n",
      "                \"affiliation\": \"napalm-star-wars:REBEL_ALLIANCE\"\n",
      "            },\n",
      "            {\n",
      "                \"name\": \"Darth Vader\",\n",
      "                \"age\": 42,\n",
      "                \"affiliation\": \"napalm-star-wars:EMPIRE\"\n",
      "            },\n",
      "            {\n",
      "                \"name\": \"Yoda\",\n",
      "                \"age\": 896,\n",
      "                \"affiliation\": \"napalm-star-wars:REBEL_ALLIANCE\"\n",
      "            }\n",
      "        ]\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "config_data = read_file()\n",
    "\n",
    "# Alternate options\n",
    "# my_universe = DummyParser(dummy_driver.get_datamodel(), native=config_data)\n",
    "# print(my_universe.process().raw_value())\n",
    "\n",
    "parsed = dummy_driver.parse(native=config_data)\n",
    "print(json.dumps(parsed.raw_value(), indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We did it!\n",
    "\n",
    "At this point, we have a populated data model that we can manipulate using Yangson.  For a deeper dive into manipulating populated data models, please see my other [tutorial at the ntc-rosetta project](https://ntc-rosetta.readthedocs.io/en/latest/tutorials/rosetta_and_yangson_walkthrough.html#Manipulating-models-with-Rosetta-and-Yangson)\n",
    "\n",
    "### Do I need to use Rosetta?\n",
    "Strictly speaking, no.  You can look at the commented lines in the previous cell for an alternative route (you'll have to separate the `get_datamodel` method).  Rosetta is an interface that nicely binds these dependencies together, but your particular use case may have different requirements.  \n",
    "\n",
    "## Translate it\n",
    "This about wraps up the story.  We'll write a simple translator to spit out yaml back out into a \"native config\" format.  Keep in mind that this is a pet example.  In production, you will need to handle replaces and merges carefully as well as add tests.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from yangify import translator\n",
    "from yangify.translator import Translator, TranslatorData\n",
    "from yangify.translator.config_tree import ConfigTree\n",
    "\n",
    "\n",
    "class Individual(Translator):\n",
    "    \"\"\"\n",
    "    Implements /napalm-star-wars:universe\n",
    "    \"\"\"\n",
    "    class Yangify(translator.TranslatorData):\n",
    "        def pre_process(self) -> None:\n",
    "            self.result = self.result.new_section(\"\")\n",
    "\n",
    "    def name(self, value: Optional[str]) -> None:\n",
    "        self.yy.result.add_command(f\"  - {value}\")\n",
    "\n",
    "    def age(self, value: Optional[int]) -> None:\n",
    "        self.yy.result.add_command(f\"    {value}\")\n",
    "\n",
    "    def affiliation(self, value: Optional[str]) -> None:\n",
    "        self.yy.result.add_command(f\"    {value[0]}\")\n",
    "\n",
    "class Universe(Translator):\n",
    "    class Yangify(translator.TranslatorData):\n",
    "        def pre_process(self) -> None:\n",
    "            self.result.new_section(\"individuals:\")\n",
    "    individual = Individual\n",
    "    \n",
    "class DummyTranslator(translator.RootTranslator):\n",
    "    class Yangify(translator.TranslatorData):\n",
    "        def init(self) -> None:\n",
    "            self.root_result = ConfigTree()\n",
    "            self.result = self.root_result\n",
    "            self.result.add_command(\"---\")\n",
    "            self.result.add_command(\"universe:\")\n",
    "\n",
    "        def post(self) -> None:\n",
    "            self.root_result = self.root_result.to_string()\n",
    "\n",
    "    universe = Universe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate the translation\n",
    "This is a sanity check to see if we're on the right path.  So far, everything looks good.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "universe:\n",
      "individuals:\n",
      "  - Obi-Wan Kenobi\n",
      "    57\n",
      "    REBEL_ALLIANCE\n",
      "  - Luke Skywalker\n",
      "    19\n",
      "    REBEL_ALLIANCE\n",
      "  - Darth Vader\n",
      "    42\n",
      "    EMPIRE\n",
      "  - Yoda\n",
      "    896\n",
      "    REBEL_ALLIANCE\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# update the driver\n",
    "DummyDriverNapalmStarWars.translator = DummyTranslator\n",
    "dummy_driver = DummyDriverNapalmStarWars()\n",
    "\n",
    "# Alternate options\n",
    "# my_universe = DummyTranslator(dummy_driver.get_datamodel(), candidate=parsed.raw_value())\n",
    "# print(my_universe.process())\n",
    "\n",
    "translated = dummy_driver.translate(candidate=parsed.raw_value())\n",
    "print(translated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Play with the model\n",
    "\n",
    "At this point, we've built a large pet example.  Does this have any practical merit?  The time I've spent on this surely hopes so!  Why don't we do what we've always wished for in a Star Wars movie - Yoda the **mercenary**!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current Yoda:  {\n",
      "  \"name\": \"Yoda\",\n",
      "  \"age\": 896,\n",
      "  \"affiliation\": \"napalm-star-wars:REBEL_ALLIANCE\"\n",
      "}\n",
      "Candidate Yoda:  {\n",
      "  \"name\": \"Yoda\",\n",
      "  \"age\": 896,\n",
      "  \"affiliation\": \"napalm-star-wars:MERCENARY\"\n",
      "}\n"
     ]
    },
    {
     "ename": "YangTypeError",
     "evalue": "[/napalm-star-wars:universe/individual/3/affiliation] invalid-type: not derived from napalm-star-wars:AFFILIATION",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mYangTypeError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-d3e1f770506c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0mstub\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcurrent_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodify_data\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mraw\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Candidate Yoda: \"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdumps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstub\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw_value\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindent\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0mstub\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalidate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/yangson/instance.py\u001b[0m in \u001b[0;36mvalidate\u001b[0;34m(self, scope, ctype)\u001b[0m\n\u001b[1;32m    334\u001b[0m             \u001b[0mYangTypeError\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mIf\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0ma\u001b[0m \u001b[0mscalar\u001b[0m \u001b[0mof\u001b[0m \u001b[0mincorrect\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    335\u001b[0m         \"\"\"\n\u001b[0;32m--> 336\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mschema_node\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    337\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    338\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0madd_defaults\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mContentType\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"InstanceNode\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/yangson/schemanode.py\u001b[0m in \u001b[0;36m_validate\u001b[0;34m(self, inst, scope, ctype)\u001b[0m\n\u001b[1;32m    953\u001b[0m         \u001b[0;34m\"\"\"Extend the superclass method.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    954\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minst\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mArrayEntry\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 955\u001b[0;31m             \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minst\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    956\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    957\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0mValidationScope\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msemantics\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/yangson/schemanode.py\u001b[0m in \u001b[0;36m_validate\u001b[0;34m(self, inst, scope, ctype)\u001b[0m\n\u001b[1;32m    779\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0mValidationScope\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msemantics\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    780\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_must\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minst\u001b[0m\u001b[0;34m)\u001b[0m        \u001b[0;31m# must expressions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 781\u001b[0;31m         \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minst\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    782\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    783\u001b[0m     def _default_instance(self, pnode: \"InstanceNode\", ctype: ContentType,\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/yangson/schemanode.py\u001b[0m in \u001b[0;36m_validate\u001b[0;34m(self, inst, scope, ctype)\u001b[0m\n\u001b[1;32m    475\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_schema_pattern\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minst\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    476\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mm\u001b[0m \u001b[0;32min\u001b[0m \u001b[0minst\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 477\u001b[0;31m             \u001b[0minst\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_member\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalidate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mscope\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    478\u001b[0m         \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minst\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    479\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/yangson/instance.py\u001b[0m in \u001b[0;36mvalidate\u001b[0;34m(self, scope, ctype)\u001b[0m\n\u001b[1;32m    334\u001b[0m             \u001b[0mYangTypeError\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mIf\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0ma\u001b[0m \u001b[0mscalar\u001b[0m \u001b[0mof\u001b[0m \u001b[0mincorrect\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    335\u001b[0m         \"\"\"\n\u001b[0;32m--> 336\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mschema_node\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    337\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    338\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0madd_defaults\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mContentType\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"InstanceNode\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/yangson/schemanode.py\u001b[0m in \u001b[0;36m_validate\u001b[0;34m(self, inst, scope, ctype)\u001b[0m\n\u001b[1;32m    779\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0mValidationScope\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msemantics\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    780\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_must\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minst\u001b[0m\u001b[0;34m)\u001b[0m        \u001b[0;31m# must expressions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 781\u001b[0;31m         \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minst\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscope\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mctype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    782\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    783\u001b[0m     def _default_instance(self, pnode: \"InstanceNode\", ctype: ContentType,\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/yangson/schemanode.py\u001b[0m in \u001b[0;36m_validate\u001b[0;34m(self, inst, scope, ctype)\u001b[0m\n\u001b[1;32m    845\u001b[0m                 inst.value not in self.type):\n\u001b[1;32m    846\u001b[0m             raise YangTypeError(inst.json_pointer(), self.type.error_tag,\n\u001b[0;32m--> 847\u001b[0;31m                                 self.type.error_message)\n\u001b[0m\u001b[1;32m    848\u001b[0m         if (isinstance(self.type, LinkType) and        # referential integrity\n\u001b[1;32m    849\u001b[0m                 \u001b[0mscope\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0mValidationScope\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msemantics\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m \u001b[0;32mand\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mYangTypeError\u001b[0m: [/napalm-star-wars:universe/individual/3/affiliation] invalid-type: not derived from napalm-star-wars:AFFILIATION"
     ]
    }
   ],
   "source": [
    "irt = parsed.datamodel.parse_resource_id(\"/napalm-star-wars:universe/individual=Yoda\")\n",
    "current_data = parsed.root.goto(irt)\n",
    "print(\"Current Yoda: \", json.dumps(current_data.raw_value(), indent=2))\n",
    "modify_data = current_data.raw_value()\n",
    "modify_data['affiliation'] = 'napalm-star-wars:MERCENARY'\n",
    "\n",
    "stub = current_data.update(modify_data, raw=True)\n",
    "print(\"Candidate Yoda: \", json.dumps(stub.raw_value(), indent=2))\n",
    "stub.validate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What broke?!\n",
    "\n",
    "Nothing!!!  This is by design.  If you'll take a look at the beginning of the article, we only loaded the first Napalm-Star-Wars model.  We **didn't** load the extended universe.  Fine, we'll simply push him all the way to the dark side.  :)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Candidate Yoda:  {\n",
      "  \"name\": \"Yoda\",\n",
      "  \"age\": 896,\n",
      "  \"affiliation\": \"napalm-star-wars:EMPIRE\"\n",
      "}\n",
      "Validated!\n",
      "---\n",
      "universe:\n",
      "individuals:\n",
      "  - Obi-Wan Kenobi\n",
      "    57\n",
      "    REBEL_ALLIANCE\n",
      "  - Luke Skywalker\n",
      "    19\n",
      "    REBEL_ALLIANCE\n",
      "  - Darth Vader\n",
      "    42\n",
      "    EMPIRE\n",
      "  - Yoda\n",
      "    896\n",
      "    EMPIRE\n",
      "\n"
     ]
    }
   ],
   "source": [
    "modify_data['affiliation'] = 'napalm-star-wars:EMPIRE'\n",
    "\n",
    "stub = current_data.update(modify_data, raw=True)\n",
    "print(\"Candidate Yoda: \", json.dumps(stub.raw_value(), indent=2))\n",
    "stub.validate()\n",
    "print(\"Validated!\")\n",
    "\n",
    "translated = dummy_driver.translate(candidate=stub.top().raw_value())\n",
    "print(translated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Thank you!\n",
    "\n",
    "As I mentioned earlier, I stand on the backs of giants.  Many thanks to Ken Celenza and the folks at Network To Code, David Barroso, and others involved in these community projects!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feedback\n",
    "If you find any errors or want to leave any kind of feedback, feel free to leave a comment as an [issue](https://github.com/dgjustice/dgjustice.github.io/issues) or find me lurking on the [NTC Slack](slack.networktocode.com)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "nikola": {
   "category": "",
   "date": "2019-09-09 10:49:38 UTC-05:00",
   "description": "",
   "link": "",
   "slug": "ntc-rosetta-and-yang-for-dummies-tutorial",
   "tags": "python",
   "title": "NTC Rosetta and YANG for Dummies Tutorial",
   "type": "text"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}