cisco-gNMI-main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understand gNMI and how to build your first gNMI client with Python to interwork with IOS-XR\n",
    "\n",
    "The following laboratory addresses the capabilities of IOS XR to communicate over gNMI (gRPC Network Management Interface). These capabilities are tightly tied to the specifications of the gNMI service. We will explore the gNMI capabilities for automated configuration of the IOS XR device and monitor in real-time the data rate of an interface of the device.\n",
    "\n",
    "<img src=\"images/use case.png\">\n",
    "\n",
    "\n",
    "All Cisco platforms currently support the gNMI protocol. In the case of IOS XR, the protocol is supported beginning with release 6.5.1:\n",
    "\n",
    "<img src=\"images/management protocols - platform support.png\">\n",
    "\n",
    "\n",
    "gNMI is an alternative to network management protocols such as NETCONF and RESTCONF and it is built in the gRPC Remote Procedure Call (gRPC) framework. Therefore, in order to understand gNMI we first need to understand gRPC."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span class=\"mark\">NOTE</span>\n",
    "\n",
    "References on gRPC and gNMI:\n",
    "- [gRPC](https://grpc.io/)\n",
    "- [gNMI Specification](https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md)\n",
    "- [Draft - Openconfig Rtgwg gNMI Spec - v01](https://tools.ietf.org/html/draft-openconfig-rtgwg-gnmi-spec-01)\n",
    "\n",
    "References on programmability of IOS XR:\n",
    "\n",
    "   ASR 9000\n",
    "  - [Programmability Configuration Guide](https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k-r7-0/programmability/configuration/guide/b-programmability-cg-asr9000-70x/b-programmability-cg-asr9000-70x_chapter_011.html)\n",
    "  - [Yang Data Models](https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k-r7-0/programmability/configuration/guide/b-programmability-cg-asr9000-70x/b-programmability-cg-asr9000-70x_chapter_01.html)\n",
    "  - [Telemetry Configuration](https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k-r7-0/telemetry/configuration/guide/b-telemetry-cg-asr9000-70x.html)\n",
    "\n",
    "NCS 5000\n",
    "  - [Programmability Configuration Guide](https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5000/programmability/70x/b-programmability-cg-ncs5000-70x/b-programmability-cg-ncs5000-70x_chapter_011.html)\n",
    "  - [YANG Data Models](https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5000/programmability/70x/b-programmability-cg-ncs5000-70x/b-programmability-cg-ncs5000-70x_chapter_01.html)\n",
    "  - [Telemetry Configuration](cisco.com/c/en/us/td/docs/iosxr/ncs5000/programmability/70x/b-programmability-cg-ncs5000-70x.html)\n",
    "\n",
    "#### gRPC overview\n",
    "gRPC is a framework dedicated for connecting services and it offers support for load-balancing, tracing, authentication and health-checking. For example, we could use gRPC to connect front-end applications with backend services or to connect management systems with network elements.\n",
    "\n",
    "In gRPC, we define a service using protocol buffers (protobuf).\n",
    "\n",
    "#### gRPC transport\n",
    "\n",
    "gRPC is based on the HTTP/2 transport protocol and it has support for Remote Procedure Calls (RPCs). These are operations that can be called remotely with a set of given parameters and they can return information back to the requestor. The following categories of RPCs exist:\n",
    "- Unary RPC - the client sends a single request to the server and receives a single response back (e.g. polling information).\n",
    "- Server streaming RPC - the client sends a request to get multiple streams of information from the server (e.g. model-driven telemetry); the client can wait for the completion marked in the metadata by the server.\n",
    "- Client streaming RPC - the client sends multiple streams of information to the server; the server typically sends a single response back\n",
    "- Bidirectional streaming RPC - streaming of multiple read-write streams between the client and the server.\n",
    "\n",
    "\n",
    "#### gRPC implementation\n",
    "\n",
    "The communication between the server and the client is established through a gRPC channel formed between the hosts. Therefore, we require the specification of the IP address and the port of the host for the server. A client could modify the traits of a channel (enable / disable compression of a message) and could monitor the state of the channel. \n",
    "\n",
    "The server side contains the implementation of the abstract service and will handle RPC calls and encode the responses. The client side contains the implementation of the same service and can call the methods on the gRPC channel established between the client and the server.\n",
    "\n",
    "#### gRPC authentication\n",
    "\n",
    "In the context of network management, gRPC supports SSL/TLS authentication. It is also possible to add custom authentication methods by extending the implementation.\n",
    "\n",
    "The client uses SSL/TLS authentication to authenticate to the server and to encrypt the messages. There is also an option to use certificates for authentication. The SSL credentials are attached to the gRPC channel.\n",
    "\n",
    "#### Protobuf\n",
    "\n",
    "The payload message transferred through the RPCs may be encoded in protobuf. Note that it is also possible to use alternative encoding formats, such as JSON.\n",
    "\n",
    "Protobuf is a form of binary serialization with which one can encode an abstract concept (such as a service) in a structured way. The format is similar to XML format but said to be simpler, smaller and hence more efficient. The extensions of a service are backwards-compatible with older versions of it.\n",
    "\n",
    "Given a protobuf file, one can generate bindings in various programming languages for the implementation of the underlying service described in the proto file. Proto files usually also have data structures defined in them which describe messages that are transferred with the RPC calls. Due to this definition, we can avoid hand-parsing the messages that are passed along. All these mean that we can transition from an abstract concept and automatize the utilization of that concept.\n",
    "\n",
    "<mark>NOTE</mark> - For a reference on the gNMI service definition, see its corresponding protobuf definition at: https://github.com/openconfig/gnmi/blob/master/proto/gnmi/gnmi.proto.\n",
    "\n",
    "#### gNMI\n",
    "\n",
    "gNMI is dedicated for managing and monitoring the state of network elements. It supports:\n",
    "- Configuration management and retrieval\n",
    "- Operational state retrieval\n",
    "- Bulk data collection via Model-Driven Telemetry (MDT)\n",
    "\n",
    "In practice, the configuration is performed by the management system (client) that has read-write permissions and is applied to the network element (target). \n",
    "\n",
    "The messages within the gRPC service are defined as protocol buffers. The data has to be structured in a tree-like structure, similar to YANG models, and it has to be addressable by paths within that tree.\n",
    "\n",
    "#### gNMI data payloads and paths\n",
    "\n",
    "In general, gNMI carries instances of YANG schemas but it can be used to carry data of other types of structures. In any case, a node will be identified by a path within the structure's tree and its value will be serialized.\n",
    "\n",
    "#### gNMI RPCs\n",
    "\n",
    "The gNMI RPCs sent between the client and the server are:\n",
    "- Capabilities RPC - learn about the features supported by the network element.\n",
    "- Subscribe RPC - a bidirectional streaming RPC; the data can be sent as soon as it is available and hence avoids overloading the network element.\n",
    "- Set RPC - a unary RPC; it is sent by the client to update the state of the target (update / delete / replace).\n",
    "- Get RPC - a unary RPC; it is sent by the client to ask a snapshot of the target's state (configuration / operational)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook will address four possible RPC calls that can be made with gNMI:\n",
    "- Retrieval of capabilities of a network element\n",
    "- Retrieval of state based on CLI format\n",
    "- Retrieval and configuration of state based on YANG and XPaths\n",
    "- Configuration of MDT streaming\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparation of the environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 0 (OPTIONAL)</mark>\n",
    "\n",
    "Enable logging at the debug level:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "import logging\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.DEBUG)\n",
    "logging.debug(\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several implementations of gNMI clients in the open-source community in various programming languages (Go, Python, C++, Java). In this laboratory, we will use the `cisco-gnmi` Python implementation for the communication with the gNMI server hosted on the IOS XR device.\n",
    "\n",
    "<mark>NOTE</mark> - For more information about the `cisco-gnmi` Python implementation see https://github.com/cisco-ie/cisco-gnmi-python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 1</mark>\n",
    "\n",
    "Import the Cisco gNMI package and its client module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cisco_gnmi import ClientBuilder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connectivity to the device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 2</mark>\n",
    "\n",
    "Assuming that the device has its gRPC interface configured (`grpc port 57777`), connect to the device using the following details:\n",
    "- IP address: `198.18.134.72`\n",
    "- gRPC port:  `57777`\n",
    "- operating system: `IOS XR`\n",
    "- username: `cisco`\n",
    "- password: `cisco`\n",
    "\n",
    "The `ClientBuilder()` instantiates a builder for the gNMI client with the IP address and the port of the gNMI target (IOS XR device). By default, the target will expect secure authentication over TLS. The connecting element (the Jupyter host) will retrieve the certificate from the target using the `set_secure_from_target()` and will derive the SSL target name from this certificate using the `set_ssl_target_override()` method. Finally, the client will authenticate with the username and password of the target using the `set_call_authentication()` method. We assume that the user has read-write permissions.\n",
    "\n",
    "Finally, we construct the client by calling `construct()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The way this works:\n",
    "# 1. Assume gRPC over TLS is configured on the IOS-XR device (grpc port 57777)\n",
    "# 2. Retrieve the certificate from the IOS-XR\n",
    "# 3. Derive the SSL target name from the certificate\n",
    "# 4. Authenticate\n",
    "\n",
    "builder = ClientBuilder('198.18.134.72:57777')\n",
    "builder.set_os('IOS XR')\n",
    "builder.set_secure_from_target()\n",
    "builder.set_ssl_target_override()\n",
    "builder.set_call_authentication('cisco', 'cisco')\n",
    "\n",
    "client = builder.construct()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieval of gNMI capabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A client can discover the capabilities of its target using the `Capabilities` RPC. The target will reply with a message that contains:\n",
    "- the version of its gNMI service\n",
    "- the supported encodings\n",
    "- the supported data models (YANG models)\n",
    "\n",
    "<mark>TASK 3</mark>\n",
    "\n",
    "Request the capabilities of the IOS XR target:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "capabilities = client.capabilities()\n",
    "print(capabilities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieval of state information based on CLI format\n",
    "\n",
    "The CLI format is, understandably, the most widely distributed method for device configuration. Despite its readability for humans, it is not the most suitable for computer programs that can now automatize steps of our state retrieval and configuration. The reason behind this is the same as for parsing logs, the CLI configuration content is semi-structured. This makes the development of automation applications laborious with potentially an unknown number of special use-cases, depending on how the commands are defined in their implementation at the source, i.e., in the IOS XR operating system.\n",
    "\n",
    "Nevertheless, practicing an already existing skill (CLI configuration) within a new context (over gNMI) can help to isolate and to understand the nature of gNMI. We will start by executing a CLI command over gNMI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For reasons of simplification and to avoid duplication of parts of code that we have for running and printing the outcome of a CLI command, we can define two methods that incorporates these two steps.\n",
    "\n",
    "The `get_cli()` wrapper accepts two parameters: an instance of the gNMI client and a command in string format. Inside this function, we will make a call to the `get_cli()` function of the client. Internally, this function will send a `Get` RPC request on behalf of the client with the following parameters:\n",
    "- encoding: ASCII\n",
    "- the path for which the data should be retrieved: CLI command\n",
    "\n",
    "<mark>TASK 4</mark>\n",
    "\n",
    "Define the following two methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_context(reply):\n",
    "    print(\"timestamp:\", reply.notification[0].timestamp)\n",
    "    print(\"path:\", reply.notification[0].update[0].path.elem[0].name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cli(client, command):\n",
    "    get_reply = client.get_cli(command)\n",
    "\n",
    "    print_context(get_reply)\n",
    "    print(get_reply.notification[0].update[0].val.ascii_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 5</mark>\n",
    "\n",
    "Firstly, let's retrieve the entire running configuration from the device. Notice that because the CLI command is sent in ASCII format, the device responds to the request with its running **configuration** in the same format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "command = 'show running-config'\n",
    "get_cli(client, command)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 6</mark>\n",
    "\n",
    "We can also retrieve a subsection of the running configuration. Let's retrieve the **configuration state** for streaming of model-driven telemetry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "command = 'show running-config telemetry model-driven'\n",
    "get_cli(client, command)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 7</mark>\n",
    "\n",
    "Correspondingly, we can also retrieve information about the operational state of the device by running the corresponding command. Let's retrieve the **operational state** of streaming of model-driven telemetry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "command = \"show telemetry model-driven subscription\"\n",
    "get_cli(client, command)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the last two commands did not retrieve any information. This is because telemetry streaming is not _yet_ configured or active on the IOS XR device. In the next tasks, we will learn how to configure telemetry streaming and how to receive the streamed information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 8</mark>\n",
    "\n",
    "Inspect the current gRPC configuration on the device:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "command = \"show running-config grpc\"\n",
    "get_cli(client, command)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieval and configuration of state information based on YANG and XPaths\n",
    "\n",
    "<span class=\"mark\">NOTE</span> - References on YANG and JSON encoding for YANG:\n",
    "  - [RFC 6020 - YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)](https://tools.ietf.org/html/rfc6020)\n",
    "  - [RFC 7950 - The YANG 1.1 Data Modeling Language](https://tools.ietf.org/html/rfc7950)\n",
    "  - [RFC 7951 - JSON Encoding of Data Modeled with YANG](https://tools.ietf.org/html/rfc7951)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### YANG - A Data Modeling Language\n",
    "\n",
    "YANG is a modelling language designed for configuration and description of the state of a system. Its purpose is to enable the manipulation of a system using the Network Configuration Protocol (NETCONF) with NETCONF Remote Procedure Calls (RPCs) and Notifications.\n",
    "\n",
    "YANG models are defined in highly structured text files and are compilable. YANG's statically-typed syntax and semantic definition ensures the verification of the model's definition at compile time, resulting in errors that can be treated early in the development process.\n",
    "\n",
    "The models have a tree-based structure with nested sub-elements, named schema nodes. Although they are human readable, the strict formatting and the RFCs defined around them ensure reliability for processing and automation of instance creation of these models.\n",
    "\n",
    "A network device is said to support a capability if it contains a data model definition of that capability in the form of a module. This module will then contain definitions of data nodes and may import or include definitions from other modules. Such a module is called self-contained and compilable.\n",
    "\n",
    "A data model can be instantiated in the Extensible Markup Language (XML) or in Javascript Object Notation (JSON)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Snippet of a YANG model\n",
    "```yang\n",
    "module Cisco-IOS-XR-telemetry-model-driven-cfg {\n",
    "\n",
    "  /*** NAMESPACE / PREFIX DEFINITION ***/\n",
    "\n",
    "  namespace \"http://cisco.com/ns/yang\"+\n",
    "    \"/Cisco-IOS-XR-telemetry-model-driven-cfg\";\n",
    "\n",
    "\n",
    "  prefix \"telemetry-model-driven-cfg\";\n",
    "\n",
    "  /*** LINKAGE (IMPORTS / INCLUDES) ***/\n",
    "\n",
    "  import ietf-inet-types { prefix \"inet\"; }\n",
    "\n",
    "  import Cisco-IOS-XR-types { prefix \"xr\"; }\n",
    "\n",
    "  import cisco-semver { prefix \"semver\"; }\n",
    "  \n",
    "  [...]\n",
    "  \n",
    "  container telemetry-model-driven {\n",
    "    status \"deprecated\";\n",
    "    description\n",
    "      \"This model is deprecated and is replaced by\n",
    "      Cisco-IOS-XR-um-telemetry-model-driven-cfg.yang\n",
    "      which will provide the compatible functionalities\n",
    "      .  Model Driven Telemetry configuration\";\n",
    "\n",
    "    container sensor-groups {\n",
    "      description \"Sensor group configuration\";\n",
    "\n",
    "      list sensor-group {\n",
    "        key \"sensor-group-identifier\";\n",
    "        description \"Sensor group configuration\";\n",
    "\n",
    "        container sensor-paths {\n",
    "          description \"Sensor path configuration\";\n",
    "\n",
    "          list sensor-path {\n",
    "            key \"telemetry-sensor-path\";\n",
    "            description \"Sensor path configuration\";\n",
    "            leaf telemetry-sensor-path {\n",
    "              type string;\n",
    "              description \"Sensor Path\";\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "        leaf sensor-group-identifier {\n",
    "          type xr:Cisco-ios-xr-string;\n",
    "          description \"The identifier for this group\";\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "```\n",
    "\n",
    "#### Snippet of a YANG instance encoded in XML\n",
    "```\n",
    "<telemetry-model-driven xmlns=\"http://cisco.com/ns/yang/Cisco-IOS-XR-telemetry-model-driven-cfg\">\n",
    "   <sensor-groups>\n",
    "    <sensor-group>\n",
    "     <sensor-group-identifier>S1</sensor-group-identifier>\n",
    "     <sensor-paths>\n",
    "      <sensor-path>\n",
    "       <telemetry-sensor-path>Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/generic-counters</telemetry-sensor-path>\n",
    "      </sensor-path>\n",
    "     </sensor-paths>\n",
    "    </sensor-group>\n",
    "   </sensor-groups>\n",
    "</telemetry-model-driven>\n",
    "```\n",
    "\n",
    "\n",
    "#### Snippet of a YANG instance encoded in JSON\n",
    "\n",
    "```\n",
    "{\n",
    "  \"Cisco-IOS-XR-telemetry-model-driven-cfg:telemetry-model-driven\":  {\n",
    "     \"sensor-groups\": {\n",
    "       \"sensor-group\": [{\n",
    "          \"sensor-group-identifier\": \"S1\", \n",
    "            \"sensor-paths\": [{\n",
    "               \"sensor-path\": {\n",
    "                   \"telemetry-sensor-path\":\"Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/generic-counters\"\n",
    "               }\n",
    "            }]\n",
    "        }]\n",
    "     }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### YANG compatibility with the gNMI protocol\n",
    "\n",
    "The gRPC Remote Procedure Call protocol is a newer protocol than NETCONF. It is the result of the adoption of microservices in new architectures. The switch from monolithic applications to ones with distributed components has encouraged the development of the gRPC, and later, of the gNMI protocols. These protocols allow for frequent and efficient RPCs in the microservices systems and they work over HTTP/2.\n",
    "\n",
    "Similarly to NETCONF, their purpose is to manage and retrieve the configuration of a system, as well as to stream telemetry data about that system. A gNMI service will carry payloads that are instances of YANG data models in its attempt to manage or retrieve the underlying information. The values for each instance of the data model are serialized using JSON, Protobuf, Bytes, ASCII or JSON_IETF encoding.\n",
    "\n",
    "Each gNMI request will have a Path defined in a structured format that, optionally, contains keys. A Path contains:\n",
    "- **origin**: optional; its presence disambiguates the path reference (e.g., specifying a namespace)\n",
    "- **elem**: array of PathElem; a PathElem is an instance of a node from the data tree and is composed of its name and its keys and attributes\n",
    "- **target**: optional; its presence is marked in the Prefix of a request and it labels a stream of data for the client that does the request\n",
    "\n",
    "Here is an example of an encoded Path that corresponds to the node `Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface[interface-name=\"MgmtEth0/RP0/CPU0/0\"]/generic-counters`:\n",
    "\n",
    "```\n",
    "path: <\n",
    "  origin: Cisco-IOS-XR-infra-statsd-oper\n",
    "  elem: <\n",
    "    name: \"infra-statistics\"\n",
    "  >\n",
    "  elem: <\n",
    "    name: \"interfaces\"\n",
    "  >\n",
    "  elem: <\n",
    "    name: \"interface\"\n",
    "    key: <\n",
    "      key: \"interface-name\"\n",
    "      value: \"MgmtEth0/RP0/CPU0/0\"\n",
    "    >\n",
    "  >\n",
    "  elem: <\n",
    "    name: \"generic-counters\"\n",
    "  >\n",
    ">\n",
    "```\n",
    "\n",
    "\n",
    "<mark>NOTE</mark> - For a reference on the gNMI service specification for the Path component, see its corresponding protobuf definition at: https://github.com/openconfig/gnmi/blob/master/proto/gnmi/gnmi.proto#L129."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### XPath\n",
    "\n",
    "The XML Path Language (XPath) is a language for referencing components in an XML document. Specifically, the most important type in XPath is the LocationPath type that will describe in an expression the absolute or relative location of a node within the hierarchy of the XML tree.\n",
    "\n",
    "XPath can be used with YANG to address nodes in both the definition and the instantiation of data models. It can further be used in NETCONF and gNMI requests to reference a node or to reference and filter the retrieved data.\n",
    "\n",
    "<mark>NOTE</mark> - References:\n",
    "- [The XPath standards](https://www.w3.org/TR/xpath/all/)\n",
    "- [Whitepaper - XPath in NETCONF and YANG](https://info.tail-f.com/hubfs/Whitepapers/Whitepaper:%20XPath%20in%20NETCONF%20and%20YANG.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 9</mark>\n",
    "\n",
    "For reasons of simplification, define a method that pretty-prints JSON content (`print_json()`).\n",
    "\n",
    "Furthermore, define another method that makes the Get request and displays the reply (`get_xpath()`). Internally, this function will send a `Get` RPC request on behalf of the client with the following parameters:\n",
    "- encoding: JSON_IETF\n",
    "- the path for which the data should be retrieved: xpath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def print_json(reply):\n",
    "    # Pretty printing the JSON reply\n",
    "    json_reply = reply.notification[0].update[0].val.json_ietf_val\n",
    "    if json_reply:    \n",
    "        val_dict = json.loads(json_reply)\n",
    "        print(json.dumps(val_dict, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_xpath(client, xpath):\n",
    "    get_reply = client.get_xpaths(xpath)\n",
    "\n",
    "    print_context(get_reply)\n",
    "    print_json(get_reply)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 10</mark>\n",
    "\n",
    "Use the XPath format to reference a YANG structure whose information we want to retrieve. Specifically, we will retrieve the interfaces configured on the IOS XR device. We specify this by first mentioning the **origin** (`Cisco-IOS-XR-ifmgr-cfg`), followed by `:`, and then followed by a container or other YANG structure that is part of the YANG model (here, `interface-configurations`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpath = 'Cisco-IOS-XR-ifmgr-cfg:interface-configurations'\n",
    "get_xpath(client, xpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuration of model-driven telemetry streaming (persistent, dial-out)\n",
    "\n",
    "Configure the IOS XR device for model-driven telemetry streaming using the **gRPC dial-out** mechanism. In this case, the device initiates a gRPC session (**dial-out**) with the MDT receiver (i.e., the host that runs the Jupyter notebook - 198.18.134.50) and exchanges SYN -- SYN-ACK -- ACK with it while establishing the connection.\n",
    "\n",
    "If the connection is successfully created, the device will start streaming telemetry data towards the collection point.\n",
    "\n",
    "The streaming job will run indefinitely on the IOS XR device, even if the receiver is unreachable. The device will repeatedly try to contact the receiver, every 30 seconds.\n",
    "\n",
    "<mark>NOTE</mark> - The dial-out mechanism is supported in both TCP and gRPC.\n",
    "\n",
    "A dial-out MDT streaming configuration is **persistent** in the configuration of the IOS XR device and will be consistent throughout device reboots. It consists of:\n",
    "- A **sensor-group** that is composed of sensor-path(s):\n",
    " - A sensor-path is a specification of the YANG component about which information should be streamed; specified in XPath format\n",
    "- A **destination-group** that is composed of destination(s):\n",
    " - A destination contains the receiver's specifications:\n",
    "   - IP address\n",
    "   - port\n",
    "   - encoding format\n",
    "   - protocol\n",
    "- A **subscription** that brings together a sensor-group, a destination-group and a time interval [ms]\n",
    "\n",
    "Here, we subscribe to updates of the operational state (`Cisco-IOS-XR-infra-statsd-oper`) of all the interfaces. Specifically, we are looking for their generic counters:\n",
    "- [multicast | broadcast] packets received | sent\n",
    "- bytes received | sent\n",
    "- output | input drops\n",
    "- output | input queue drops\n",
    "- CRC errors\n",
    "- ...\n",
    "\n",
    "\n",
    "In this example, we expect to receive data regularly, every 60 seconds.\n",
    "\n",
    "<mark>TASK 11</mark>\n",
    "\n",
    "Create the subscription:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = \"\"\"{\n",
    "  \"Cisco-IOS-XR-telemetry-model-driven-cfg:telemetry-model-driven\":  {\n",
    "     \"sensor-groups\": {\n",
    "       \"sensor-group\": [{\n",
    "          \"sensor-group-identifier\": \"S1\", \n",
    "            \"sensor-paths\": [{\n",
    "               \"sensor-path\": {\n",
    "                   \"telemetry-sensor-path\":\"Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/generic-counters\"\n",
    "               }\n",
    "            }]\n",
    "        }]\n",
    "     },\n",
    "    \"destination-groups\": {\n",
    "        \"destination-group\": [{\n",
    "            \"destination-id\": \"D1\",\n",
    "            \"ipv4-destinations\": {\n",
    "                \"ipv4-destination\": [{\n",
    "                    \"protocol\": {\n",
    "                        \"protocol\": \"grpc\",\n",
    "                        \"no-tls\": [null]\n",
    "                     },\n",
    "                    \"encoding\": \"self-describing-gpb\",\n",
    "                    \"ipv4-address\": \"198.18.134.50\",\n",
    "                    \"destination-port\": 57000\n",
    "                }]\n",
    "            }\n",
    "        }]\n",
    "    },\n",
    "    \"subscriptions\": {\n",
    "        \"subscription\": [{\n",
    "            \"sensor-profiles\": {\n",
    "                \"sensor-profile\": [{\n",
    "                    \"sensorgroupid\": \"S1\",\n",
    "                    \"sample-interval\": 10000\n",
    "                }]\n",
    "            },\n",
    "            \"destination-profiles\": {\n",
    "                \"destination-profile\": [{\n",
    "                    \"destination-id\": \"D1\"\n",
    "                }]\n",
    "            },\n",
    "            \"subscription-identifier\": \"Sub1\"\n",
    "        }]\n",
    "    }\n",
    "  }\n",
    "}\"\"\"\n",
    "\n",
    "client.set_json(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 12</mark>\n",
    "\n",
    "Confirm that the new MDT **configuration** has been saved on the device:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpath = 'Cisco-IOS-XR-telemetry-model-driven-cfg:telemetry-model-driven'\n",
    "get_xpath(client, xpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieve the **operational state**  of the telemetry streaming subscription. Notice that the subscription is *active*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpath = 'Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven'\n",
    "get_xpath(client, xpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The host 198.18.134.50 already has the collection components for ingesting, storing and visualizing telemetry data. We are using the following set of Docker containers:\n",
    "- Telegraf - a collection container that runs the cisco_telemetry_mdt plugin for ingestion of model-driven telemetry data\n",
    "- InfluxDB - a storage container that runs a database which stores the data\n",
    "- Chronograf - a visualization container that runs a web application which allows the exploration of data\n",
    "\n",
    "The collector listens on port 57000 for incoming connections. IOS XR will initiate the connection, dialling out to the collector. After the connection establishment, the device will begin to stream data over gRPC to the collector which will send it to a database named telemetry.\n",
    "\n",
    "<mark>NOTE</mark> - Please refer to the following references for details about the three components and the ingestion of telemetry data:\n",
    "- [InfluxData](https://www.influxdata.com/)\n",
    "- [Input plugin for Model-driven telemetry over gRPC](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cisco_telemetry_mdt)\n",
    "- [Input plugin for Model-driven telemetry over gNMI](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/cisco_telemetry_gnmi)\n",
    "\n",
    "<mark>TASK 13</mark>\n",
    "\n",
    "In a separate browser tab, open the web application for data visualization located at http://198.18.134.50:8888. Then, hover over the left hand-side and click on Dashboards. Open the `Traffic Monitoring` dashboard.\n",
    "\n",
    "<mark>NOTE</mark> - If you are missing the dashboard, use the `Import Dashboard` button to upload the dashboard located at `monitoring/chronograf/Traffic Monitoring.json`.\n",
    "\n",
    "<img src=\"images/traffic.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 14</mark>\n",
    "\n",
    "In the next task, we will learn about MDT streaming triggered with the dial-in mechanism. Remove the current dial-out configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpath = 'Cisco-IOS-XR-telemetry-model-driven-cfg:telemetry-model-driven'\n",
    "client.delete_xpaths(xpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 15</mark>\n",
    "\n",
    "Confirm that the configuration has been deleted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpath = 'Cisco-IOS-XR-telemetry-model-driven-cfg:telemetry-model-driven'\n",
    "get_xpath(client, xpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuration of model-driven telemetry streaming (dynamic, dial-in)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 16</mark>\n",
    "\n",
    "Open the `cisco-gNMI-dynamic-subscription` Jupyter notebook and run it to create a subscription that will be active throughout the laboratory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TASK 23</mark>\n",
    "\n",
    "Check if there is model-driven telemetry **configuration** on the device:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpath = 'Cisco-IOS-XR-telemetry-model-driven-cfg:telemetry-model-driven'\n",
    "get_xpath(client, xpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that there is no actual telemetry configured on the device. The gNMI-based subscription was created dynamically by the gNMI client, without touching the configuration of the device. Instead, this host has instantiated a **dynamic** subscription over the RPC channel. This subscription does not have local persistence on the IOS XR device, and is not persistent between device reboots. However, the existence of the dynamic subscription is marked in the operational state of the device.\n",
    "\n",
    "\n",
    "<mark>NOTE</mark> - For more information about persistent and dynamic subscription, see the `openconfig-telemetry.yang` YANG model that the gNMI implementation used as reference for the subscription functionality:\n",
    "- persistent: https://github.com/openconfig/public/blob/master/release/models/telemetry/openconfig-telemetry.yang#L247\n",
    "- dynamic: https://github.com/openconfig/public/blob/master/release/models/telemetry/openconfig-telemetry.yang#L414\n",
    "\n",
    "Let's retrieve the **operational state** of telemetry streaming with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpath = \"Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven\"\n",
    "get_xpath(client, xpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that there is a new _active_ subscription reported. The values of the IDs for the sensor-group, destination-group and subscription request that were created as a result of the dynamic subscription request follow the format: **GNMI__XYZ**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}