docs/apis/python-bindings/tutorials/ClassAds-Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# ClassAds Introduction\n",
    "\n",
    "Launch this tutorial in a Jupyter Notebook on Binder: \n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/htcondor/htcondor-python-bindings-tutorials/master?urlpath=lab/tree/ClassAds-Introduction.ipynb)\n",
    "\n",
    "In this tutorial, we will learn the basics of the [ClassAd language](https://htcondor.org/classad/classad.html),\n",
    "the policy and data exchange language that underpins all of HTCondor.\n",
    "ClassAds are fundamental in the HTCondor ecosystem, so understanding them will be good preparation for future tutorials.\n",
    "\n",
    "The Python implementation of the ClassAd language is in the `classad` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:20.935747Z",
     "iopub.status.busy": "2021-09-16T13:14:20.934837Z",
     "iopub.status.idle": "2021-09-16T13:14:20.950484Z",
     "shell.execute_reply": "2021-09-16T13:14:20.951097Z"
    },
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "import classad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "The ClassAd language is built around _values_ and _expressions_.  If you know Python, both concepts are familiar.  Examples of familiar values include:\n",
    "- Integers (`1`, `2`, `3`),\n",
    "- Floating point numbers (`3.145`, `-1e-6`)\n",
    "- Booleans (`true` and `false`).\n",
    "\n",
    "Examples of expressions are:\n",
    "- Attribute references: `foo`\n",
    "- Boolean expressions: `a && b`\n",
    "- Arithmetic expressions: `123 + c`\n",
    "- Function calls: `ifThenElse(foo == 123, 3.14, 5.2)`\n",
    "\n",
    "Expressions can be evaluated to values.\n",
    "Unlike many programming languages, expressions are lazily-evaluated: they are kept in memory as expressions until a value is explicitly requested.\n",
    "ClassAds holding expressions to be evaluated later are how many internal parts of HTCondor, like job requirements, are expressed.\n",
    "\n",
    "Expressions are represented in Python with `ExprTree` objects.\n",
    "The desired ClassAd expression is passed as a string to the constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:20.956100Z",
     "iopub.status.busy": "2021-09-16T13:14:20.955274Z",
     "iopub.status.idle": "2021-09-16T13:14:20.966073Z",
     "shell.execute_reply": "2021-09-16T13:14:20.965254Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ClassAd arithemetic expression: 1 + 4 (of type <class 'classad.classad.ExprTree'>)\n"
     ]
    }
   ],
   "source": [
    "arith_expr = classad.ExprTree(\"1 + 4\")\n",
    "print(f\"ClassAd arithemetic expression: {arith_expr} (of type {type(arith_expr)})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expressions can be evaluated on-demand:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:20.974120Z",
     "iopub.status.busy": "2021-09-16T13:14:20.973370Z",
     "iopub.status.idle": "2021-09-16T13:14:20.977183Z",
     "shell.execute_reply": "2021-09-16T13:14:20.977893Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "print(arith_expr.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an expression that includes a ClassAd function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:20.984949Z",
     "iopub.status.busy": "2021-09-16T13:14:20.981327Z",
     "iopub.status.idle": "2021-09-16T13:14:20.989871Z",
     "shell.execute_reply": "2021-09-16T13:14:20.990636Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Function expression: ifThenElse(4 > 6,123,456)\n",
      "Corresponding value: 456 (of type <class 'int'>)\n"
     ]
    }
   ],
   "source": [
    "function_expr = classad.ExprTree(\"ifThenElse(4 > 6, 123, 456)\")\n",
    "print(f\"Function expression: {function_expr}\")\n",
    "\n",
    "value = function_expr.eval()\n",
    "print(f\"Corresponding value: {value} (of type {type(value)})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Notice that, when possible, we convert ClassAd values to Python values.  Hence, the result of evaluating the expression above is the Python `int` `456`.\n",
    "\n",
    "There are two important values in the ClassAd language that have no direct equivalent in Python: `Undefined` and `Error`.\n",
    "\n",
    "`Undefined` occurs when a reference occurs to an attribute that is not defined; it is analogous to a `NameError` exception in Python (but there is no concept of an exception in ClassAds).\n",
    "For example, evaluating an unset attribute produces `Undefined`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:20.996426Z",
     "iopub.status.busy": "2021-09-16T13:14:20.995424Z",
     "iopub.status.idle": "2021-09-16T13:14:21.000790Z",
     "shell.execute_reply": "2021-09-16T13:14:21.001456Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Undefined\n"
     ]
    }
   ],
   "source": [
    "print(classad.ExprTree(\"foo\").eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Error` occurs primarily when an expression combines two different types or when a function call occurs with the incorrect arguments.\n",
    "Note that even in this case, no Python exception is raised!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:21.010051Z",
     "iopub.status.busy": "2021-09-16T13:14:21.005324Z",
     "iopub.status.idle": "2021-09-16T13:14:21.013757Z",
     "shell.execute_reply": "2021-09-16T13:14:21.014363Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error\n",
      "Error\n"
     ]
    }
   ],
   "source": [
    "print(classad.ExprTree('5 + \"bar\"').eval())\n",
    "print(classad.ExprTree('ifThenElse(1, 2, 3, 4, 5)').eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## ClassAds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "The concept that makes the ClassAd language special is, of course, the _ClassAd_!\n",
    "\n",
    "The ClassAd is analogous to a Python or JSON dictionary. \n",
    "_Unlike_ a dictionary, which is a set of unique key-value pairs, the ClassAd object is a set of key-_expression_ pairs.\n",
    "The expressions in the ad can contain attribute references to other keys in the ad, which will be followed when evaluated.\n",
    "\n",
    "There are two common ways to represent ClassAds in text.\n",
    "The \"new ClassAd\" format:\n",
    "```\n",
    "[\n",
    "  a = 1;\n",
    "  b = \"foo\";\n",
    "  c = b\n",
    "]\n",
    "```\n",
    "\n",
    "And the \"old ClassAd\" format:\n",
    "```\n",
    "a = 1\n",
    "b = \"foo\"\n",
    "c = b\n",
    "```\n",
    "\n",
    "Despite the \"new\" and \"old\" monikers, \"new\" is over a decade old.\n",
    "HTCondor command line tools utilize the \"old\" representation.\n",
    "The Python bindings default to \"new\".\n",
    "\n",
    "A `ClassAd` object may be initialized via a string in either of the above representation.\n",
    "As a ClassAd is so similar to a Python dictionary, they may also be constructed from a dictionary.\n",
    "\n",
    "Let's construct some ClassAds!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:21.023167Z",
     "iopub.status.busy": "2021-09-16T13:14:21.022140Z",
     "iopub.status.idle": "2021-09-16T13:14:21.025657Z",
     "shell.execute_reply": "2021-09-16T13:14:21.026336Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    [\n",
      "        a = 1; \n",
      "        b = \"foo\"; \n",
      "        c = b; \n",
      "        d = a + 4\n",
      "    ]\n"
     ]
    }
   ],
   "source": [
    "ad1 = classad.ClassAd(\"\"\"\n",
    "[\n",
    "  a = 1;\n",
    "  b = \"foo\";\n",
    "  c = b;\n",
    "  d = a + 4;\n",
    "]\"\"\")\n",
    "print(ad1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can construct the same ClassAd from a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:21.033464Z",
     "iopub.status.busy": "2021-09-16T13:14:21.032546Z",
     "iopub.status.idle": "2021-09-16T13:14:21.039208Z",
     "shell.execute_reply": "2021-09-16T13:14:21.039865Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    [\n",
      "        d = a + 4; \n",
      "        c = b; \n",
      "        b = \"foo\"; \n",
      "        a = 1\n",
      "    ]\n"
     ]
    }
   ],
   "source": [
    "ad_from_dict = classad.ClassAd(\n",
    "{\n",
    "    \"a\": 1,\n",
    "    \"b\": \"foo\",\n",
    "    \"c\": classad.ExprTree(\"b\"),\n",
    "    \"d\": classad.ExprTree(\"a + 4\"),\n",
    "})\n",
    "print(ad_from_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "ClassAds are quite similar to dictionaries; in Python, the `ClassAd` object behaves similarly to a dictionary and has similar convenience methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:21.045956Z",
     "iopub.status.busy": "2021-09-16T13:14:21.043367Z",
     "iopub.status.idle": "2021-09-16T13:14:21.118383Z",
     "shell.execute_reply": "2021-09-16T13:14:21.119081Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "'not_here'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_116/3690994919.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mad1\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"a\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mad1\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"not_here\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m: 'not_here'"
     ]
    }
   ],
   "source": [
    "print(ad1[\"a\"])\n",
    "print(ad1[\"not_here\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:21.124267Z",
     "iopub.status.busy": "2021-09-16T13:14:21.123379Z",
     "iopub.status.idle": "2021-09-16T13:14:21.128194Z",
     "shell.execute_reply": "2021-09-16T13:14:21.129128Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "print(ad1.get(\"not_here\", 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:21.134465Z",
     "iopub.status.busy": "2021-09-16T13:14:21.133556Z",
     "iopub.status.idle": "2021-09-16T13:14:21.141845Z",
     "shell.execute_reply": "2021-09-16T13:14:21.143112Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    [\n",
      "        f = true; \n",
      "        e = 8; \n",
      "        a = 1; \n",
      "        b = \"foo\"; \n",
      "        c = b; \n",
      "        d = a + 4\n",
      "    ]\n"
     ]
    }
   ],
   "source": [
    "ad1.update({\"e\": 8, \"f\": True})\n",
    "print(ad1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Remember our example of an `Undefined` attribute above?  We now can evaluate references within the context of the ad:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:21.150961Z",
     "iopub.status.busy": "2021-09-16T13:14:21.147771Z",
     "iopub.status.idle": "2021-09-16T13:14:21.154353Z",
     "shell.execute_reply": "2021-09-16T13:14:21.155023Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "print(ad1.eval(\"d\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Note that an expression is still not evaluated until requested, even if it is invalid:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:21.160823Z",
     "iopub.status.busy": "2021-09-16T13:14:21.159473Z",
     "iopub.status.idle": "2021-09-16T13:14:21.178270Z",
     "shell.execute_reply": "2021-09-16T13:14:21.176531Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b + 5\n",
      "<class 'classad.classad.ExprTree'>\n",
      "Error\n"
     ]
    }
   ],
   "source": [
    "ad1[\"g\"] = classad.ExprTree(\"b + 5\")\n",
    "print(ad1[\"g\"])\n",
    "print(type(ad1[\"g\"]))\n",
    "print(ad1.eval(\"g\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Onto HTCondor!\n",
    "--------------\n",
    "\n",
    "ClassAds and expressions are core concepts in interacting with HTCondor.\n",
    "Internally, machines and jobs are represented as ClassAds; expressions are used to filter objects and to define policy.\n",
    "\n",
    "There's much more to learn in ClassAds!  For now, you have enough background to continue to the next tutorial - [HTCondor Introduction](HTCondor-Introduction.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}