docs/apis/python-bindings/tutorials/Advanced-Job-Submission-And-Management.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# Advanced Job Submission and Management\n",
    "\n",
    "Launch this tutorial in a Jupyter Notebook on Binder: \n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/htcondor/htcondor-python-bindings-tutorials/master?urlpath=lab/tree/Advanced-Job-Submission-And-Management.ipynb)\n",
    "\n",
    "The two most common HTCondor command line tools are `condor_q` and `condor_submit`.\n",
    "In the previous module, we learned about the `xquery()` method that corresponds to `condor_q`. Here, we will learn the Python binding equivalent of `condor_submit` in greater detail.\n",
    "\n",
    "We start by importing the relevant modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:09.820287Z",
     "iopub.status.busy": "2021-09-16T13:14:09.814798Z",
     "iopub.status.idle": "2021-09-16T13:14:10.126946Z",
     "shell.execute_reply": "2021-09-16T13:14:10.128371Z"
    },
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "import htcondor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Submitting Jobs\n",
    "---------------\n",
    "\n",
    "We will submit jobs utilizing the dedicated `Submit` object.\n",
    "\n",
    "`Submit` objects consist of key-value pairs. Unlike ClassAds, the values do not have an inherent type (such as strings, integers, or booleans); they are evaluated with macro expansion at submit time. Where reasonable, they behave like Python dictionaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.142830Z",
     "iopub.status.busy": "2021-09-16T13:14:10.141323Z",
     "iopub.status.idle": "2021-09-16T13:14:10.162467Z",
     "shell.execute_reply": "2021-09-16T13:14:10.164992Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "foo = 1\n",
      "bar = 2\n",
      "baz = $(foo)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sub = htcondor.Submit({\"foo\": \"1\", \"bar\": \"2\", \"baz\": \"$(foo)\"})\n",
    "print(sub)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.217519Z",
     "iopub.status.busy": "2021-09-16T13:14:10.215728Z",
     "iopub.status.idle": "2021-09-16T13:14:10.281484Z",
     "shell.execute_reply": "2021-09-16T13:14:10.283622Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "foo = 1\n",
      "bar = 2\n",
      "baz = $(foo)\n",
      "qux = 3\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sub[\"qux\"] = 3\n",
    "print(sub)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.299584Z",
     "iopub.status.busy": "2021-09-16T13:14:10.298223Z",
     "iopub.status.idle": "2021-09-16T13:14:10.315334Z",
     "shell.execute_reply": "2021-09-16T13:14:10.316954Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "print(sub.expand(\"baz\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "The available attributes and their semantics are documented in the `condor_submit` [manual](https://htcondor.readthedocs.io/en/latest/man-pages/condor_submit.html),\n",
    "sowe won’t repeat them here.\n",
    "A minimal realistic submit object may look like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.331557Z",
     "iopub.status.busy": "2021-09-16T13:14:10.328028Z",
     "iopub.status.idle": "2021-09-16T13:14:10.337564Z",
     "shell.execute_reply": "2021-09-16T13:14:10.336233Z"
    },
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "sub = htcondor.Submit({\n",
    "    \"executable\": \"/bin/sleep\",\n",
    "    \"arguments\": \"5m\"\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "To go from a submit object to job in a schedd, one must use the `submit` method of a `htcondor.Schedd`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.353531Z",
     "iopub.status.busy": "2021-09-16T13:14:10.351967Z",
     "iopub.status.idle": "2021-09-16T13:14:10.393008Z",
     "shell.execute_reply": "2021-09-16T13:14:10.391442Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "schedd = htcondor.Schedd()          # create a schedd object connected to the local schedd\n",
    "submit_result = schedd.submit(sub)  # queue one job\n",
    "print(submit_result.cluster())      # print the job's ClusterId"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "By default, each invocation of `submit` will submit a single job.  A more common use case is to submit many jobs at once - often identical.  Suppose we don't want to submit a single \"sleep\" job, but 10; instead of writing a `for`-loop around the `submit` method, we can use the `count` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.401864Z",
     "iopub.status.busy": "2021-09-16T13:14:10.400456Z",
     "iopub.status.idle": "2021-09-16T13:14:10.496164Z",
     "shell.execute_reply": "2021-09-16T13:14:10.493503Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "submit_result = schedd.submit(sub, count=10)\n",
    "\n",
    "print(submit_result.cluster())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now query for those jobs in the queue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.513302Z",
     "iopub.status.busy": "2021-09-16T13:14:10.507079Z",
     "iopub.status.idle": "2021-09-16T13:14:10.652018Z",
     "shell.execute_reply": "2021-09-16T13:14:10.653223Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[ ClusterId = 2; ProcId = 0; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 1; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 2; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 3; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 4; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 5; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 6; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 7; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 8; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ ClusterId = 2; ProcId = 9; EnteredCurrentStatus = 1631798050; JobStatus = 1; ServerTime = 1631798050 ]]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schedd.query(\n",
    "    constraint='ClusterId =?= {}'.format(submit_result.cluster()),\n",
    "    projection=[\"ClusterId\", \"ProcId\", \"JobStatus\", \"EnteredCurrentStatus\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's not entirely useful to submit many identical jobs -- but rather each one needs to vary slightly based on its ID (the \"process ID\") within the job cluster.  For this, the `Submit` object in Python behaves  similarly to submit files: references within the submit command are evaluated as macros at submit time.\n",
    "\n",
    "For example, suppose we want the argument to `sleep` to vary based on the process ID:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.669142Z",
     "iopub.status.busy": "2021-09-16T13:14:10.664227Z",
     "iopub.status.idle": "2021-09-16T13:14:10.681620Z",
     "shell.execute_reply": "2021-09-16T13:14:10.683547Z"
    }
   },
   "outputs": [],
   "source": [
    "sub = htcondor.Submit({\"executable\": \"/bin/sleep\", \"arguments\": \"$(Process)s\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the `$(Process)` string will be substituted with the process ID at submit time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.706917Z",
     "iopub.status.busy": "2021-09-16T13:14:10.695919Z",
     "iopub.status.idle": "2021-09-16T13:14:10.781434Z",
     "shell.execute_reply": "2021-09-16T13:14:10.783318Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[[ Args = \"0s\"; ClusterId = 3; ProcId = 0; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"1s\"; ClusterId = 3; ProcId = 1; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"2s\"; ClusterId = 3; ProcId = 2; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"3s\"; ClusterId = 3; ProcId = 3; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"4s\"; ClusterId = 3; ProcId = 4; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"5s\"; ClusterId = 3; ProcId = 5; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"6s\"; ClusterId = 3; ProcId = 6; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"7s\"; ClusterId = 3; ProcId = 7; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"8s\"; ClusterId = 3; ProcId = 8; JobStatus = 1; ServerTime = 1631798050 ],\n",
       " [ Args = \"9s\"; ClusterId = 3; ProcId = 9; JobStatus = 1; ServerTime = 1631798050 ]]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submit_result = schedd.submit(sub, count=10)\n",
    "\n",
    "print(submit_result.cluster())\n",
    "\n",
    "schedd.query(\n",
    "    constraint='ClusterId=?={}'.format(submit_result.cluster()),\n",
    "    projection=[\"ClusterId\", \"ProcId\", \"JobStatus\", \"Args\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The macro evaluation behavior (and the various usable tricks and techniques) are identical between the python bindings and the `condor_submit` executable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Managing Jobs\n",
    "\n",
    "Once a job is in queue, the schedd will try its best to execute it to completion. There are several cases where a user may want to interrupt the normal flow of jobs. Perhaps the results are no longer needed; perhaps the job needs to be edited to correct a submission error. These actions fall under the purview of _job management_.\n",
    "\n",
    "There are two `Schedd` methods dedicated to job management:\n",
    "\n",
    "* `edit()`: Change an attribute for a set of jobs to a given expression. If invoked within a transaction, multiple calls to `edit` are visible atomically.\n",
    "   * The set of jobs to change can be given as a ClassAd expression. If no jobs match the filter, _then an exception is thrown_.\n",
    "* `act()`: Change the state of a job to a given state (remove, hold, suspend, etc).\n",
    "\n",
    "Both methods take a _job specification_: either a ClassAd expression (such as `Owner =?= \"janedoe\"`)\n",
    "or a list of job IDs (such as `[\"1.1\", \"2.2\", \"2.3\"]`).\n",
    "The `act` method takes an argument from the `JobAction` enum.\n",
    "The commonly-used values are:\n",
    "\n",
    "* `Hold`: put a job on hold, vacating a running job if necessary.  A job will stay in the hold\n",
    "   state until explicitly acted upon by the admin or owner.\n",
    "* `Release`: Release a job from the hold state, returning it to Idle.\n",
    "* `Remove`: Remove a job from the Schedd's queue, cleaning it up first on the remote host (if running).\n",
    "   This requires the remote host to acknowledge it has successfully vacated the job, meaning ``Remove`` may\n",
    "   not be instantaneous.\n",
    "* `Vacate`: Cause a running job to be killed on the remote resource and return to idle state.  With\n",
    "  `Vacate`, jobs may be given significant time to cleanly shut down.\n",
    "\n",
    "Here's an example of job management in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.797229Z",
     "iopub.status.busy": "2021-09-16T13:14:10.795271Z",
     "iopub.status.idle": "2021-09-16T13:14:10.872285Z",
     "shell.execute_reply": "2021-09-16T13:14:10.871097Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ ServerTime = 1631798050; ProcId = 0; JobStatus = 1; foo = \"bar\" ]\n",
      "[ ServerTime = 1631798050; ProcId = 1; JobStatus = 1; foo = \"bar\" ]\n",
      "[ ServerTime = 1631798050; ProcId = 2; JobStatus = 1 ]\n",
      "[ ServerTime = 1631798050; ProcId = 3; JobStatus = 1 ]\n",
      "[ ServerTime = 1631798050; ProcId = 4; JobStatus = 1 ]\n"
     ]
    }
   ],
   "source": [
    "submit_result = schedd.submit(sub, count=5)  # queues 5 copies of this job\n",
    "schedd.edit([f\"{submit_result.cluster()}.{idx}\" for idx in range(2)], \"foo\", '\"bar\"')  # sets attribute foo to the string \"bar\" for the first two jobs\n",
    "    \n",
    "for ad in schedd.xquery(\n",
    "    constraint=f\"ClusterId == {submit_result.cluster()}\",\n",
    "    projection=[\"ProcId\", \"JobStatus\", \"foo\"],\n",
    "):\n",
    "    print(repr(ad))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:10.908877Z",
     "iopub.status.busy": "2021-09-16T13:14:10.890081Z",
     "iopub.status.idle": "2021-09-16T13:14:11.174104Z",
     "shell.execute_reply": "2021-09-16T13:14:11.102405Z"
    },
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ ServerTime = 1631798050; ProcId = 0; JobStatus = 1; foo = \"bar\" ]\n",
      "[ ServerTime = 1631798050; ProcId = 1; JobStatus = 1; foo = \"bar\" ]\n",
      "[ ServerTime = 1631798051; ProcId = 2; JobStatus = 5 ]\n",
      "[ ServerTime = 1631798051; ProcId = 3; JobStatus = 5 ]\n",
      "[ ServerTime = 1631798051; ProcId = 4; JobStatus = 5 ]\n"
     ]
    }
   ],
   "source": [
    "schedd.act(htcondor.JobAction.Hold, f\"ClusterId == {submit_result.cluster()} && ProcId >= 2\")\n",
    "\n",
    "for ad in schedd.xquery(\n",
    "    constraint=f\"ClusterId == {submit_result.cluster()}\",\n",
    "    projection=[\"ProcId\", \"JobStatus\", \"foo\"],\n",
    "):\n",
    "    print(repr(ad))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's clean up after ourselves (this will remove all of the jobs you own from the queue)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-16T13:14:11.253336Z",
     "iopub.status.busy": "2021-09-16T13:14:11.251955Z",
     "iopub.status.idle": "2021-09-16T13:14:11.365850Z",
     "shell.execute_reply": "2021-09-16T13:14:11.390481Z"
    },
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ TotalJobAds = 26; TotalPermissionDenied = 0; TotalAlreadyDone = 0; TotalNotFound = 0; TotalSuccess = 26; TotalChangedAds = 1; TotalBadStatus = 0; TotalError = 0 ]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import getpass\n",
    "\n",
    "schedd.act(htcondor.JobAction.Remove, f'Owner == \"{getpass.getuser()}\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## That's It!\n",
    "\n",
    "You've made it through the very basics of the Python bindings.  While there are many other features the Python\n",
    "module has to offer, we have covered enough to replace the command line tools of `condor_q`, `condor_submit`,\n",
    "`condor_status`, `condor_rm` and others."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}