jsonnet/Jsonnet Pipeline Specs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4c664282-aaca-4c85-81a6-39dd021e2d62",
   "metadata": {},
   "source": [
    "# Jsonnet Pipeline Specs\n",
    "\n",
    "A **jsonnet pipeline specification** is a way to create pipeline specifications with `jsonnet` (pronounced \"jay sonnet\"), a wrapping layer that can return JSON pipeline specs. This provides a few different benefits as we'll see in this notebook:  \n",
    "\n",
    "1. Wrap a pipeline definition in a function.\n",
    "2. Parameterize at creation time\n",
    "3. Validate internal logic\n",
    "4. Pipeline specs can be hosted - Create pipeline from URL"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fef92ec-c533-44e6-a09d-40401cddc44e",
   "metadata": {},
   "source": [
    "## Step 0: OpenCV Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56f04a08-6edb-4c53-b2b1-1bd85cae3e0d",
   "metadata": {},
   "source": [
    "Create a repo of images that will be used for edges and montage pipelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "be28deb5-530c-404e-bf01-51a3ee9b3390",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl create repo images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aa689346-6f57-4fec-b99e-d1be208bf686",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME   CREATED                SIZE (MASTER) ACCESS LEVEL \n",
      "images Less than a second ago ≤ 0B          [repoOwner]   \n"
     ]
    }
   ],
   "source": [
    "!pachctl list repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5425ba82-d675-42a1-ba4c-e7600dec6d5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl put file images@master:liberty.png -f http://imgur.com/46Q8nDz.png\n",
    "!pachctl put file images@master:AT-AT.png -f http://imgur.com/8MN9Kg0.png\n",
    "!pachctl put file images@master:kitten.png -f http://imgur.com/g2QnNqa.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13a3ca84-7d9b-4979-a170-6ad05841b820",
   "metadata": {},
   "source": [
    "## Step 1: Jsonnet Pipeline Specs Intro\n",
    "\n",
    "Previously, Pachyderm Pipelines could be written in `json` or `yaml`, with Jsonnet Pipeline Spec, you can now write specs with a new language called Jsonnet.\n",
    "\n",
    "Unlike other templating languages (like the Go templates used by helm, for example), Jsonnet has functions and is actually Turing complete.\n",
    "\n",
    "Let's take a look at a jsonnet file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2e4b723c-30a7-4b00-80e8-dc6613eaf04e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "////\n",
      "// Template arguments:\n",
      "//\n",
      "// suffix : An arbitrary suffix appended to the name of this pipeline, for\n",
      "//          disambiguation when multiple instances are created.\n",
      "// src : the repo from which this pipeline will read the images to which\n",
      "//       it applies edge detection.\n",
      "////\n",
      "function(suffix, src)\n",
      "{\n",
      "  pipeline: { name: \"edges-\"+suffix },\n",
      "  description: \"OpenCV edge detection on \"+src,\n",
      "  input: {\n",
      "    pfs: {\n",
      "      name: \"images\",\n",
      "      glob: \"/*\",\n",
      "      repo: src,\n",
      "    }\n",
      "  },\n",
      "  transform: {\n",
      "    cmd: [ \"python3\", \"/edges.py\" ],\n",
      "    image: \"pachyderm/opencv:0.0.1\"\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!cat edges.jsonnet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26972022-5e19-41f4-abdd-06c1f82d95aa",
   "metadata": {},
   "source": [
    "The first thing to notice is that the format is not that different from the `json` version of the edges pipeline. The `jsonnet` file allows us to create templates allowing us to parameterize our pipelines at creation. \n",
    "\n",
    "This means that pipeline specs written in `jsonnet` can take parameters and have internal logic, and generally can be made much more adaptable and reusable.\n",
    "\n",
    "For this pipeline, the parameters are just `suffix`, which only affects the pipeline's name and `src`, which is the repo where this pipeline reads its images from. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f8f87f2-ff1c-4ce9-9564-3e5b10bb1834",
   "metadata": {},
   "source": [
    "Jsonnet pipeline specs can be called and parameterized directly from the CLI. We can do this by adding the `--jsonnet` flag and setting our args on the command line. \n",
    "\n",
    "```shell\n",
    "$ pachctl create pipeline -h \n",
    "\n",
    "Create a new pipeline from a pipeline specification. For details on the format, see https://docs.pachyderm.com/latest/reference/pipeline_spec/.\n",
    "\n",
    "Usage:\n",
    "  pachctl create pipeline [flags]\n",
    "\n",
    "Flags:\n",
    "      --arg strings       Top-level argument passed to the Jsonnet template in --jsonnet (which must be set if any --arg arugments are passed). Value must be of the form 'param=value'. For multiple args, --arg may be set more than once, or it may be passed a comma-separated list of 'param=value' pairs.\n",
    "  -f, --file string       A JSON file (url or filepath) containing one or more pipelines. \"-\" reads from stdin (the default behavior). Exactly one of --file and --jsonnet must be set.\n",
    "  -h, --help              help for pipeline\n",
    "      --jsonnet string    BETA: A Jsonnet template file (url or filepath) for one or more pipelines. \"-\" reads from stdin. Exactly one of --file and --jsonnet must be set. Jsonnet templates must contain a top-level function; strings can be passed to this function with --arg (below)\n",
    "  -p, --push-images       If true, push local docker images into the docker registry.\n",
    "  -r, --registry string   The registry to push images to. (default \"index.docker.io\")\n",
    "  -u, --username string   The username to push images as.\n",
    "\n",
    "Global Flags:\n",
    "      --no-color   Turn off colors.\n",
    "  -v, --verbose    Output verbose logs\n",
    "  ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64c4517f-8451-4b50-8e27-a6f99dff6a99",
   "metadata": {},
   "source": [
    "For the edges pipeline, we are going to set the `suffix` arg to `1`, which will set our pipeline name to `edges-1` and we'll set our `src` to the images repo that we created earlier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b8f2899f-0dcc-43ef-895e-02ea371cd8fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl create pipeline --jsonnet edges.jsonnet --arg suffix=1 --arg src=images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "46f8c8af-595a-44d5-bdc3-2cf841f944cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME    VERSION INPUT     CREATED        STATE / LAST JOB  DESCRIPTION                     \n",
      "edges-1 1       images:/* 20 seconds ago \u001b[32mrunning\u001b[0m / \u001b[32msuccess\u001b[0m OpenCV edge detection on images \n"
     ]
    }
   ],
   "source": [
    "!pachctl list pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "24c71c81-0431-4de7-8239-718a6aa03501",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: edges-1\n",
      "Description: OpenCV edge detection on images\n",
      "Created: About a minute ago \n",
      "State: \u001b[32mrunning\u001b[0m\n",
      "Reason: \n",
      "Workers Available: 1/1\n",
      "Stopped: false\n",
      "Parallelism Spec: <nil>\n",
      "\n",
      "\n",
      "Datum Timeout: (duration: nil Duration)\n",
      "Job Timeout: (duration: nil Duration)\n",
      "Input:\n",
      "{\n",
      "  \"pfs\": {\n",
      "    \"name\": \"images\",\n",
      "    \"repo\": \"images\",\n",
      "    \"repo_type\": \"user\",\n",
      "    \"branch\": \"master\",\n",
      "    \"glob\": \"/*\"\n",
      "  }\n",
      "}\n",
      "\n",
      "Output Branch: master\n",
      "Transform:\n",
      "{\n",
      "  \"image\": \"pachyderm/opencv:0.0.1\",\n",
      "  \"cmd\": [\n",
      "    \"python3\",\n",
      "    \"/edges.py\"\n",
      "  ]\n",
      "}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!pachctl inspect pipeline edges-1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e481f498-af80-4578-b1e6-79d8f824984f",
   "metadata": {},
   "source": [
    "## Step 2: Create a Pipeline from a URL\n",
    "Next, we'll create the montage pipeline. Only, this time we'll create it from a hosted version of the pipeline, passing the address of the template on GitHub. \n",
    "\n",
    "To view this template, [click here](https://raw.githubusercontent.com/pachyderm/examples/master/jsonnet/montage.jsonnet).\n",
    "\n",
    "The montage pipeline has a few additional parameters that we can set, for example specifying `repo` or `repo@branch`. It allso shows how we can incorporate assertions, allowing us to validate arguments before a pipeline is created. \n",
    "\n",
    "Even with all of this logic we are still able to configure the pipeline during execution using the `--arg` parameters like so: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ed358ae5-f7a5-4a28-9831-d6c41c44d0bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl create pipeline \\\n",
    "--jsonnet https://raw.githubusercontent.com/pachyderm/examples/master/jsonnet/montage.jsonnet \\\n",
    "--arg suffix=1 \\\n",
    "--arg left=images@master \\\n",
    "--arg right=edges-1@master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b1390cc9-45b7-44de-ae5a-5377edc876e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: montage-1\n",
      "Description: A pipeline that combines images from images@master and edges-1@master into a montage.\n",
      "Created: 34 seconds ago \n",
      "State: \u001b[32mrunning\u001b[0m\n",
      "Reason: \n",
      "Workers Available: 1/1\n",
      "Stopped: false\n",
      "Parallelism Spec: <nil>\n",
      "\n",
      "\n",
      "Datum Timeout: (duration: nil Duration)\n",
      "Job Timeout: (duration: nil Duration)\n",
      "Input:\n",
      "{\n",
      "  \"cross\": [\n",
      "    {\n",
      "      \"pfs\": {\n",
      "        \"name\": \"left\",\n",
      "        \"repo\": \"images\",\n",
      "        \"repo_type\": \"user\",\n",
      "        \"branch\": \"master\",\n",
      "        \"glob\": \"/\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"pfs\": {\n",
      "        \"name\": \"right\",\n",
      "        \"repo\": \"edges-1\",\n",
      "        \"repo_type\": \"user\",\n",
      "        \"branch\": \"master\",\n",
      "        \"glob\": \"/\"\n",
      "      }\n",
      "    }\n",
      "  ]\n",
      "}\n",
      "\n",
      "Output Branch: master\n",
      "Transform:\n",
      "{\n",
      "  \"image\": \"dpokidov/imagemagick:7.0.10-58\",\n",
      "  \"cmd\": [\n",
      "    \"sh\"\n",
      "  ],\n",
      "  \"stdin\": [\n",
      "    \"montage -shadow -background SkyBlue -geometry 300x300+2+2 /pfs/left/* /pfs/right/* /pfs/out/montage.png\"\n",
      "  ]\n",
      "}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!pachctl inspect pipeline montage-1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "355149ef-b2a0-4490-a87c-50972ffe09ad",
   "metadata": {},
   "source": [
    "## Step 3: Reusing pipelines (Montage within a montage)\n",
    "To show off where this feature is going, though let's create another instance of the montage pipeline. \n",
    "\n",
    "Our montage pipeline takes in two repos of images and creates a montage. Normally, we would have to duplicate our pipeline spec, but now we don'thave to touch the pipeline spec at all. We just modify the arguments we send to the jsonnet pipeline spec, and we have another instance of the pipeline (`montage-2`). \n",
    "\n",
    "This means that we can create another pipeline using the same template and just change the `suffix` argument in this case. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "129bd9b2-d521-40bf-abd1-e395e10b0e21",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl create pipeline \\\n",
    "--jsonnet https://raw.githubusercontent.com/pachyderm/examples/master/jsonnet/montage.jsonnet \\\n",
    "--arg suffix=2 \\\n",
    "--arg left=montage-1@master \\\n",
    "--arg right=edges-1@master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a423edc7-4e4b-4645-af26-87177a36c479",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: montage-2\n",
      "Description: A pipeline that combines images from montage-1@master and edges-1@master into a montage.\n",
      "Created: 13 seconds ago \n",
      "State: \u001b[32mrunning\u001b[0m\n",
      "Reason: \n",
      "Workers Available: 1/1\n",
      "Stopped: false\n",
      "Parallelism Spec: <nil>\n",
      "\n",
      "\n",
      "Datum Timeout: (duration: nil Duration)\n",
      "Job Timeout: (duration: nil Duration)\n",
      "Input:\n",
      "{\n",
      "  \"cross\": [\n",
      "    {\n",
      "      \"pfs\": {\n",
      "        \"name\": \"left\",\n",
      "        \"repo\": \"montage-1\",\n",
      "        \"repo_type\": \"user\",\n",
      "        \"branch\": \"master\",\n",
      "        \"glob\": \"/\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"pfs\": {\n",
      "        \"name\": \"right\",\n",
      "        \"repo\": \"edges-1\",\n",
      "        \"repo_type\": \"user\",\n",
      "        \"branch\": \"master\",\n",
      "        \"glob\": \"/\"\n",
      "      }\n",
      "    }\n",
      "  ]\n",
      "}\n",
      "\n",
      "Output Branch: master\n",
      "Transform:\n",
      "{\n",
      "  \"image\": \"dpokidov/imagemagick:7.0.10-58\",\n",
      "  \"cmd\": [\n",
      "    \"sh\"\n",
      "  ],\n",
      "  \"stdin\": [\n",
      "    \"montage -shadow -background SkyBlue -geometry 300x300+2+2 /pfs/left/* /pfs/right/* /pfs/out/montage.png\"\n",
      "  ]\n",
      "}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!pachctl inspect pipeline montage-2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7407af8d-1819-4f85-b555-6476ee81b657",
   "metadata": {},
   "source": [
    "## Import hosted Jsonnet\n",
    "\n",
    "One addtional feature is that we can also import hosted jsonnet as well. This allows you to compose logic and reuse common functions across templates. This works excellently when you have standard validation that is shared across pipeline specifications. \n",
    "\n",
    "E.g.\n",
    "\n",
    "```json\n",
    "import https://raw.githubusercontent.com/pachyderm_specs/myspecs/functions.jsonnet\n",
    "\n",
    "function(suffix, src)\n",
    "{\n",
    "  pipeline: { name: \"edges-\"+suffix },\n",
    "  description: \"OpenCV edge detection on \"+src,\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc09dac2-9a45-46f0-bd6c-5bdc8b810d9e",
   "metadata": {},
   "source": [
    "## Cleanup Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "d7f390b5-a8f5-40b1-a0ca-48e6732af887",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete pipelines created\n",
    "!pachctl delete pipeline montage-2\n",
    "!pachctl delete pipeline montage-1\n",
    "!pachctl delete pipeline edges-1\n",
    "\n",
    "# Delete images repo\n",
    "!pachctl delete repo images"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}