Intro to Pachyderm Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9433c125-8d2e-4d10-ac24-bd7d5134265c",
   "metadata": {},
   "source": [
    "# Intro to Pachyderm Tutorial\n",
    "<img src=\"images/nb_header.png\" alt=\"Drawing\" style=\"width: 600px;\"/>\n",
    "\n",
    "### Welcome to Pachyderm! \n",
    "\n",
    "Pachyderm is an incredibly powerful platform, and can be used for many kinds of data-centered applications. In this notebook, we will introduce you to the basic concepts of data versioning and data pipelines and how they work in Pachyderm. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a588e2-dada-4382-8331-91501cc9aa11",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f29bdfd6-795b-41ac-afae-4028ced49c4a",
   "metadata": {},
   "source": [
    "For this tutorial, we will use the `pachctl` command line interface. This means that any of these commands can be run from your terminal as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c94592d-fc9c-49ee-9ed2-cfe1925735a5",
   "metadata": {},
   "source": [
    "In a self-hosted Pachyderm cluster, you will have to install the Pachyderm client and connect to it before `pachctl` will run. For more information, see the [Getting Started](https://docs.pachyderm.com/latest/getting_started/local_installation/) docs. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12b81835-b7f8-413b-87c5-9fa5b6a4986f",
   "metadata": {},
   "source": [
    "Let's make sure that we're connected to the Pachyderm cluster by checking the version. \n",
    "\n",
    "(`pachctl` is the version of the client running locally, `pachd` is the version of the Pachyderm server running in the cluster) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "26d3153d-5f12-4409-82af-1b5b230e636b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "COMPONENT           VERSION             \n",
      "pachctl             2.1.2               \n",
      "pachd               2.1.2               \n"
     ]
    }
   ],
   "source": [
    "!pachctl version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f21caa8f-e072-4121-8efb-968bfd442b55",
   "metadata": {},
   "source": [
    "We can always see the help to understand how a particular `pachctl` command works by adding the `--help` flag. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f482ac66-3cc4-4910-9bd2-8d04787b3211",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Access the Pachyderm API.\n",
      "\n",
      "Environment variables:\n",
      "  PACH_CONFIG=<path>, the path where pachctl will attempt to load your config.\n",
      "  JAEGER_ENDPOINT=<host>:<port>, the Jaeger server to connect to, if PACH_TRACE\n",
      "    is set\n",
      "  PACH_TRACE={true,false}, if true, and JAEGER_ENDPOINT is set, attach a Jaeger\n",
      "    trace to any outgoing RPCs.\n",
      "  PACH_TRACE_DURATION=<duration>, the amount of time for which PPS should trace\n",
      "    a pipeline after 'pachctl create-pipeline' (PACH_TRACE must also be set).\n",
      "\n",
      "Usage:\n",
      "  pachctl [command]\n",
      "\n",
      "Administration Commands:\n",
      "  auth         Auth commands manage access to data in a Pachyderm cluster\n",
      "  enterprise   Enterprise commands enable Pachyderm Enterprise features\n",
      "  idp          Commands to manage identity provider integrations\n",
      "\n",
      "Commands by Action:\n",
      "  copy         Copy a Pachyderm resource.\n",
      "  create       Create a new instance of a Pachyderm resource.\n",
      "  delete       Delete an existing Pachyderm resource.\n",
      "  diff         Show the differences between two Pachyderm resources.\n",
      "  edit         Edit the value of an existing Pachyderm resource.\n",
      "  finish       Finish a Pachyderm resource.\n",
      "  get          Get the raw data represented by a Pachyderm resource.\n",
      "  glob         Print a list of Pachyderm resources matching a glob pattern.\n",
      "  inspect      Show detailed information about a Pachyderm resource.\n",
      "  list         Print a list of Pachyderm resources of a specific type.\n",
      "  put          Insert data into Pachyderm.\n",
      "  restart      Cancel and restart an ongoing task.\n",
      "  squash       Squash an existing Pachyderm resource.\n",
      "  start        Start a Pachyderm resource.\n",
      "  stop         Cancel an ongoing task.\n",
      "  subscribe    Wait for notifications of changes to a Pachyderm resource.\n",
      "  update       Change the properties of an existing Pachyderm resource.\n",
      "  wait         Wait for the side-effects of a Pachyderm resource to propagate.\n",
      "\n",
      "Other Commands:\n",
      "  completion   Print or install terminal completion code.\n",
      "  config       Manages the pachyderm config.\n",
      "  debug        Debug commands for analyzing a running cluster.\n",
      "  exit         Exit the pachctl shell.\n",
      "  fsck         Run a file system consistency check on pfs.\n",
      "  license      License commmands manage the Enterprise License service\n",
      "  logs         Return logs from a job.\n",
      "  mount        Mount pfs locally. This command blocks.\n",
      "  port-forward Forward a port on the local machine to pachd. This command blocks.\n",
      "  resume       Resume a stopped task.\n",
      "  run          Manually run a Pachyderm resource.\n",
      "  shell        Run the pachyderm shell.\n",
      "  unmount      Unmount pfs.\n",
      "  version      Print Pachyderm version information.\n",
      "\n",
      "Additional help topics:\n",
      "  branch       Docs for branches.\n",
      "  commit       Docs for commits.\n",
      "  datum        Docs for datums.\n",
      "  file         Docs for files.\n",
      "  job          Docs for jobs.\n",
      "  object       Docs for objects.\n",
      "  pipeline     Docs for pipelines.\n",
      "  repo         Docs for repos.\n",
      "  transaction  Docs for transactions.\n",
      "\n",
      "Use \"pachctl [command] --help\" for more information about a command.\n"
     ]
    }
   ],
   "source": [
    "!pachctl --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b84fef61-4c29-4e4d-bd2e-dc88585fafa0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Create a new repo.\n",
      "\n",
      "Usage:\n",
      "  pachctl create repo <repo> [flags]\n",
      "\n",
      "Flags:\n",
      "  -d, --description string   A description of the repo.\n",
      "  -h, --help                 help for repo\n",
      "\n",
      "Global Flags:\n",
      "      --no-color   Turn off colors.\n",
      "  -v, --verbose    Output verbose logs\n"
     ]
    }
   ],
   "source": [
    "!pachctl create repo --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc8056eb-a409-4170-961a-c1712c9be27d",
   "metadata": {},
   "source": [
    "## Pachyderm Data Repositories"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c017a3ad-1394-4248-8418-ff38eba53474",
   "metadata": {},
   "source": [
    "Pachyderm organizes data into data repositories. This is somewhat similar to git as we'll see, but scales much better for all file types, such as images, machine learning models, csv files, and many others."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68fca77a-a8a5-48bd-9715-ee9c040678a4",
   "metadata": {},
   "source": [
    "Let's first start by creating a data repository. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f93658c4-92d3-4db7-a3af-c25bcc9ba10a",
   "metadata": {},
   "source": [
    "### Create a data repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "78a24648-619e-4f69-8699-917cc95c0621",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl create repo data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a4c72ca0-55ce-44e9-a575-ab5a24ce94bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME CREATED      SIZE (MASTER) ACCESS LEVEL \n",
      "data 1 second ago ≤ 0B          [repoOwner]   \n"
     ]
    }
   ],
   "source": [
    "!pachctl list repo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04fd661d-0259-4f77-91af-57cf9122f3a2",
   "metadata": {},
   "source": [
    "A data repository, similar to a git repository, will be what we use to organize and reference data. \n",
    "\n",
    "We can also view and explore our data repository in the Pachyderm [Console](https://docs.pachyderm.com/2.0.x-beta/getting_started/beginner_tutorial/#exploring-your-dag-in-pachyderm-console), which should look something like the following.\n",
    "\n",
    "<img src=\"images/console_repo.png\" alt=\"Drawing\" style=\"width: 300px;\"/>\n",
    "\n",
    "\n",
    "\n",
    "When we list our repos, we can see that we have an empty data repository, so let's add some data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0556f450-d9c2-4254-8849-0000d6feb843",
   "metadata": {},
   "source": [
    "### Add data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8096f8a4-d365-4f4b-bd4a-314bbedbfab8",
   "metadata": {},
   "source": [
    "First, we'll create a small csv file locally with some of the iris data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a439d383-933f-4e6e-9da8-c30ac75e47c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /tmp/iris.csv\n"
     ]
    }
   ],
   "source": [
    "%%writefile /tmp/iris.csv\n",
    "5.1,3.5,1.4,0.2,Iris-setosa\n",
    "4.9,3.0,1.4,0.2,Iris-setosa\n",
    "4.7,3.2,1.3,0.2,Iris-setosa\n",
    "4.6,3.1,1.5,0.2,Iris-setosa\n",
    "7.0,3.2,4.7,1.4,Iris-versicolor\n",
    "6.4,3.2,4.5,1.5,Iris-versicolor\n",
    "6.9,3.1,4.9,1.5,Iris-versicolor\n",
    "5.5,2.3,4.0,1.3,Iris-versicolor\n",
    "6.3,3.3,6.0,2.5,Iris-virginica\n",
    "5.8,2.7,5.1,1.9,Iris-virginica\n",
    "7.1,3.0,5.9,2.1,Iris-virginica\n",
    "6.3,2.9,5.6,1.8,Iris-virginica"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1209c03-0433-4da5-bbb7-b5e0bfa5f971",
   "metadata": {},
   "source": [
    "Data repositories in Pachyderm automatically track versions of the data placed in them. Similar to Git, we organize our data via branches, so we will push our data to the master branch of our data repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "659d2c65-7f91-4ed9-a4d6-e10afd8ab192",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n"
     ]
    }
   ],
   "source": [
    "!pachctl put file data@master -f /tmp/iris.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29b5e966-4aba-407b-ade6-c9dfa5644b00",
   "metadata": {},
   "source": [
    "We can look at the data that's been uploaded to our data repository, by listing the files on the master branch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "901830e6-9001-4b13-bb8a-50c632c45ed4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME      TYPE SIZE \n",
      "/iris.csv file 364B \n"
     ]
    }
   ],
   "source": [
    "!pachctl list file data@master"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979e3fd6-015e-4076-b6cb-71d197e7b031",
   "metadata": {},
   "source": [
    "### Delete data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d24611ae-6df8-48cd-941f-7ab677c06b12",
   "metadata": {},
   "source": [
    "Similarly, if we want to delete our file, we can do that as well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fb6eff77-f953-4c80-b5a4-787bc12ddf16",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl delete file data@master:/iris.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bbdfdc44-b733-46fa-962c-4743447d3d1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME TYPE SIZE \n"
     ]
    }
   ],
   "source": [
    "!pachctl list file data@master"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59d85083-3eff-47d7-9397-9436d6b52cc1",
   "metadata": {},
   "source": [
    "Now, if we add it back again..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4be7fd21-561a-4aff-ab59-52ddaa128b37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris.csv 364.00 b / 364.00 b [================================] 0s 0.00 b/s\n"
     ]
    }
   ],
   "source": [
    "!pachctl put file data@master -f /tmp/iris.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6080bdca-40a6-413f-94b9-733f3eae51bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME      TYPE SIZE \n",
      "/iris.csv file 364B \n"
     ]
    }
   ],
   "source": [
    "!pachctl list file data@master"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7024ff25-dd76-4d9d-90f4-0909f619f5df",
   "metadata": {},
   "source": [
    "No surprise, our file is there again. But when we list all of the commits that have been made to our repository, we can see the history of data on the master branch."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db59c420-2e17-4f2e-a6fc-26c9ac2c903f",
   "metadata": {},
   "source": [
    "### Data commits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c741f2be-104f-4385-abda-8b637e2d1ad6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPO BRANCH COMMIT                           FINISHED       SIZE ORIGIN DESCRIPTION\n",
      "data master 5221144c1ab940adaae182faa2561ffc 6 seconds ago  364B USER    \n",
      "data master 546de4f481c44470aa27ce5c8df89c4c 10 seconds ago 0B   USER    \n",
      "data master 4030a517ea444c51a87bc76a90c0844f 20 seconds ago 364B USER    \n"
     ]
    }
   ],
   "source": [
    "!pachctl list commit data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2386ecb9-3672-456e-9291-c11da2df0a87",
   "metadata": {},
   "source": [
    "Pachyderm keeps a record of all the changes that happen to the data repository. This way if we ever want to revert to a previous version of our data repository (dataset in this case), we can do it.\n",
    "\n",
    "For example, if we wanted to go back in time to the first file we added, we can move the \"head\" of our master branch to the first commit. To do this, we run the following \n",
    "\n",
    "**Note:** the commit hashes will be different. Copy and past the hash(es) above to run it yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8bcd251b-6d92-4bd8-b275-dc46d1c86d2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl create branch data@master --head 4030a517ea444c51a87bc76a90c0844f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4f687a4a-9b5b-4da2-ad8a-df62587665e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BRANCH HEAD                             TRIGGER \n",
      "master 4030a517ea444c51a87bc76a90c0844f -       \n"
     ]
    }
   ],
   "source": [
    "!pachctl list branch data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dff0fe8-c2ac-4fa0-8285-21db3c8d1ea1",
   "metadata": {},
   "source": [
    "As we can see when we list the history of our branch, we now only see the first commit (the head of our master branch). \n",
    "\n",
    "Let's go back to our most recent commit. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "970190ef-89a5-487b-8c69-901cdbaf2b49",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl create branch data@master --head 5221144c1ab940adaae182faa2561ffc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd0bf9eb-93eb-4401-b4ff-18b2535c16b2",
   "metadata": {},
   "source": [
    "We can also use [Ancestry Syntax](https://docs.pachyderm.com/latest/concepts/data-concepts/history/#ancestry-syntax) to traverse and explore commits. `^` for the parent of the commit or we can reference the commits in numerical order using `.n`, where `n` is the commit number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "971dde23-2fc1-4fca-b427-244c0c866861",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPO BRANCH COMMIT                           FINISHED      SIZE ORIGIN DESCRIPTION\n",
      "data master 546de4f481c44470aa27ce5c8df89c4c 2 minutes ago 0B   USER    \n",
      "data master 4030a517ea444c51a87bc76a90c0844f 2 minutes ago 364B USER    \n"
     ]
    }
   ],
   "source": [
    "!pachctl list commit data@master^"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8021401b-0c48-4e4f-abb3-d0fa0288ed88",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPO BRANCH COMMIT                           FINISHED      SIZE ORIGIN DESCRIPTION\n",
      "data master 4030a517ea444c51a87bc76a90c0844f 2 minutes ago 364B USER    \n"
     ]
    }
   ],
   "source": [
    "!pachctl list commit data@master.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b0337d31-956b-4f4e-9297-ca6a07b5f8ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPO BRANCH COMMIT                           FINISHED      SIZE ORIGIN DESCRIPTION\n",
      "data master 546de4f481c44470aa27ce5c8df89c4c 2 minutes ago 0B   USER    \n",
      "data master 4030a517ea444c51a87bc76a90c0844f 2 minutes ago 364B USER    \n"
     ]
    }
   ],
   "source": [
    "!pachctl list commit data@master.-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "101811b3-c5fe-46bf-af86-34d9e2657e46",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BRANCH HEAD                             TRIGGER \n",
      "master 5221144c1ab940adaae182faa2561ffc -       \n"
     ]
    }
   ],
   "source": [
    "!pachctl list branch data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6de862d-5a1a-4ca7-b658-2eb58c953292",
   "metadata": {},
   "source": [
    "### Awesome Pachyderm Feature - Efficient Storage! \n",
    "\n",
    "If we list our repo info again, we can see that the *entire size* of the repo is just as big as original file, even though we added it a second time! Pachyderm is really smart in how it handles data. It can understand when the content of a file is a duplicate of something it's seen before to minimize the amount of storage needed. \n",
    "\n",
    "This means it's much, much cheaper to store and version data in Pachyderm than any other platform. \n",
    "\n",
    "Note: Deduplication happens per chunk (e.g. 8MBs per chunk), not per file. For more information on why this is better, see [this blog](https://medium.com/@jdoliner/debunking-the-fud-about-data-version-control-implementations-55cbe72014fb) on content-based chunking. \n",
    "\n",
    "(This feature was introduced in Pachyderm 2.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "17ae72ae-002f-4858-93eb-184d0ee1f213",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME CREATED       SIZE (MASTER) ACCESS LEVEL \n",
      "data 4 minutes ago ≤ 364B        [repoOwner]   \n"
     ]
    }
   ],
   "source": [
    "!pachctl list repo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1066ec0f-9d10-46f3-9a37-27abd057d1ff",
   "metadata": {},
   "source": [
    "## Pachyderm Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d97e945-cc60-40e7-aa0e-c5bd01b6ead0",
   "metadata": {},
   "source": [
    "Managing and versioning data by itself is only half the story. Once you have data, you typically want to do something with it, whether it's transform it, run tests on it, or even train a model. \n",
    "\n",
    "**A Pachyderm Pipeline is how you apply code to your data.**\n",
    "\n",
    "Pipelines work seemlessly with data inside your data repositories, but even better, these pipelines can be triggered by your data! \n",
    "\n",
    "This means that we can deploy a pipeline to transform the data from our `data` repo, and anytime we modify our data, the pipeline will automatically re-run. \n",
    "\n",
    "Initially, this can be a hard concept to grasp, so let's walk through an example."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa5b13f-08e9-4a77-bce0-ff526130b034",
   "metadata": {},
   "source": [
    "### Count Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a3d69d6-652a-4a1c-8c40-c31ba270ff6d",
   "metadata": {},
   "source": [
    "Let's say we just want to count the number of lines in our csv file. We can create a Pachyderm Pipeline that looks like the `yaml` below that uses a shell command to count the number of lines (we'll see why we use shell later on)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3768a08b-45ff-4675-b797-3de52d93da0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /tmp/count.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile /tmp/count.yaml\n",
    "pipeline:\n",
    "    name: 'count'\n",
    "description: 'Count the number of lines in a csv file'\n",
    "input:\n",
    "    pfs:\n",
    "        repo: 'data'\n",
    "        branch: 'master'\n",
    "        glob: '/'\n",
    "transform:\n",
    "    image: alpine:3.14.0\n",
    "    cmd: ['/bin/sh']\n",
    "    stdin: ['wc -l /pfs/data/iris.csv > /pfs/out/line_count.txt']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1571ce33-5d77-471a-a965-7a12d4ba0d14",
   "metadata": {},
   "source": [
    "### Pipelines in detail"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "020c934b-5d04-497b-b289-665a04d42a6e",
   "metadata": {},
   "source": [
    "Let's break this pipeline down section by section and explain it: \n",
    "\n",
    "Every pipeline must have a unique name. In our case, we will call this one `count`. It's also good practice to give our pipeline a description to help others know what it does. \n",
    "\n",
    "When the pipeline runs, it will also **create a data repository** `count` for any files created when the pipeline runs. \n",
    "```yaml\n",
    "pipeline:\n",
    "  name: count\n",
    "description: Count the number of lines in a csv file\n",
    "```\n",
    "\n",
    "The `input` section defines what Pachyderm Data Repositories (or other type of input) will be connected to the pipeline. In our case, the `master` branch of our `data` repo will be used. \n",
    "\n",
    "When the pipeline runs, it will map the files from the `master` branch of our `data` repo, into the file system at `/pfs/data/` (`/pfs/` stands for Pachyderm File System). \n",
    "\n",
    "We'll talk more about glob patterns in another tutorial, but in this example, `/` means that every file on the head commit of the master branch is accessible to the the pipeline. \n",
    "\n",
    "```yaml\n",
    "input:\n",
    "  pfs:\n",
    "    repo: data\n",
    "    branch: master\n",
    "    glob: /\n",
    "```\n",
    "\n",
    "The `transform` portion of the pipeline defines what code should be run when the pipeline executes. Pachyderm Pipelines use Docker containers to allow code written in any language to be executed as a pipeline. In this case, we are using a Docker container `alpine:3.14.0` as our Docker image. When this pipeline runs, it execute the `cmd` along with the `stdin` inside our container. \n",
    "\n",
    "Our `stdin` command, will count the number of lines in `/pfs/data/iris.csv` and write the output to `/pfs/out/line_count.txt`. `/pfs/out` is a special location in Pachyderm pipelines. Anything written to this directory will be *commited* to the `count` data repository (automatically created) as the output of the pipeline.\n",
    "\n",
    "```yaml\n",
    "transform:\n",
    "  image: alpine:3.14.0\n",
    "  cmd: ['/bin/sh']\n",
    "  stdin: ['wc -l /pfs/data/iris.csv > /pfs/out/line_count.txt']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2757c80b-635c-42cf-96c5-5066fcd55a91",
   "metadata": {},
   "source": [
    "### Creating pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9b960cf-8f9c-45fa-8667-d5265ad91163",
   "metadata": {},
   "source": [
    "We can submit our pipeline to Pachyderm by using the `create pipeline` command.\n",
    "\n",
    "We can also view our pipelines in the Pachyderm Console as well. Notice it automatically creates the output data repository with the same name. \n",
    "\n",
    "<img src=\"images/console_pipeline.png\" alt=\"Drawing\" style=\"width: 800px;\"/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "dfb064a9-9769-4ab2-8c55-96fb95a4bb3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pachctl create pipeline -f /tmp/count.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f8f7e13-6946-4b5d-ad7a-de59f564d3e3",
   "metadata": {},
   "source": [
    "### Monitor pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25072ac6-8ab8-4df9-b07e-57f400fcd934",
   "metadata": {},
   "source": [
    "If we list our pipelines, we can see the status of them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c5975b48-f940-4530-9770-4cf63c272fe5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME  VERSION INPUT  CREATED       STATE / LAST JOB DESCRIPTION                             \n",
      "count 1       data:/ 3 seconds ago \u001b[32mrunning\u001b[0m / -      Count the number of lines in a csv file \n"
     ]
    }
   ],
   "source": [
    "!pachctl list pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e008be50-7194-4624-8912-ce132f3f02d1",
   "metadata": {},
   "source": [
    "It looks like our pipeline is `running` and the last job succeeded. Let's take a look at the job."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c93360f-d95e-4002-92bf-633b303ecd99",
   "metadata": {},
   "source": [
    "A job is an execution of our pipeline. We can see our job status by running: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "6fde3fbe-cef7-40dd-8038-ca92ac02773d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID                               SUBJOBS PROGRESS CREATED       MODIFIED\n",
      "710fd548c63b47bb95e5545f217ef05f 1       \u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m 3 minutes ago 3 minutes ago \n"
     ]
    }
   ],
   "source": [
    "!pachctl list job"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54bc007a-accc-4265-a324-6fe74cfb1854",
   "metadata": {},
   "source": [
    "We can also see that we have a new data repository called `count` that holds the output of our pipeline. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55c78471-ae43-4cd9-97fc-252ead2fea83",
   "metadata": {},
   "source": [
    "### View pipeline output commits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "cd30ce97-c8c6-4a30-87ba-486bee9507ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME  CREATED       SIZE (MASTER) ACCESS LEVEL \n",
      "count 3 minutes ago ≤ 22B         [repoOwner]  Output repo for pipeline count. \n",
      "data  8 minutes ago ≤ 364B        [repoOwner]                                  \n"
     ]
    }
   ],
   "source": [
    "!pachctl list repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5e502f77-aed6-486e-9b5f-5787efdbff10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME            TYPE SIZE \n",
      "/line_count.txt file 22B  \n"
     ]
    }
   ],
   "source": [
    "!pachctl list file count@master"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2e74f2c-bba3-4beb-89ed-3e8b89123ec7",
   "metadata": {},
   "source": [
    "Let's download the file created by our `count` pipeline and see what's in it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f79c5f91-0168-4d6d-9179-9f9fe2025c15",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/line_count.txt 0.00 b / 22.00 b [-----------------------------] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/line_count.txt 22.00 b / 22.00 b [============================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/line_count.txt 22.00 b / 22.00 b [============================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/line_count.txt 22.00 b / 22.00 b [============================] 0s 0.00 b/s\n"
     ]
    }
   ],
   "source": [
    "!pachctl get file count@master:/line_count.txt -o /tmp/line_count.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c1c37c-6bc5-4bb9-a7ee-dbd564ca8917",
   "metadata": {},
   "source": [
    "We can see that our output file correctly counted the number of lines in our csv file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1c5017b9-1c82-4419-8bb6-a3e3eb133265",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12 /pfs/data/iris.csv\n"
     ]
    }
   ],
   "source": [
    "# Output file\n",
    "!cat /tmp/line_count.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "0fbdfeef-e914-4ea1-bec8-a09a051753fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12 /tmp/iris.csv\n"
     ]
    }
   ],
   "source": [
    "# Original file\n",
    "!wc -l /tmp/iris.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29198a13-ccad-4c6f-a0ae-5502c4483b11",
   "metadata": {},
   "source": [
    "### Data-Driven Pipelines\n",
    "If we recall, all of our pipelines in Pachyderm are data-driven. They are always ready to run whenever the data contained in an input repository changes. So let's do that. Let's update our iris data (this time with 24 lines). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "40896fc6-08f8-432f-87b7-06a6e348b5d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /tmp/iris_v2.csv\n"
     ]
    }
   ],
   "source": [
    "%%writefile /tmp/iris_v2.csv\n",
    "5.1,3.5,1.4,0.2,Iris-setosa\n",
    "4.9,3.0,1.4,0.2,Iris-setosa\n",
    "4.7,3.2,1.3,0.2,Iris-setosa\n",
    "4.6,3.1,1.5,0.2,Iris-setosa\n",
    "7.0,3.2,4.7,1.4,Iris-versicolor\n",
    "6.4,3.2,4.5,1.5,Iris-versicolor\n",
    "6.9,3.1,4.9,1.5,Iris-versicolor\n",
    "5.5,2.3,4.0,1.3,Iris-versicolor\n",
    "6.3,3.3,6.0,2.5,Iris-virginica\n",
    "5.8,2.7,5.1,1.9,Iris-virginica\n",
    "7.1,3.0,5.9,2.1,Iris-virginica\n",
    "6.3,2.9,5.6,1.8,Iris-virginica\n",
    "5.1,3.5,1.4,0.2,Iris-setosa\n",
    "4.9,3.0,1.4,0.2,Iris-setosa\n",
    "4.7,3.2,1.3,0.2,Iris-setosa\n",
    "4.6,3.1,1.5,0.2,Iris-setosa\n",
    "7.0,3.2,4.7,1.4,Iris-versicolor\n",
    "6.4,3.2,4.5,1.5,Iris-versicolor\n",
    "6.9,3.1,4.9,1.5,Iris-versicolor\n",
    "5.5,2.3,4.0,1.3,Iris-versicolor\n",
    "6.3,3.3,6.0,2.5,Iris-virginica\n",
    "5.8,2.7,5.1,1.9,Iris-virginica\n",
    "7.1,3.0,5.9,2.1,Iris-virginica\n",
    "6.3,2.9,5.6,1.8,Iris-virginica"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ca82ff5-f369-4d88-8c9a-8897354823ec",
   "metadata": {},
   "source": [
    "We'll overwrite our original file with the command: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "eb4f1157-38b3-4370-8860-0cf52a801159",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/iris_v2.csv 728.00 b / 728.00 b [=============================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris_v2.csv 728.00 b / 728.00 b [=============================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris_v2.csv 728.00 b / 728.00 b [=============================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris_v2.csv 728.00 b / 728.00 b [=============================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris_v2.csv 728.00 b / 728.00 b [=============================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris_v2.csv 728.00 b / 728.00 b [=============================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/iris_v2.csv 728.00 b / 728.00 b [=============================] 0s 0.00 b/s\n"
     ]
    }
   ],
   "source": [
    "!pachctl put file data@master:iris.csv -f /tmp/iris_v2.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "015a1487-9343-495a-a329-175dc37a34e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME      TYPE SIZE \n",
      "/iris.csv file 728B \n"
     ]
    }
   ],
   "source": [
    "!pachctl list file data@master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "9d41888c-af67-461e-b47e-aa16fe51f3c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPO BRANCH COMMIT                           FINISHED       SIZE ORIGIN DESCRIPTION\n",
      "data master 47d8673c69994490bb6a964a995352e7 12 seconds ago 728B USER    \n",
      "data master 5221144c1ab940adaae182faa2561ffc 8 minutes ago  364B USER    \n",
      "data master 546de4f481c44470aa27ce5c8df89c4c 8 minutes ago  0B   USER    \n",
      "data master 4030a517ea444c51a87bc76a90c0844f 8 minutes ago  364B USER    \n"
     ]
    }
   ],
   "source": [
    "!pachctl list commit data@master"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a34f4c2-d210-4320-bf73-c2d0b77b511a",
   "metadata": {},
   "source": [
    "We have a new commit to our `data` repository, so let's see what's happened to our pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "a230a323-2480-4d4c-8935-f6329baaf138",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID                               SUBJOBS PROGRESS CREATED        MODIFIED\n",
      "47d8673c69994490bb6a964a995352e7 1       \u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m 19 seconds ago 19 seconds ago \n",
      "710fd548c63b47bb95e5545f217ef05f 1       \u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m\u001b[32m▇\u001b[0m 4 minutes ago  4 minutes ago  \n"
     ]
    }
   ],
   "source": [
    "!pachctl list job"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60314f13-6954-4239-8fc3-2307b164af7d",
   "metadata": {},
   "source": [
    "We have a new job that has just run. But remember, we only uploaded a file to our input repo. Pachyderm intelligently tells pipelines to run when their input data changes. If we look at the output of our `count` repository, we now see 2 commits. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3b2d5619-4baf-4a7b-826f-0236dc0e5b50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPO  BRANCH COMMIT                           FINISHED       SIZE ORIGIN DESCRIPTION\n",
      "count master 47d8673c69994490bb6a964a995352e7 23 seconds ago 22B  AUTO    \n",
      "count master 710fd548c63b47bb95e5545f217ef05f 3 minutes ago  22B  AUTO    \n"
     ]
    }
   ],
   "source": [
    "!pachctl list commit count@master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "d913e552-a4ef-4472-9859-80092a4ba13b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/line_count_v2.txt 0.00 b / 22.00 b [--------------------------] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/line_count_v2.txt 22.00 b / 22.00 b [=========================] 0s 0.00 b/s\n",
      "\u001b[1A\u001b[J/tmp/line_count_v2.txt 22.00 b / 22.00 b [=========================] 0s 0.00 b/s\n"
     ]
    }
   ],
   "source": [
    "!pachctl get file count@master:/line_count.txt -o /tmp/line_count_v2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "0a96abad-df36-4345-8d7b-622c9d579af4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24 /pfs/data/iris.csv\n"
     ]
    }
   ],
   "source": [
    "!cat /tmp/line_count_v2.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eecc3ddd-8123-4330-8fe8-4f9d2a4a1240",
   "metadata": {},
   "source": [
    "### Awesome Pachyderm Feature - Data Lineage!\n",
    "\n",
    "The data-driven nature of Pachyderm Pipelines allow you to reliably maintain data and process lineage at scale. Combining versioning data with code in Docker containers for pipelines, Pachyderm can be used to automate, debug, and maintain any data + code workflow. \n",
    "\n",
    "For example, if we want to know the lineage of our most recent `line_count.txt`, we can run: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "54660be2-ab65-4c13-b9b5-098b7af1e247",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPO  BRANCH COMMIT                           FINISHED       SIZE ORIGIN DESCRIPTION\n",
      "count master 47d8673c69994490bb6a964a995352e7 32 seconds ago 22B  AUTO    \n",
      "count master 710fd548c63b47bb95e5545f217ef05f 4 minutes ago  22B  AUTO    \n"
     ]
    }
   ],
   "source": [
    "!pachctl list commit count@master"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d4ca264-e570-499c-a269-6d708e833556",
   "metadata": {},
   "source": [
    "This gives us the unique commit for that run of the `count` pipeline. We can use this commit to see the unique combination of inputs and pipelines that resulted in this file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61378f3a-c892-4725-be5f-55368b26be4a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPO       BRANCH COMMIT                           FINISHED       SIZE     ORIGIN DESCRIPTION\n",
      "data       master 47d8673c69994490bb6a964a995352e7 54 seconds ago 728B     USER    \n",
      "count.spec master 47d8673c69994490bb6a964a995352e7 55 seconds ago 0B       ALIAS   \n",
      "count.meta master 47d8673c69994490bb6a964a995352e7 47 seconds ago 1.331KiB AUTO    \n",
      "count      master 47d8673c69994490bb6a964a995352e7 49 seconds ago 22B      AUTO    \n"
     ]
    }
   ],
   "source": [
    "!pachctl list commit 47d8673c69994490bb6a964a995352e7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8baedec4-796a-45a2-b09d-8c2042a0e8e3",
   "metadata": {},
   "source": [
    "We will gloss over some details here, but the important thing is, we can see the commit to the `data` repo was initiated by a `USER`. We can see exactly what commit triggered the pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "75352e1e-cae6-4566-8f5c-67477432d859",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME      TYPE SIZE \n",
      "/iris.csv file 728B \n"
     ]
    }
   ],
   "source": [
    "!pachctl list file data@47d8673c69994490bb6a964a995352e7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "687d45e4-1d7c-4a72-ba83-469b5ab5d1e7",
   "metadata": {},
   "source": [
    "If we inspect the job associated with this commit, then we can get all the information about what pipeline was run on the data from this commit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "54bded50-fb07-4a46-8aa7-6fd6ee0905d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID: 47d8673c69994490bb6a964a995352e7\n",
      "Pipeline: count\n",
      "Started: About a minute ago \n",
      "Duration: 5 seconds \n",
      "State: \u001b[32msuccess\u001b[0m\n",
      "Reason: \n",
      "Processed: 1\n",
      "Failed: 0\n",
      "Skipped: 0\n",
      "Recovered: 0\n",
      "Total: 1\n",
      "Data Downloaded: 728B\n",
      "Data Uploaded: 22B\n",
      "Download Time: Less than a second\n",
      "Process Time: Less than a second\n",
      "Upload Time: Less than a second\n",
      "Datum Timeout: (duration: nil Duration)\n",
      "Job Timeout: (duration: nil Duration)\n",
      "Worker Status:\n",
      "WORKER              JOB                 DATUM               STARTED             \n",
      "Restarts: 0\n",
      "ParallelismSpec: <nil>\n",
      "\n",
      "\n",
      "\n",
      "Input:\n",
      "{\n",
      "  \"pfs\": {\n",
      "    \"name\": \"data\",\n",
      "    \"repo\": \"data\",\n",
      "    \"repo_type\": \"user\",\n",
      "    \"branch\": \"master\",\n",
      "    \"commit\": \"47d8673c69994490bb6a964a995352e7\",\n",
      "    \"glob\": \"/\"\n",
      "  }\n",
      "}\n",
      "\n",
      "Transform:\n",
      "{\n",
      "  \"image\": \"alpine:3.14.0\",\n",
      "  \"cmd\": [\n",
      "    \"/bin/sh\"\n",
      "  ],\n",
      "  \"stdin\": [\n",
      "    \"wc -l /pfs/data/iris.csv > /pfs/out/line_count.txt\"\n",
      "  ]\n",
      "} \n",
      "Output Commit: 47d8673c69994490bb6a964a995352e7 \n"
     ]
    }
   ],
   "source": [
    "!pachctl inspect job count@47d8673c69994490bb6a964a995352e7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fcd1396-a66c-4ec7-bec7-1d0e57cfeb69",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Pachyderm is an incredibly powerful platform, providing the data foundation for machine learning and other data-driven workflows. In this tutorial, we walked through some of the basic concepts that will get you started with Pachyderm. \n",
    "\n",
    "\n",
    "Be sure to check out our [examples](https://github.com/pachyderm/examples) for use-case specific tutorials or [join our community on Slack](https://www.pachyderm.com/slack/) to learn how to apply Pachyderm to your use-case.\n",
    "\n",
    "See our [documentation](https://docs.pachyderm.com/latest/) for more in-depth explainations on the topics covered here. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}