Rankle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rankle\n",
    "\n",
    "_N.B. See the [Jupyter Notebook](https://jupyter.org/) documentation for information on using this document interactively._\n",
    "\n",
    "Rankle is a Clojure library with experiments involving array programming and specifically the [J](http://jsoftware.com) language's implementation of _rank_.\n",
    "\n",
    "If you just want to read this document, skip straight to [Rank in J](#Rank-in-J).\n",
    "\n",
    "If you want to run the cells in this document, locally you need:\n",
    "\n",
    "* [Clojure](https://www.clojure.org/guides/getting_started)\n",
    "* [J](http://jsoftware.com/start.htm)\n",
    "\n",
    "This Jupyter notebook requires the [clojupyter kernel](https://github.com/semperos/clojupyter) to run (note I use a fork that updates the version of Clojure used by the kernel), and expects you to have defined a `J_INSTALL_DIR` environment variable where it can find your local J installation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "1. [Helpers](#Helpers)\n",
    "1. [Rank in J](#Rank-in-J)\n",
    "1. [Rank in Rankle](#Rank-in-Rankle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helpers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ns rankle.ipynb\n",
    "  (:require [cemerick.pomegranate :as pg]\n",
    "            [clojure.string :as str]\n",
    "            [clojure.java.shell :as sh]))\n",
    "\n",
    "(pg/add-classpath \"./src\")\n",
    "\n",
    "(require '[com.semperos.rankle :as r])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### J Execution\n",
    "\n",
    "J code can be executed by passing it over STDIN to the `$J_INSTALL_DIR/bin/jconsole` program. This document provides Clojure functions to make it trivial to execute J code from these Clojure notebook cells, assuming you first setup the `$J_INSTALL_DIR` environment variable and run `jupyter-notebook` such that it sees that definition.\n",
    "    \n",
    "The `j` function takes a Clojure string representing a J program, executes it with J, and prints the results.\n",
    "    \n",
    "The `jdebug` function first prints the Clojure string of J code, then runs J's own `;:` (\"words\") verb which shows how J parses the input, and then runs `j` on the input. This is especially useful when the encoding of the J program in the Clojure string requires escape characters)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⌁ J INPUT ⌁\n",
      "<\"0 i. 3 3\n",
      "\n",
      "⌁ J OUTPUT ⌁\n",
      "┌─┬─┬─┐\n",
      "│0│1│2│\n",
      "├─┼─┼─┤\n",
      "│3│4│5│\n",
      "├─┼─┼─┤\n",
      "│6│7│8│\n",
      "└─┴─┴─┘\n",
      "   \n"
     ]
    }
   ],
   "source": [
    "(def J_INSTALL_DIR \"J_INSTALL_DIR\")\n",
    "(def j-install-dir (System/getenv J_INSTALL_DIR))\n",
    "\n",
    "(if-not (str/blank? j-install-dir)\n",
    "  (def j-cmd (str j-install-dir \"bin/jconsole\"))\n",
    "  (throw (ex-info (str \"You must set a \" J_INSTALL_DIR \" environment variable with the full path \"\n",
    "                       \"to the directory of your J installation.\")\n",
    "                  {:env-var-not-found J_INSTALL_DIR})))\n",
    "\n",
    "(defn j\n",
    "  \"Invoke a J program and print its output.\"\n",
    "  [jcode]\n",
    "  (letfn [(wrap [s] (str s \"\\n exit''\"))\n",
    "          (deindent [s] (subs s 3))]\n",
    "    (println (deindent (:out (sh/sh j-cmd :in (wrap jcode)))))))\n",
    "\n",
    "(defn jdebug \n",
    "  ([jcode] (jdebug jcode nil))\n",
    "  ([jcode {:keys [parse?]\n",
    "           :or {parse? false}}]\n",
    "   (println \"⌁ J INPUT ⌁\")\n",
    "   (println jcode)\n",
    "   (println)\n",
    "   (when parse? (j (str \";: '\" jcode \"'\")))\n",
    "   (println \"⌁ J OUTPUT ⌁\")\n",
    "   (j jcode)))\n",
    "\n",
    "(jdebug \"<\\\"0 i. 3 3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rank in J\n",
    "\n",
    "J features the notion of [rank](https://en.wikipedia.org/wiki/Rank_(J_programming_language) to describe the dimensionality of J data and how its operations behave along those dimensions. Rank is fundamental to J's power and expressivity. Since a complete primer of the J programming language is outside the scope of this document, please refer to J's extensive online documentation as well as the labs and studio examples provided as part of its installation on your system.\n",
    "\n",
    "The rank of a J noun (data) is the number of its dimensions. Scalar values have rank 0; a flat array (a list) has rank 1; a two-dimensional array (a table) has rank 2; and so on.\n",
    "\n",
    "J verbs (functions that only return nouns) also have rank:\n",
    "\n",
    "> When you write a verb, it has a _verb rank_ which tells the maximum rank of operand(s) the verb can handle. Any operand of higher rank is automatically chopped up into _cells_ whose rank does not exceed the rank of the verb. The results of applying the verb on the cells are collected into the final result of the verb.\n",
    "\n",
    "To understand what benefits a first-class concept of rank could provide to a Clojure program, this section demonstrates rank in terms of J's implementation. See the [Rank in Rankle](#Rank-in-Rankle) section for Clojure equivalents provided by Rankle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rank of Data (Nouns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Programmatically, the rank of data is the count of its shape, or expressed differently, the number of its dimensions. In J, the verb [`#` (\"tally\")](https://code.jsoftware.com/wiki/Vocabulary/number#monadic) returns the count of top-level [items](https://code.jsoftware.com/wiki/Vocabulary/AET#Item) in its operand, while [`$` (\"shape of\")]((https://code.jsoftware.com/wiki/Vocabulary/dollar#monadic) returns the full shape of its operand, i.e. it returns an array of the counts of its operand's items, starting at the top-level and traversing nested arrays exhaustively.\n",
    "\n",
    "Follow the code examples below to understand how `#` and `$` work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Count of array of 4 numbers,\n",
    ";; returning the scalar value 4\n",
    "(j \"# 1 2 3 4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Shape of array of 4 numbers,\n",
    ";; returning an _array_ of 1 item, the number 4.\n",
    ";;\n",
    ";; J's printed representation of a single-item array,\n",
    ";; unfortunately, is indistinguishable from a single\n",
    ";; scalar value.\n",
    "(j \"$ 1 2 3 4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 2 3\n",
      "4 5 6\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; A 2-by-3 table:\n",
    "(j \"1 2 3 ,: 4 5 6\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Count of 2-by-3 table,\n",
    ";; returning the scalar value 2,\n",
    ";; since the rows are the table's top-level items.\n",
    "(j \"# 1 2 3 ,: 4 5 6\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 3\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Shape of 2-by-3 table,\n",
    ";; returning an array of counts\n",
    ";; starting at the top level of the data.\n",
    ";; It has 2 rows, each with 3 items.\n",
    "(j \"$ 1 2 3 ,: 4 5 6\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Programmatic definition of the rank of this 2-by-3 table,\n",
    ";; indicating this table is of rank 2 because its shape\n",
    ";; consists of two items.\n",
    "(j \"# $ 1 2 3 ,: 4 5 6\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; The rank of a simple array is 1,\n",
    ";; because its shape is a list with\n",
    ";; a single item.\n",
    "(j \"# $ 1 2 3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; The rank of a scalar is 0, because\n",
    ";; its shape is an empty list.\n",
    "(j \"# $ 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rank of Functions (Verbs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "J verbs have a default rank in their monadic (one argument/operand) and dyadic (two arguments/operands) cases that provide maximally-useful semantics. To quote again from J's wiki:\n",
    "\n",
    "> Most verbs have as small a rank as possible to take maximum advantage of \\[the fact that J automatically chops up operands into cells whose rank does not exceed the rank of the verb\\]. For example, the arithmetic verbs, like +, have rank 0 because they can operate on atoms individually.\n",
    "\n",
    "Since `+` in J has rank `0`, rather than having to write explicit loops or use a function like Clojure's `map`, by default J will apply `+` to every item in its argument, whether that argument be a scalar, a 1-dimensional array, or an N-dimensional array. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11 12 13\n",
      "   \n"
     ]
    }
   ],
   "source": [
    "(j \"1 + 10 11 12\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13 104 1005\n",
      "   \n"
     ]
    }
   ],
   "source": [
    "(j \"10 100 1000 + 3 4 5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   0  100  200  300  400\n",
      " 500  600  700  800  900\n",
      "1000 1100 1200 1300 1400\n",
      "1500 1600 1700 1800 1900\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Array of rank 2:\n",
    "(j \"100 * i. 4 5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 0  1\n",
      " 2  3\n",
      " 4  5\n",
      " 6  7\n",
      " 8  9\n",
      "\n",
      "10 11\n",
      "12 13\n",
      "14 15\n",
      "16 17\n",
      "18 19\n",
      "\n",
      "20 21\n",
      "22 23\n",
      "24 25\n",
      "26 27\n",
      "28 29\n",
      "\n",
      "30 31\n",
      "32 33\n",
      "34 35\n",
      "36 37\n",
      "38 39\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Array of rank 3:\n",
    "(j \"i. 4 5 2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   0    1\n",
      " 102  103\n",
      " 204  205\n",
      " 306  307\n",
      " 408  409\n",
      "\n",
      " 510  511\n",
      " 612  613\n",
      " 714  715\n",
      " 816  817\n",
      " 918  919\n",
      "\n",
      "1020 1021\n",
      "1122 1123\n",
      "1224 1225\n",
      "1326 1327\n",
      "1428 1429\n",
      "\n",
      "1530 1531\n",
      "1632 1633\n",
      "1734 1735\n",
      "1836 1837\n",
      "1938 1939\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Adding the rank 2 and rank 3 arrays.\n",
    ";; This is possible because both share a common frame,\n",
    ";; i.e. their first axes match.\n",
    "(j \"(100 * i. 4 5) + i.4 5 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See [J's wiki page on agreement](https://code.jsoftware.com/wiki/Vocabulary/Agreement) for a detailed explanation of the more complex examples presented above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "J exposes the ability to change a verb's rank when invoked. The [`\"` (rank conjunction)](https://code.jsoftware.com/wiki/Vocabulary/quote) expects a verb as its left operand and a rank number as its right, returning a new verb that behaves like the original but along the specified rank.\n",
    "\n",
    "To investigate the ranks of verbs, we will use the [`b.` (\"verb info\")](https://code.jsoftware.com/wiki/Vocabulary/bdotu) adverb which shows diagnostic information about verbs. With `0` as its right-hand operand, this returns a 3-integer list of:\n",
    "\n",
    "1. The rank of the verb when used monadically (i.e., the rank of the y-argument)\n",
    "1. The rank of the x-argument when used dyadically\n",
    "1. The rank of the y-argument when used dyadically\n",
    "\n",
    "In J's parlance, monadic means \"with one argument\" and dyadic means \"with two arguments.\" J verbs only support monadic and dyadic arities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 0 0\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; The default rank of + is 0 0 0, i.e.:\n",
    ";; operates on individual atoms of x and y,\n",
    ";; producing a result of the same shape\n",
    "(j \"+ b.0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_ _ _\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; The / (\"over\") adverb causes its left operand (a verb)\n",
    ";; to be inserted in-between items of its right operand\n",
    ";; (here, an array). It results in a verb which operates\n",
    ";; on its x and y arguments as a whole, i.e. rank infinity.\n",
    "(j \"+/ b.0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; So by default, this sums the items of a simple list:\n",
    "(j \"+/ 1 2 3 4 5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 1 2 3 4\n",
      "5 6 7 8 9\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; A 2-by-5 table will demonstrate what happens\n",
    ";; when we use +/ with different explicit ranks.\n",
    "(j \"i. 2 5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 7 9 11 13\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; The default rank is infinite, which means the verb created\n",
    ";; by the combination of +/ works on its whole argument by default.\n",
    ";; With this in mind, it will put + in between top-level items of the 2-by-5\n",
    ";; table, i.e., it puts + between the sub-arrays that are one rank lower\n",
    ";; than the overall data, here the rows of the table which are simple lists\n",
    ";; and therefore rank 1:\n",
    "(j \"+/ i. 2 5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 7 9 11 13\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; The above is equivalent to the following:\n",
    "(j \"0 1 2 3 4 + 5 6 7 8 9\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⌁ J INPUT ⌁\n",
      "+/\"1 i. 2 5\n",
      "\n",
      "⌁ J OUTPUT ⌁\n",
      "10 35\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Now if we want to sum differently, we can assign our verb +/\n",
    ";; an explicitly different rank using \" (rank conjunction)\n",
    ";; To target the rank-1 items of this noun, we will assign\n",
    ";; +/ a rank of 1 and thereby sum the rows of the table:\n",
    "(jdebug \"+/\\\"1 i. 2 5\")                 ; jdebug because we have to escape the \" character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⌁ J INPUT ⌁\n",
      "+/\"2 i. 2 5\n",
      "\n",
      "⌁ J OUTPUT ⌁\n",
      "5 7 9 11 13\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Specifying a verb rank of 2 gives us the same behavior\n",
    ";; as the default (infinite) rank in this case, because\n",
    ";; our argument is of rank 2:\n",
    "(jdebug \"+/\\\"2 i. 2 5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⌁ J INPUT ⌁\n",
      "i. 2 5 3\n",
      "\n",
      "⌁ J OUTPUT ⌁\n",
      " 0  1  2\n",
      " 3  4  5\n",
      " 6  7  8\n",
      " 9 10 11\n",
      "12 13 14\n",
      "\n",
      "15 16 17\n",
      "18 19 20\n",
      "21 22 23\n",
      "24 25 26\n",
      "27 28 29\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Using data of a higher rank can demonstrate further:\n",
    "(jdebug \"i. 2 5 3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⌁ J INPUT ⌁\n",
      "+/ i. 2 5 3\n",
      "\n",
      "⌁ J OUTPUT ⌁\n",
      "15 17 19\n",
      "21 23 25\n",
      "27 29 31\n",
      "33 35 37\n",
      "39 41 43\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Default rank of +/ again is infinite, so\n",
    ";; it operates on top-level items of the data,\n",
    ";; which in this case are 5-by-3 tables:\n",
    "(jdebug \"+/ i. 2 5 3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 0  1  2\n",
      " 3  4  5\n",
      " 6  7  8\n",
      " 9 10 11\n",
      "12 13 14\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Which is equivalent to adding:\n",
    "(j \"i. 5 3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15 16 17\n",
      "18 19 20\n",
      "21 22 23\n",
      "24 25 26\n",
      "27 28 29\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; to:\n",
    "(j \"15 + i. 5 3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15 17 19\n",
      "21 23 25\n",
      "27 29 31\n",
      "33 35 37\n",
      "39 41 43\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; written:\n",
    "(j \"(i. 5 3) + (15 + i. 5 3)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⌁ J INPUT ⌁\n",
      "+/\"1 i. 2 5 3\n",
      "\n",
      "⌁ J OUTPUT ⌁\n",
      " 3 12 21 30 39\n",
      "48 57 66 75 84\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Use +/ with rank 1,\n",
    ";; which sums the rows of each constituent table,\n",
    ";; because they are the only rank-1 cells in this overall data,\n",
    ";; returning a 2-by-5 shaped result:\n",
    "(jdebug \"+/\\\"1 i. 2 5 3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⌁ J INPUT ⌁\n",
      "+/\"2 i. 2 5 3\n",
      "\n",
      "⌁ J OUTPUT ⌁\n",
      " 30  35  40\n",
      "105 110 115\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; Using +/ with rank 2 targets the rank-2 cells\n",
    ";; of this data. The rank-2 cells are the two 5-by-3\n",
    ";; tables.\n",
    ";; This results in summing the columns of each\n",
    ";; constituent table:\n",
    "(jdebug \"+/\\\"2 i. 2 5 3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 35 40\n",
      "   \n"
     ]
    }
   ],
   "source": [
    ";; This is equivalent to putting + between\n",
    ";; the rows of each constituent table. The first row\n",
    ";; in the above output `30 35 40` is obtained by\n",
    ";; this equivalent expression, which shows us manually\n",
    ";; adding the rows of the first constituent table:\n",
    "(j \"0 1 2 + 3 4 5 + 6 7 8 + 9 10 11 + 12 13 14\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rank in Rankle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rankle contains many table-related experiments and utilities, but this section focuses on those related to rank.\n",
    "\n",
    "The namespace alias `r` is used for vars in the `com.semperos.rankle` namespace."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rank of Data\n",
    "\n",
    "As in J, the rank of data is determined by the length of its shape. The shape of data is calculated using the `r/shape` function and behaves as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/shape 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/shape [4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/shape [4 5 6])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2 3]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/shape [[1 2 3] [4 5 6]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_N.B.: Using `r/shape` with maps is not currently defined._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rank of the above examples are therefore:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/rank 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/rank [4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/rank [4 5 6])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/rank [[1 2 3] [4 5 6]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_N.B.: The shape and rank of strings, maps, and ragged collections is not currently defined._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rank of Functions\n",
    "\n",
    "Currently, the rank of a function is indicated by adding a `:rank` entry to the function var's metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "##Inf"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(:rank (meta #'r/count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The definition of `count` defers completely to `clojure.core/count` and so behaves as you would expect:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0 3 2]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(mapv count [[], [1 2 3], [[4 5 6] [7 8 9]]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a single argument, the `r/rank` function simply returns the rank of its argument. With two arguments, `r/rank` expects the first to be a function and the second to be a number for the new rank, returning a function that applies the given function with respect to the given rank.\n",
    "\n",
    "We will use `r/count` to demonstrate in Clojure what we demonstrated above in J."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/count [])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(r/count [1 2 3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0  1  2\n",
      "  3  4  5\n"
     ]
    }
   ],
   "source": [
    ";; The `r/print-aligned` function gives us a J-like\n",
    ";; print-out of multidimensional data.\n",
    "(r/print-aligned [[0 1 2] [3 4 5]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0  1  2\n",
      "  3  4  5\n"
     ]
    }
   ],
   "source": [
    ";; We also have an `r/reshape` which behaves like J's dyadic `$`\n",
    "(r/print-aligned (r/reshape [2 3] (range 6)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(def tbl-2x3 (r/reshape [2 3] (range 6)))\n",
    "(r/count tbl-2x3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3 3)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((r/rank r/count 1) tbl-2x3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((r/rank r/count 2) tbl-2x3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0  1  2  3\n",
      "  4  5  6  7\n",
      "  8  9 10 11\n",
      "\n",
      " 12 13 14 15\n",
      " 16 17 18 19\n",
      " 20 21 22 23\n",
      "\n"
     ]
    }
   ],
   "source": [
    ";; Higher rank lets us do more:\n",
    "(def tbl-2x3x4 (r/reshape [2 3 4] (range 24)))\n",
    "(r/print-aligned tbl-2x3x4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ";; Rank infinity means that r/count works\n",
    ";; on the data as a whole, counting its top-level\n",
    ";; items:\n",
    "(r/count tbl-2x3x4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4 4 4) (4 4 4))"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ";; By changing the rank of r/count to 1, it will\n",
    ";; try to count those cells (sub-parts of the overall data)\n",
    ";; that are of rank 1, in this case, it will count the items in the\n",
    ";; flat lists (which are rank 1 by definition)that make up the rows\n",
    ";; of each constituent 3x4 table:\n",
    "((r/rank r/count 1) tbl-2x3x4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3 3)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ";; With rank 2, the r/count function will search out the\n",
    ";; cells that are rank 2 and count those. The two tables\n",
    ";; in this data are, by definition, rank 2, and thus count\n",
    ";; returns the number of top-level items (rows) in those tables:\n",
    "((r/rank r/count 2) tbl-2x3x4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ";; The entire tbl-2x3x4 data structure is of rank 3,\n",
    ";; so changing r/count from rank ##Inf to rank 3 is\n",
    ";; equivalent in this case:\n",
    "((r/rank r/count 3) tbl-2x3x4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To hammer the point home, with normal Clojure `count`, we would use `map` to traverse the 2-by-3-by-4 table as follows to get counts of the lower-ranked collections within:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(count tbl-2x3x4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3 3)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ";; Count the rank-2 cells:\n",
    "(map count tbl-2x3x4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4 4 4) (4 4 4))"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ";; Count the rank-1 cells:\n",
    "(map (partial map count) tbl-2x3x4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's closely compare this to working with the smaller 2-by-3 table from earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(count tbl-2x3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3 3)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(map count tbl-2x3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of `tbl-2x3x4`, doing `(map count ___)` results in counting rank-2 cells (tables) within the overall data, returning the number of rows in each table. In the case of `tbl-2x3`, doing the same `(map count ___)` results in counting rank-1 cells (lists) within the overall data, returning the number of items in each row of the data.\n",
    "\n",
    "So even though a first-class concept of rank allows us to perform `map`-like operations, this comparison of `(map count ___)` underscores a fundamental difference in the nature of their abstractions:\n",
    "\n",
    "> Clojure sequence operations allow us to apply functions to data by navigating the data structurally, whereas rank allows us to apply functions to data by targeting specific dimensions of the data, regardless of its overall dimensionality.\n",
    "\n",
    "The kind of abstractions that rank affords, then, provide value only when working with data that has been arranged into a homogeneous, non-ragged representation like a multidimensional array. While this sounds like a severe limitation, in practice it allows us to use the many powerful array-programming approaches developed by the APL and J communities over the last 50 years, which, although based in larger part on properties of arithmetic and algebra than other languages, provide concise solutions both to analytical and \"enteprise\" problems.\n",
    "\n",
    "To continue, try reading the [Practical Rankle Examples](PracticalRankleExamples.ipynb) notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Clojure",
   "language": "clojure",
   "name": "clojure"
  },
  "language_info": {
   "file_extension": ".clj",
   "mimetype": "text/x-clojure",
   "name": "clojure",
   "version": "1.10.0-RC2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}