AtomAndElement.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This may change but it will get you going with package buckysoap."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the command line:\n",
    "\n",
    "    pip install numpy\n",
    "    pip install lazy\n",
    "    pip install filelock\n",
    "    git clone https://github.com/leonhardbrenner/buckysoap.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### At the top of your script include the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path += ['/home/lbrenner/buckysoap/src']\n",
    "import buckysoap as bs\n",
    "from buckysoap import Atom, Element, Ring, Field\n",
    "\n",
    "#Monkey patch Element to display rows\n",
    "element_display = Element.display\n",
    "def display(element, *a, **kw):\n",
    "    element_display(element, *a, **kw)\n",
    "    print \"(%s rows)\" % len(element)\n",
    "    return element\n",
    "Element.display = display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Meet the Atom."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's create a few and prove that they are Atoms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
      "<class 'buckysoap.atom.Atom'>\n"
     ]
    }
   ],
   "source": [
    "print bs.zeros(10, int)\n",
    "print bs.ones(10, int)\n",
    "print bs.arange(10)\n",
    "print type(bs.zeros(10, int))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some typical operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n",
      "[1.0, 0.5, 0.33333333333333331, 0.25, 0.20000000000000001, 0.16666666666666666, 0.14285714285714285, 0.125, 0.1111111111111111, 0.10000000000000001]\n"
     ]
    }
   ],
   "source": [
    "print bs.ones(10, int) + bs.arange(10)\n",
    "print bs.ones(10) / (bs.arange(10) + 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous Atoms are alot like np.ndarray because Atom just extends np.ndarray. Let's look at what functionality Atom adds to np.ndarray. We are going to pass a list of lists where some of\n",
    "the lists contain None, some are EMPTY and the root list may contain None."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x =  [[0, 1, 2, None, 4], None, [], [5, 6]]\n",
      "x.asarray() =  [0 1 2 0 4 5 6]\n",
      "x.mask() =  [ True  True  True False  True  True  True]\n",
      "x.bincounts =  [Atom([5, None, 0, 2])]\n"
     ]
    }
   ],
   "source": [
    "x = bs.Atom.fromlist([[0, 1, 2, None, 4], None, [], [5, 6]])\n",
    "print 'x = ', x\n",
    "print 'x.asarray() = ', x.asarray() #This np.ndarray portion of the data\n",
    "print 'x.mask() = ',    x.mask      #This exists on all Atoms to signify None\n",
    "print 'x.bincounts = ', x.bincounts #This represents the bincounts on each axis of our Atom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this last example we can see that an Atom is made up of: \n",
    "\n",
    "    x.asarray() - the data in the form of a ndarray\n",
    "    x.mask() - the mask to be applied to the data\n",
    "    x.bincounts - bins is a common term used for aggregation. Notice that it is a list. You will see why in the next example.\n",
    "\n",
    "In this example we will look at how a list of lists of lists is represented by an Atom.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y =  [[[0, 1, 2, None, 4], None, [5, 6]], [[7, 8, 9, 4], [None, 10, 11]]]\n",
      "y.asarray() =  [ 0  1  2  0  4  5  6  7  8  9  4  0 10 11]\n",
      "y.mask() =  [ True  True  True False  True  True  True  True  True  True  True False\n",
      "  True  True]\n",
      "y.bincounts =  [Atom([5, None, 2, 4, 3]), Atom([3, 2])]\n",
      "x.cardinality =  2\n",
      "y.cardinality =  3\n",
      "x.counts =  [5, None, 0, 2]\n",
      "y.counts =  [[5, None, 2], [4, 3]]\n"
     ]
    }
   ],
   "source": [
    "y = bs.Atom.fromlist([[[0, 1, 2, None, 4], None, [5, 6]], [[7, 8, 9, 4], [None, 10, 11]]])\n",
    "print 'y = ', y\n",
    "print 'y.asarray() = ', y.asarray()\n",
    "print 'y.mask() = ',    y.mask\n",
    "print 'y.bincounts = ', y.bincounts\n",
    "print 'x.cardinality = ', x.cardinality\n",
    "print 'y.cardinality = ', y.cardinality\n",
    "print 'x.counts = ', x.counts\n",
    "print 'y.counts = ', y.counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's operate on these Atoms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x * 2 =  [[0, 2, 4, None, 8], None, [], [10, 12]]\n",
      "x + 2 =  [[2, 3, 4, None, 6], None, [], [7, 8]]\n",
      "y - 2 =  [[[-2, -1, 0, None, 2], None, [3, 4]], [[5, 6, 7, 2], [None, 8, 9]]]\n",
      "(y + 1) / 2 =  [[[0.5, 1.0, 1.5, None, 2.5], None, [3.0, 3.5]], [[4.0, 4.5, 5.0, 2.5], [None, 5.5, 6.0]]]\n",
      "x =  [[0, 1, 2, None, 4], None, [], [5, 6]]\n",
      "x.sum() =  [7, None, 0, 11]\n",
      "x.sum().sum() =  18\n",
      "y =  [[[0, 1, 2, None, 4], None, [5, 6]], [[7, 8, 9, 4], [None, 10, 11]]]\n",
      "y.sum() =  [[7, None, 11], [28, 21]]\n",
      "y.sum().sum() =  [18, 49]\n",
      "y.sum().sum().sum() =  67\n",
      "y.average() =  [[1.75, None, 5.5], [7.0, 10.5]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lbrenner/buckysoap/src/buckysoap/atom.py:485: RuntimeWarning: invalid value encountered in divide\n",
      "  return self.__class__(getattr(self.asarray(), name)(other.asarray()),\n"
     ]
    }
   ],
   "source": [
    "print \"x * 2 = \", x * 2\n",
    "print \"x + 2 = \", x + 2\n",
    "print \"y - 2 = \", y - 2\n",
    "print \"(y + 1) / 2 = \", (y + 1) / 2\n",
    "print \"x = \", x\n",
    "print \"x.sum() = \", x.sum()\n",
    "print \"x.sum().sum() = \", x.sum().sum()\n",
    "print \"y = \", y\n",
    "print \"y.sum() = \", y.sum()\n",
    "print \"y.sum().sum() = \", y.sum().sum()\n",
    "print \"y.sum().sum().sum() = \", y.sum().sum().sum()\n",
    "print \"y.average() = \", y.average()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can stack Atoms together. Note: Atoms must have the same cardinality. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y.vstack(y) =  [[[0, 1, 2, None, 4], None, [5, 6]], [[7, 8, 9, 4], [None, 10, 11]], [[0, 1, 2, None, 4], None, [5, 6]], [[7, 8, 9, 4], [None, 10, 11]]]\n"
     ]
    }
   ],
   "source": [
    "print \"y.vstack(y) = \", y.vstack(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indexing is the big trick in Atom. Using the bincounts I am able index these multidemensional arrays at each level of the tree. The indexing is mostly handled using run_length.index which uses run_length.range. Here is a link to the sourcecode:\n",
    "    https://github.com/leonhardbrenner/buckysoap/blob/master/src/buckysoap/run_length.py\n",
    "    \n",
    "which lead to developement of Atom then Element run_length index is used in __getitem__ of Atom. Take a look:\n",
    "    https://github.com/leonhardbrenner/buckysoap/blob/master/src/buckysoap/atom.py\n",
    "    \n",
    "Now let's do some indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z = [100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111]\n",
      "y = [[[0, 1, 2, None, 4], None, [5, 6]], [[7, 8, 9, 4], [None, 10, 11]]]\n",
      "z[y] = [[[100, 101, 102, None, 104], None, [105, 106]], [[107, 108, 109, 104], [None, 110, 111]]]\n"
     ]
    }
   ],
   "source": [
    "z = bs.arange(12) + 100\n",
    "print 'z =', z\n",
    "print 'y =', y\n",
    "print 'z[y] =', z[y]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this last example the cardinality of the index is passed on to the Atom being indexed. In this next example the Atom being indexed will have a cardinality to 2 and the index will have a cardinality of 3. The resulting Atom will have cardinality of 4. All of this is happening with out looping through list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z2 = [[0], [1, 2], [3, 4, 5], [6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19, 20], [21, 22, 23, 24, 25, 26, 27], [28, 29, 30, 31, 32, 33, 34, 35], [36, 37, 38, 39, 40, 41, 42, 43, 44], [45, 46, 47, 48, 49, 50, 51, 52, 53, 54], [55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65], [66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77]]\n",
      "z2[y] = [[[[0], [1, 2], [3, 4, 5], None, [10, 11, 12, 13, 14]], None, [[15, 16, 17, 18, 19, 20], [21, 22, 23, 24, 25, 26, 27]]], [[[28, 29, 30, 31, 32, 33, 34, 35], [36, 37, 38, 39, 40, 41, 42, 43, 44], [45, 46, 47, 48, 49, 50, 51, 52, 53, 54], [10, 11, 12, 13, 14]], [None, [55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65], [66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77]]]]\n",
      "z2.cardinality =  2\n",
      "y.cardinality =  3\n",
      "z2[y].cardinality =  4\n"
     ]
    }
   ],
   "source": [
    "z2 = bs.arange(78)\n",
    "z2.bincounts.append((bs.arange(12) + 1))\n",
    "print 'z2 =', z2\n",
    "print 'z2[y] =', z2[y]\n",
    "print 'z2.cardinality = ', z2.cardinality\n",
    "print 'y.cardinality = ', y.cardinality\n",
    "print 'z2[y].cardinality = ', z2[y].cardinality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining Atoms to form an Element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An Element can combine multiple Atoms and can be combined with other Atoms or Elements. In this first example we will construct an Atom then assign columns: range, zero and one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zero range one\n",
      "   0     0   1\n",
      "   0     1   1\n",
      "   0     2   1\n",
      "   0     3   1\n",
      "   0     4   1\n",
      "   0     5   1\n",
      "   0     6   1\n",
      "   0     7   1\n",
      "   0     8   1\n",
      "   0     9   1\n",
      "(10 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b441ed0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e1 = Element(name='Sample')(\n",
    "    range = bs.arange(10),\n",
    "    zero = bs.zeros(10, int),\n",
    "    one = bs.ones(10, int))\n",
    "e1.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will use currying to create Atom(group_id)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zero range one group_id1 group_id2\n",
      "   0     0   1         0         0\n",
      "   0     1   1         2         2\n",
      "   0     2   1         0         4\n",
      "   0     3   1         2         0\n",
      "   0     4   1         0         2\n",
      "   0     5   1         2         4\n",
      "   0     6   1         0         0\n",
      "   0     7   1         2         2\n",
      "   0     8   1         0         4\n",
      "   0     9   1         2         0\n",
      "(10 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b452110>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e2 = e1(\n",
    "    group_id1 = lambda x: (x.range % 2) * 2,\n",
    "    group_id2 = lambda x: (x.range % 3) * 2\n",
    ")\n",
    "e2.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for group operations. In the Atom indexing example we show how an index imposes it's cardinality on the Atom being indexed. This makes it very easy to build a group operation. The implementation is only a few lines so take a look but the group index is the 2 column sort_index and the bincounts which are calculated by comparing x[:-1] and x[1:]. The __index__ is applied in __getattr__ when Element needs to deligate to __source__. This is more then you need to know."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample[0]\n",
      "    zero=[0, 0]\n",
      "    range=[0, 6]\n",
      "    one=[1, 1]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "Sample[1]\n",
      "    zero=[0]\n",
      "    range=[4]\n",
      "    one=[1]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "Sample[2]\n",
      "    zero=[0, 0]\n",
      "    range=[2, 8]\n",
      "    one=[1, 1]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "Sample[3]\n",
      "    zero=[0, 0]\n",
      "    range=[3, 9]\n",
      "    one=[1, 1]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "Sample[4]\n",
      "    zero=[0, 0]\n",
      "    range=[1, 7]\n",
      "    one=[1, 1]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "Sample[5]\n",
      "    zero=[0]\n",
      "    range=[5]\n",
      "    one=[1]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "(6 rows)\n",
      "e3.__index__ =  [[0, 6], [4], [2, 8], [3, 9], [1, 7], [5]]\n",
      "e3.__source__.display():\n",
      "zero range one group_id1 group_id2\n",
      "   0     0   1         0         0\n",
      "   0     1   1         2         2\n",
      "   0     2   1         0         4\n",
      "   0     3   1         2         0\n",
      "   0     4   1         0         2\n",
      "   0     5   1         2         4\n",
      "   0     6   1         0         0\n",
      "   0     7   1         2         2\n",
      "   0     8   1         0         4\n",
      "   0     9   1         2         0\n",
      "(10 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b452110>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e3 = e2.group('group_id1,group_id2')\n",
    "e3.display()\n",
    "print \"e3.__index__ = \", e3.__index__\n",
    "print \"e3.__source__.display():\"\n",
    "e3.__source__.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try some now let's try sum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample[0]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[0, 6]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[1]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "    range_sum=4\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[4]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "Sample[2]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[2, 8]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[3]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[3, 9]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[4]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[1, 7]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[5]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "    range_sum=5\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[5]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "(6 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b45bad0>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e4 = (\n",
    "    e3(range_sum = lambda x: x.range.sum(),\n",
    "       zero_sum = lambda x: x.zero.sum(),\n",
    "       one_sum = lambda x: x.one.sum())\n",
    "    ('group_id1,group_id2,range_sum,zero_sum,one_sum,range,zero,one'))\n",
    "e4.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample[0]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[0, 6]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[1]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "    range_sum=4\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[4]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "Sample[2]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[2, 8]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[3]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[3, 9]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[4]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[1, 7]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[5]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "    range_sum=5\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[5]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "(6 rows)\n",
      "Sample[0]\n",
      "    zero=[0, 0]\n",
      "    range=[0, 6]\n",
      "    one=[1, 1]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "Sample[1]\n",
      "    zero=[0]\n",
      "    range=[4]\n",
      "    one=[1]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "Sample[2]\n",
      "    zero=[0, 0]\n",
      "    range=[2, 8]\n",
      "    one=[1, 1]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "Sample[3]\n",
      "    zero=[0, 0]\n",
      "    range=[3, 9]\n",
      "    one=[1, 1]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "Sample[4]\n",
      "    zero=[0, 0]\n",
      "    range=[1, 7]\n",
      "    one=[1, 1]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "Sample[5]\n",
      "    zero=[0]\n",
      "    range=[5]\n",
      "    one=[1]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "(6 rows)\n",
      "zero range one group_id1 group_id2\n",
      "   0     0   1         0         0\n",
      "   0     1   1         2         2\n",
      "   0     2   1         0         4\n",
      "   0     3   1         2         0\n",
      "   0     4   1         0         2\n",
      "   0     5   1         2         4\n",
      "   0     6   1         0         0\n",
      "   0     7   1         2         2\n",
      "   0     8   1         0         4\n",
      "   0     9   1         2         0\n",
      "(10 rows)\n",
      "zero range one\n",
      "   0     0   1\n",
      "   0     1   1\n",
      "   0     2   1\n",
      "   0     3   1\n",
      "   0     4   1\n",
      "   0     5   1\n",
      "   0     6   1\n",
      "   0     7   1\n",
      "   0     8   1\n",
      "   0     9   1\n",
      "(10 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b441ed0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e4.display()\n",
    "e3.display()\n",
    "e2.display()\n",
    "e1.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at the expand method. The examples should be clear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample[0]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=0\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[1]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=6\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[2]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "    range_sum=4\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=4\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "Sample[3]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=2\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[4]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=8\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[5]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=3\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[6]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=9\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[7]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=1\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[8]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=7\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[9]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "    range_sum=5\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=5\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "(10 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b4521d0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e4.expand('range').display()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "group_id1 group_id2 range_sum zero_sum one_sum range zero one\n",
      "        0         0         6        0       2     0    0   1\n",
      "        0         0         6        0       2     6    0   1\n",
      "        0         2         4        0       1     4    0   1\n",
      "        0         4        10        0       2     2    0   1\n",
      "        0         4        10        0       2     8    0   1\n",
      "        2         0        12        0       2     3    0   1\n",
      "        2         0        12        0       2     9    0   1\n",
      "        2         2         8        0       2     1    0   1\n",
      "        2         2         8        0       2     7    0   1\n",
      "        2         4         5        0       1     5    0   1\n",
      "(10 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b4525d0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e4.expand('range,zero,one').display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's introduce the join which is also implemented using nothing but Numpy. Take a look at the code:\n",
    "\n",
    "https://github.com/leonhardbrenner/buckysoap/blob/master/src/buckysoap/join.py\n",
    "\n",
    "The Element uses this packaage method join. Let's start with the setup. I am creating Element(groups) which I will join to Element(e4)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "group_id    name\n",
      "       0 group_5\n",
      "       1 group_4\n",
      "       2 group_3\n",
      "       3 group_2\n",
      "       4 group_1\n",
      "       5 group_0\n",
      "(6 rows)\n",
      "Sample[0]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[0, 6]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[1]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "    range_sum=4\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[4]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "Sample[2]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[2, 8]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[3]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[3, 9]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[4]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[1, 7]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "Sample[5]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "    range_sum=5\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[5]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "(6 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b45bad0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "groups = (\n",
    "    bs.Element(name='Groups')\n",
    "    (group_id = bs.arange(6))\n",
    "    (name = lambda x: ['group_%d' % y for y in x.group_id[::-1]]))\n",
    "groups.display()\n",
    "e4.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the join:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample[0]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[0, 6]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=0\n",
      "        name=group_5\n",
      "Sample[1]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[3, 9]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=0\n",
      "        name=group_5\n",
      "Sample[2]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "    range_sum=4\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[4]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "    Groups\n",
      "        group_id=2\n",
      "        name=group_3\n",
      "Sample[3]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[1, 7]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=2\n",
      "        name=group_3\n",
      "Sample[4]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[2, 8]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=4\n",
      "        name=group_1\n",
      "Sample[5]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "    range_sum=5\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[5]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "    Groups\n",
      "        group_id=4\n",
      "        name=group_1\n",
      "(6 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b45bd90>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "join = e4.inner(groups, group_id='group_id2')\n",
    "join.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can sort these new elements and we can stack them together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample[0]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[2, 8]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=4\n",
      "        name=group_1\n",
      "Sample[1]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "    range_sum=5\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[5]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "    Groups\n",
      "        group_id=4\n",
      "        name=group_1\n",
      "Sample[2]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "    range_sum=4\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[4]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "    Groups\n",
      "        group_id=2\n",
      "        name=group_3\n",
      "Sample[3]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[1, 7]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=2\n",
      "        name=group_3\n",
      "Sample[4]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[0, 6]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=0\n",
      "        name=group_5\n",
      "Sample[5]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[3, 9]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=0\n",
      "        name=group_5\n",
      "(6 rows)\n"
     ]
    }
   ],
   "source": [
    "join = join.sort_by('Groups.name').display()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample[0]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[2, 8]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=4\n",
      "        name=group_1\n",
      "Sample[1]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "    range_sum=5\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[5]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "    Groups\n",
      "        group_id=4\n",
      "        name=group_1\n",
      "Sample[2]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "    range_sum=4\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[4]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "    Groups\n",
      "        group_id=2\n",
      "        name=group_3\n",
      "Sample[3]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[1, 7]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=2\n",
      "        name=group_3\n",
      "Sample[4]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[0, 6]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=0\n",
      "        name=group_5\n",
      "Sample[5]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[3, 9]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=0\n",
      "        name=group_5\n",
      "Sample[6]\n",
      "    group_id1=0\n",
      "    group_id2=4\n",
      "    range_sum=10\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[2, 8]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=4\n",
      "        name=group_1\n",
      "Sample[7]\n",
      "    group_id1=2\n",
      "    group_id2=4\n",
      "    range_sum=5\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[5]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "    Groups\n",
      "        group_id=4\n",
      "        name=group_1\n",
      "Sample[8]\n",
      "    group_id1=0\n",
      "    group_id2=2\n",
      "    range_sum=4\n",
      "    zero_sum=0\n",
      "    one_sum=1\n",
      "    range=[4]\n",
      "    zero=[0]\n",
      "    one=[1]\n",
      "    Groups\n",
      "        group_id=2\n",
      "        name=group_3\n",
      "Sample[9]\n",
      "    group_id1=2\n",
      "    group_id2=2\n",
      "    range_sum=8\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[1, 7]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=2\n",
      "        name=group_3\n",
      "Sample[10]\n",
      "    group_id1=0\n",
      "    group_id2=0\n",
      "    range_sum=6\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[0, 6]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=0\n",
      "        name=group_5\n",
      "Sample[11]\n",
      "    group_id1=2\n",
      "    group_id2=0\n",
      "    range_sum=12\n",
      "    zero_sum=0\n",
      "    one_sum=2\n",
      "    range=[3, 9]\n",
      "    zero=[0, 0]\n",
      "    one=[1, 1]\n",
      "    Groups\n",
      "        group_id=0\n",
      "        name=group_5\n",
      "(12 rows)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc90b452190>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "join.vstack(join).display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the hierarchy looks kind of like XML or JSON. Check this out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<?xml version=\"1.0\" ?>\n",
      "<root>\n",
      "    <Sample group_id1=\"0\" group_id2=\"4\" one=\"1 1\" one_sum=\"2\" range=\"2 8\" range_sum=\"10\" zero=\"0 0\" zero_sum=\"0\">\n",
      "        <Groups group_id=\"4\" name=\"group_1\"/>\n",
      "    </Sample>\n",
      "    <Sample group_id1=\"2\" group_id2=\"4\" one=\"1\" one_sum=\"1\" range=\"5\" range_sum=\"5\" zero=\"0\" zero_sum=\"0\">\n",
      "        <Groups group_id=\"4\" name=\"group_1\"/>\n",
      "    </Sample>\n",
      "    <Sample group_id1=\"0\" group_id2=\"2\" one=\"1\" one_sum=\"1\" range=\"4\" range_sum=\"4\" zero=\"0\" zero_sum=\"0\">\n",
      "        <Groups group_id=\"2\" name=\"group_3\"/>\n",
      "    </Sample>\n",
      "    <Sample group_id1=\"2\" group_id2=\"2\" one=\"1 1\" one_sum=\"2\" range=\"1 7\" range_sum=\"8\" zero=\"0 0\" zero_sum=\"0\">\n",
      "        <Groups group_id=\"2\" name=\"group_3\"/>\n",
      "    </Sample>\n",
      "    <Sample group_id1=\"0\" group_id2=\"0\" one=\"1 1\" one_sum=\"2\" range=\"0 6\" range_sum=\"6\" zero=\"0 0\" zero_sum=\"0\">\n",
      "        <Groups group_id=\"0\" name=\"group_5\"/>\n",
      "    </Sample>\n",
      "    <Sample group_id1=\"2\" group_id2=\"0\" one=\"1 1\" one_sum=\"2\" range=\"3 9\" range_sum=\"12\" zero=\"0 0\" zero_sum=\"0\">\n",
      "        <Groups group_id=\"0\" name=\"group_5\"/>\n",
      "    </Sample>\n",
      "</root>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print join.toxml()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<?xml version=\"1.0\" ?>\n",
      "<root>\n",
      "    <Sample>\n",
      "        <group_id1>0</group_id1>\n",
      "        <group_id2>4</group_id2>\n",
      "        <range_sum>10</range_sum>\n",
      "        <zero_sum>0</zero_sum>\n",
      "        <one_sum>2</one_sum>\n",
      "        <range>2 8</range>\n",
      "        <zero>0 0</zero>\n",
      "        <one>1 1</one>\n",
      "        <Groups>\n",
      "            <group_id>4</group_id>\n",
      "            <name>group_1</name>\n",
      "        </Groups>\n",
      "    </Sample>\n",
      "    <Sample>\n",
      "        <group_id1>2</group_id1>\n",
      "        <group_id2>4</group_id2>\n",
      "        <range_sum>5</range_sum>\n",
      "        <zero_sum>0</zero_sum>\n",
      "        <one_sum>1</one_sum>\n",
      "        <range>5</range>\n",
      "        <zero>0</zero>\n",
      "        <one>1</one>\n",
      "        <Groups>\n",
      "            <group_id>4</group_id>\n",
      "            <name>group_1</name>\n",
      "        </Groups>\n",
      "    </Sample>\n",
      "    <Sample>\n",
      "        <group_id1>0</group_id1>\n",
      "        <group_id2>2</group_id2>\n",
      "        <range_sum>4</range_sum>\n",
      "        <zero_sum>0</zero_sum>\n",
      "        <one_sum>1</one_sum>\n",
      "        <range>4</range>\n",
      "        <zero>0</zero>\n",
      "        <one>1</one>\n",
      "        <Groups>\n",
      "            <group_id>2</group_id>\n",
      "            <name>group_3</name>\n",
      "        </Groups>\n",
      "    </Sample>\n",
      "    <Sample>\n",
      "        <group_id1>2</group_id1>\n",
      "        <group_id2>2</group_id2>\n",
      "        <range_sum>8</range_sum>\n",
      "        <zero_sum>0</zero_sum>\n",
      "        <one_sum>2</one_sum>\n",
      "        <range>1 7</range>\n",
      "        <zero>0 0</zero>\n",
      "        <one>1 1</one>\n",
      "        <Groups>\n",
      "            <group_id>2</group_id>\n",
      "            <name>group_3</name>\n",
      "        </Groups>\n",
      "    </Sample>\n",
      "    <Sample>\n",
      "        <group_id1>0</group_id1>\n",
      "        <group_id2>0</group_id2>\n",
      "        <range_sum>6</range_sum>\n",
      "        <zero_sum>0</zero_sum>\n",
      "        <one_sum>2</one_sum>\n",
      "        <range>0 6</range>\n",
      "        <zero>0 0</zero>\n",
      "        <one>1 1</one>\n",
      "        <Groups>\n",
      "            <group_id>0</group_id>\n",
      "            <name>group_5</name>\n",
      "        </Groups>\n",
      "    </Sample>\n",
      "    <Sample>\n",
      "        <group_id1>2</group_id1>\n",
      "        <group_id2>0</group_id2>\n",
      "        <range_sum>12</range_sum>\n",
      "        <zero_sum>0</zero_sum>\n",
      "        <one_sum>2</one_sum>\n",
      "        <range>3 9</range>\n",
      "        <zero>0 0</zero>\n",
      "        <one>1 1</one>\n",
      "        <Groups>\n",
      "            <group_id>0</group_id>\n",
      "            <name>group_5</name>\n",
      "        </Groups>\n",
      "    </Sample>\n",
      "</root>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print join.toxml(use_attributes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"Sample\": [\n",
      "        {\n",
      "            \"group_id1\": \"0\",\n",
      "            \"group_id2\": \"4\",\n",
      "            \"range_sum\": \"10\",\n",
      "            \"zero_sum\": \"0\",\n",
      "            \"one_sum\": \"2\",\n",
      "            \"range\": [\n",
      "                2,\n",
      "                8\n",
      "            ],\n",
      "            \"zero\": [\n",
      "                0,\n",
      "                0\n",
      "            ],\n",
      "            \"one\": [\n",
      "                1,\n",
      "                1\n",
      "            ],\n",
      "            \"Groups\": {\n",
      "                \"group_id\": \"4\",\n",
      "                \"name\": \"group_1\"\n",
      "            }\n",
      "        },\n",
      "        {\n",
      "            \"group_id1\": \"2\",\n",
      "            \"group_id2\": \"4\",\n",
      "            \"range_sum\": \"5\",\n",
      "            \"zero_sum\": \"0\",\n",
      "            \"one_sum\": \"1\",\n",
      "            \"range\": [\n",
      "                5\n",
      "            ],\n",
      "            \"zero\": [\n",
      "                0\n",
      "            ],\n",
      "            \"one\": [\n",
      "                1\n",
      "            ],\n",
      "            \"Groups\": {\n",
      "                \"group_id\": \"4\",\n",
      "                \"name\": \"group_1\"\n",
      "            }\n",
      "        },\n",
      "        {\n",
      "            \"group_id1\": \"0\",\n",
      "            \"group_id2\": \"2\",\n",
      "            \"range_sum\": \"4\",\n",
      "            \"zero_sum\": \"0\",\n",
      "            \"one_sum\": \"1\",\n",
      "            \"range\": [\n",
      "                4\n",
      "            ],\n",
      "            \"zero\": [\n",
      "                0\n",
      "            ],\n",
      "            \"one\": [\n",
      "                1\n",
      "            ],\n",
      "            \"Groups\": {\n",
      "                \"group_id\": \"2\",\n",
      "                \"name\": \"group_3\"\n",
      "            }\n",
      "        },\n",
      "        {\n",
      "            \"group_id1\": \"2\",\n",
      "            \"group_id2\": \"2\",\n",
      "            \"range_sum\": \"8\",\n",
      "            \"zero_sum\": \"0\",\n",
      "            \"one_sum\": \"2\",\n",
      "            \"range\": [\n",
      "                1,\n",
      "                7\n",
      "            ],\n",
      "            \"zero\": [\n",
      "                0,\n",
      "                0\n",
      "            ],\n",
      "            \"one\": [\n",
      "                1,\n",
      "                1\n",
      "            ],\n",
      "            \"Groups\": {\n",
      "                \"group_id\": \"2\",\n",
      "                \"name\": \"group_3\"\n",
      "            }\n",
      "        },\n",
      "        {\n",
      "            \"group_id1\": \"0\",\n",
      "            \"group_id2\": \"0\",\n",
      "            \"range_sum\": \"6\",\n",
      "            \"zero_sum\": \"0\",\n",
      "            \"one_sum\": \"2\",\n",
      "            \"range\": [\n",
      "                0,\n",
      "                6\n",
      "            ],\n",
      "            \"zero\": [\n",
      "                0,\n",
      "                0\n",
      "            ],\n",
      "            \"one\": [\n",
      "                1,\n",
      "                1\n",
      "            ],\n",
      "            \"Groups\": {\n",
      "                \"group_id\": \"0\",\n",
      "                \"name\": \"group_5\"\n",
      "            }\n",
      "        },\n",
      "        {\n",
      "            \"group_id1\": \"2\",\n",
      "            \"group_id2\": \"0\",\n",
      "            \"range_sum\": \"12\",\n",
      "            \"zero_sum\": \"0\",\n",
      "            \"one_sum\": \"2\",\n",
      "            \"range\": [\n",
      "                3,\n",
      "                9\n",
      "            ],\n",
      "            \"zero\": [\n",
      "                0,\n",
      "                0\n",
      "            ],\n",
      "            \"one\": [\n",
      "                1,\n",
      "                1\n",
      "            ],\n",
      "            \"Groups\": {\n",
      "                \"group_id\": \"0\",\n",
      "                \"name\": \"group_5\"\n",
      "            }\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print join.tojson()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's do some caching. For this I am relying on numpy.savez and numpy.savez_compressed. I have yet to find a faster way of storing a billion values. Here is the documentation:\n",
    "\n",
    "http://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html\n",
    "\n",
    "I use it in Atom.load and Atom.persist:\n",
    "\n",
    "I splay columns into individual npz files:\n",
    "\n",
    "    lbrenner@josie:~/buckysoap$ ls cache/sample\n",
    "    one.npz  range.npz  zero.npz\n",
    "\n",
    "Each Atom's npz file contains data, mask and neccessary bincounts.\n",
    "\n",
    "     Filemode      Length  Date         Time      File\n",
    "    - ----------  --------  -----------  --------  --------\n",
    "      -rw-------       160   4-Jun-2015  12:23:08  data.npy\n",
    "      -rw-------        90   4-Jun-2015  12:23:08  mask.npy\n",
    "    - ----------  --------  -----------  --------  --------\n",
    "                   250                         2 files\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting cache/sample\n",
      "Running Factory\n",
      "cache/sample = 10\n",
      "range zero one\n",
      "    0    0   1\n",
      "    1    0   1\n",
      "    2    0   1\n",
      "    3    0   1\n",
      "    4    0   1\n",
      "    5    0   1\n",
      "    6    0   1\n",
      "    7    0   1\n",
      "    8    0   1\n",
      "    9    0   1\n",
      "(10 rows)\n",
      "range zero one\n",
      "    0    0   1\n",
      "    1    0   1\n",
      "    2    0   1\n",
      "    3    0   1\n",
      "    4    0   1\n",
      "    5    0   1\n",
      "    6    0   1\n",
      "    7    0   1\n",
      "    8    0   1\n",
      "    9    0   1\n",
      "(10 rows)\n",
      "Running Factory\n",
      "cache/sample = 10\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<buckysoap.element.Element at 0x7fc8e9fc0dd0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def cache_sample():\n",
    "    def factory():\n",
    "        print \"Running Factory\"\n",
    "        return bs.Element(name='Sample')(\n",
    "            range = bs.arange(10),\n",
    "            zero = bs.zeros(10, int),\n",
    "            one = bs.ones(10, int))\n",
    "    return bs.Element(\n",
    "        name='sample',\n",
    "        cnames='range,zero,one',\n",
    "        cachedir='cache/sample',\n",
    "        factory=factory)\n",
    "\n",
    "#We are just deleting directory for the sake of demonstration\n",
    "import os\n",
    "if os.path.exists('cache/sample'):\n",
    "    print \"Deleting cache/sample\"\n",
    "    import shutil\n",
    "    shutil.rmtree('cache/sample')\n",
    "cache_sample().display()\n",
    "cache_sample().display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to map to and from pandas use: element.to_pandas() Element.from_pandas(). I will provide examples but my current instance does not have it running. I would recommend using Enthought or Continuum."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}