06_AdvancedTutorials/Managing_data_with_datreant.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Managing data with datreant\n",
    "\n",
    "`datreant` is a Python package for managing data across many different directories.\n",
<<<<<<< HEAD
=======
    "\n",
>>>>>>> 0c5cef6... started datreant notebook
    "We'll jump straight in to a messy directory of simulation results,\n",
    "these are moderately organised, but are a little scattered:\n",
    "\n",
    "```\n",
    "datreant_data\n",
    " +-- other_dir\n",
<<<<<<< HEAD
    " |   +-- sim_T300\n",
    " |   +-- sim_T400\n",
    " |   +-- sim_T500\n",
    " |   +-- sim_T600\n",
    " +-- sim_T100\n",
    " +-- sim_T200\n",
    " +-- sim_T300\n",
    " +-- sim_T400\n",
    "```\n",
    "\n",
    "Using `datreant` we can quickly find all directories we have previously tagged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from asciitree import LeftAligned\n",
    "from collections import OrderedDict as OD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "contents = OD(\n",
    "   datreant_data=OD(\n",
    "       other_dir=OD(\n",
    "           sim_T300={},\n",
    "           sim_T400={},\n",
    "           sim_T500={},\n",
    "           sim_T600={},\n",
    "       ),\n",
    "       sim_T100={},\n",
    "       sim_T200={},\n",
    "       sim_T300={},\n",
    "       sim_T400={},\n",
    "   )\n",
    ")"
=======
    " |   +-- sim_T298\n",
    " |   |   +-- conf.gro\n",
    " |   +-- sim_T303\n",
    " |   |   +-- conf.gro\n",
    " |   +-- sim_T323\n",
    " |   |   +-- conf.gro\n",
    " |   +-- sim_T333\n",
    " |       +-- conf.gro\n",
    " +-- sim_T343\n",
    " |   +-- conf.gro\n",
    " +-- sim_T348\n",
    " |   +-- conf.gro\n",
    " +-- sim_T355\n",
    "     +-- conf.gro\n",
    "```\n",
    "\n",
    "Using `datreant` we can quickly work with the data that is distributed across these different directories.\n",
    "\n",
    "First we can use `datreant.discover` to find all data directories that we have previously identified."
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "datreant_data\n",
      " +-- other_dir\n",
      " |   +-- sim_T300\n",
      " |   +-- sim_T400\n",
      " |   +-- sim_T500\n",
      " |   +-- sim_T600\n",
      " +-- sim_T100\n",
      " +-- sim_T200\n",
      " +-- sim_T300\n",
      " +-- sim_T400\n"
     ]
    }
   ],
   "source": [
    "print(LeftAligned()(contents))"
=======
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Bundle(['sim_T303', 'sim_T333', 'sim_T298', 'sim_T343', 'sim_T355', 'sim_T348', 'sim_T323'])>"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import datreant as dtr\n",
    "import os\n",
    "\n",
    "# find all datreant objects in this directory\n",
    "treants = dtr.discover('datreant_data/')\n",
    "\n",
    "treants"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 24,
=======
   "execution_count": 109,
>>>>>>> 0c5cef6... started datreant notebook
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
<<<<<<< HEAD
       "<Bundle(['foo', 'sim_T100', 'sim_T200', 'sim_T300', 'sim_T500', 'sim_T600', 'sim_T300', 'sim_T400', 'sim_T400'])>"
      ]
     },
     "execution_count": 24,
=======
       "<AggCategories({'temperature': [303, 333, 298, 343, 355, 348, 323]})>"
      ]
     },
     "execution_count": 109,
>>>>>>> 0c5cef6... started datreant notebook
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
<<<<<<< HEAD
    "import datreant as dtr\n",
    "import os\n",
    "\n",
    "treants = dtr.discover('.')  # find all datreant objects in this directory\n",
    "\n",
    "treants"
=======
    "treants.categories"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
<<<<<<< HEAD
    "From these results, we can then query"
=======
    "This returns a `Bundle`, which represent a collection of different data directories.\n",
    "\n",
    "From these results, we can then query information related to them, which are stored in the `.tags` and `.categories` attributes:"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 7,
=======
   "execution_count": 110,
>>>>>>> 0c5cef6... started datreant notebook
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
<<<<<<< HEAD
       "{'Simulation'}"
      ]
     },
     "execution_count": 7,
=======
       "<AggCategories({'temperature': [303, 333, 298, 343, 355, 348, 323]})>"
      ]
     },
     "execution_count": 110,
>>>>>>> 0c5cef6... started datreant notebook
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
<<<<<<< HEAD
    "treants.tags.any"
=======
    "treants.categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Individual members of a `Bundle` are `Treant` objects, which represent a particular data directory and store metadata about that data"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 8,
=======
   "execution_count": 111,
>>>>>>> 0c5cef6... started datreant notebook
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
<<<<<<< HEAD
       "<AggCategories({'temperature': [100, 200, 300, 500, 600, 300, 400, 400]})>"
      ]
     },
     "execution_count": 8,
=======
       "<Tags(['Simulation'])>"
      ]
     },
     "execution_count": 111,
>>>>>>> 0c5cef6... started datreant notebook
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
<<<<<<< HEAD
    "treants.categories"
=======
    "treant = treants[0]\n",
    "\n",
    "treant.tags"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 9,
=======
   "execution_count": 112,
>>>>>>> 0c5cef6... started datreant notebook
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
<<<<<<< HEAD
       "<Bundle(['sim_T300', 'sim_T300'])>"
      ]
     },
     "execution_count": 9,
=======
       "<Categories({'temperature': 303})>"
      ]
     },
     "execution_count": 112,
>>>>>>> 0c5cef6... started datreant notebook
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
<<<<<<< HEAD
    "treants.get(temperature=300)"
=======
    "treant.categories"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/richardgowers/code/WorkshopHackathon2018/06_AdvancedTutorials/datreant_data/sim_T200/\n"
     ]
    }
   ],
   "source": [
    "for t in treants.get(temperature=200):\n",
    "    print(t.abspath)"
=======
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/richardgowers/code/WorkshopHackathon2018/06_AdvancedTutorials/datreant_data/sim_T303/'"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treant.abspath"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
<<<<<<< HEAD
    "### datreant objects\n",
    "\n",
    "explain Treant, Bundle, Leaf?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constructing your own datreant database\n",
    "\n",
    "Now we've seen how this can work onc"
=======
    "If we wanted to run a particular analysis on each of these simulation results, we could loop over the treants like so:"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtr.Treant?"
=======
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "import MDAnalysis as mda\n",
    "\n",
    "temps = []\n",
    "results = []\n",
    "\n",
    "# loop over all Treants\n",
    "for t in treants:\n",
    "    # grab some metadata from this Treant\n",
    "    T = t.categories['temperature']\n",
    "    \n",
    "    # create a MDAnalysis Universe by looking inside this Treant\n",
    "    # for a particular data file\n",
    "    u = mda.Universe(t['conf.gro'].abspath)\n",
    "    \n",
    "    density = u.atoms.masses.sum() / u.trajectory.ts.volume\n",
    "    \n",
    "    # permanently store this result for quick searching\n",
    "    t.categories['density'] = density\n",
    "    \n",
    "    temps.append(T)\n",
    "    results.append(density)"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 20,
=======
   "execution_count": 115,
>>>>>>> 0c5cef6... started datreant notebook
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
<<<<<<< HEAD
       "<Treant: 'foo'>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtr.Treant('foo')"
=======
       "Text(0.5,0,'Temperature')"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFj9JREFUeJzt3X+0pVV93/H3x5lBxhqDwqTyM0Mioi5r+XGdmJgYo4Fh2QpmkSox/sAuQhsyy5W2mRay7FKJWW1Da1djSBQiBmkSSAji0JCMYBKrqT/mTpiAQKZMUMPMaL2CE0OcAIPf/nGeC4fDuXefgTn33nPv+7XWWXOe/eznnL25h/u5z36es3eqCkmS5vOMxW6AJGnpMywkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJalq92A04VI466qhav379YjdDkibK9u3bv1FV61r1lk1YrF+/nunp6cVuhiRNlCRfGaXeWIehkpyVZGeSXUkunqPOG5PcleTOJL/TV/72JPd0j7ePs52SpPmN7cwiySrgcuAMYDewLcmWqrqrr85JwCXAK6vqm0m+pyt/HvBuYAooYHt37DfH1V5J0tzGeWaxAdhVVfdW1cPAtcA5A3V+Brh8NgSq6utd+Ubglqp6oNt3C3DWGNsqSZrHOMPiWOC+vu3dXVm/FwIvTPLnST6X5KyDOFaStEDGeYE7Q8oGF89YDZwEvBo4Dvh0kpeOeCxJLgQuBDjhhBOeTlslSfMY55nFbuD4vu3jgL1D6ny8qh6pqi8BO+mFxyjHUlVXVNVUVU2tW9e880uS9BSNMyy2ASclOTHJYcB5wJaBOjcCPwaQ5Ch6w1L3AluBM5M8N8lzgTO7MknSIhjbMFRVHUiyid4v+VXAVVV1Z5JLgemq2sLjoXAX8CiwuaruB0jyS/QCB+DSqnpgXG2VJM0vy2UN7qmpqfJLeZJ0cJJsr6qpVj3nhpIkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaVi92Axbbjbft4bKtO9m7bz/HHLGWzRtP5g2nHrvYzZKkJWVFh8WNt+3hkhvuYP8jjwKwZ99+LrnhDgADQ5L6rOhhqMu27nwsKGbtf+RRLtu6c5FaJElL04oOi7379h9UuSStVCs6LI45Yu1BlUvSSrWiw2LzxpNZu2bVE8rWrlnF5o0nL1KLJGlpWtEXuGcvYns3lCTNb0WHBfQCw3CQpPmt6GEoSdJoxhoWSc5KsjPJriQXD9l/fpKZJDu6xwV9+34lyZ1J7k7yq0kyzrZKkuY2tmGoJKuAy4EzgN3AtiRbququgarXVdWmgWN/CHgl8LKu6DPAjwJ/Nq72SpLmNs4ziw3Arqq6t6oeBq4Fzhnx2AIOBw4DngmsAf7fWFopSWoaZ1gcC9zXt727Kxt0bpLbk1yf5HiAqvos8KfAV7vH1qq6e4xtlSTNY5xhMewaQw1s3wSsr6qXAbcCVwMkeQHwYuA4egHzmiSvetIbJBcmmU4yPTMzc0gbL0l63DjDYjdwfN/2ccDe/gpVdX9VPdRtXgmc3j3/CeBzVfVgVT0I/BHwisE3qKorqmqqqqbWrVt3yDsgSeoZZ1hsA05KcmKSw4DzgC39FZIc3bd5NjA71PQ3wI8mWZ1kDb2L2w5DSdIiGdvdUFV1IMkmYCuwCriqqu5McikwXVVbgHcmORs4ADwAnN8dfj3wGuAOekNXf1xVN42rrZKk+aVq8DLCZJqamqrp6enFboYkTZQk26tqqlXPb3BLkppW/NxQC8XlWyVNMsNiAbh8q6RJ5zDUAnD5VkmTzrBYAC7fKmnSGRYLwOVbJU06w2IBuHyrpEnnBe4F4PKtkiadYbFAXL5V0iRzGEqS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTc0NJA1wCV3oyw0Lq4xK40nAOQ0l9XAJXGs6wkPq4BK40nGEh9XEJXGk4w0Lq4xK40nBe4Jb6uASuNJxhIQ1wCVzpyRyGkiQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTSOFRZI/SPLPkhgukrQCjfrL/zeANwP3JPnPSV40xjZJkpaYkcKiqm6tqp8GTgO+DNyS5P8keUeSNXMdl+SsJDuT7Epy8ZD95yeZSbKje1zQt++EJJ9IcneSu5KsP9jOSZIOjZGn+0hyJPAW4K3AbcBvAz8MvB149ZD6q4DLgTOA3cC2JFuq6q6BqtdV1aYhb/lR4Jer6pYkzwa+M2pbJUmH1khhkeQG4EXANcDrq+qr3a7rkkzPcdgGYFdV3du9xrXAOcBgWAx7v5cAq6vqFoCqenCUdkqam8vF6ukY9ZrFb1bVS6rqP80GRZJnAlTV1BzHHAvc17e9uysbdG6S25Ncn+T4ruyFwL4kNyS5Lcll3ZmKpKdgdrnYPfv2Uzy+XOyNt+1Z7KZpQowaFu8bUvbZxjEZUlYD2zcB66vqZcCtwNVd+WrgR4BfAF4OfB9w/pPeILkwyXSS6ZmZmUZzpJXL5WL1dM0bFkmen+R0YG2SU5Oc1j1eDTyr8dq7geP7to8D9vZXqKr7q+qhbvNK4PS+Y2+rqnur6gBwI72L6wwcf0VVTVXV1Lp16xrNkVYul4vV09W6ZrGR3l/0xwHv7yv/O+AXG8duA05KciKwBziP3u23j0lydN/1j7OBu/uOfW6SdVU1A7wGmOvaiKSGY45Yy54hweBysRrVvGFRVVcDVyc5t6r+4GBeuKoOJNkEbAVWAVdV1Z1JLgWmq2oL8M4kZwMHgAfohpqq6tEkvwB8MkmA7fTOPCQ9BZs3nswlN9zxhKEol4vVwUjV4GWEvp3JW6rqfyb5dzz5egNV9f4hhy2Kqampmp725EOai3dDaZgk2+e5UekxrWGof9T9++yn3yRJi8nlYvV0tIahPtT9+96FaY4kaSkadSLBX0nynCRrknwyyTeSvGXcjZMkLQ2jfs/izKr6FvDP6d3W+kJg89haJUlaUkYNi9nJAl8H/G5VPTCm9kiSlqBRJxK8KclfAfuBi5KsA/5hfM2SJC0lo05RfjHwg8BUVT0C/D29SQElSSvAyFOUAy8G1ifpP+ajh7g9kqQlaNQpyq8Bvh/YAcx+BbQwLCRpRRj1zGIKeEnN93VvSdKyNerdUF8Enj/OhkiSlq5RzyyOAu5K8gVgdkpxqurssbRKkrSkjBoW7xlnIyRJS9tIYVFVn0ryvcBJVXVrkmfRm3ZckhaNM+kunFHnhvoZ4HrgQ13RsfRWr5OkReG64gtr1AvcPwe8EvgWQFXdA3zPuBolSS2uK76wRg2Lh6rq4dmN7ot53kYradG4rvjCGjUsPpXkF4G1Sc4Afh+4aXzNkqT5zbV+uOuKj8eoYXExMAPcAfwr4GbgXeNqlCS1bN54MmvXPPE+G9cVH59R74b6TpIbgRurambMbZKkptm7nrwbamHMGxZJArwb2ASkK3oU+EBVXboA7ZOkObmu+MJpDUP9PL27oF5eVUdW1fOAHwBemeTfjL11kqQloRUWbwN+qqq+NFtQVfcCb+n2SZJWgFZYrKmqbwwWdtct1gypL0lahlph8fBT3CdJWkZad0P90yTfGlIe4PAxtEeStATNGxZV5WSBkqSRv5QnSVrBDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVLTWMMiyVlJdibZleTiIfvPTzKTZEf3uGBg/3OS7Enya+NspyRpfiMtq/pUJFkFXA6cAewGtiXZUlV3DVS9rqo2zfEyvwR8alxtlCSNZpxnFhuAXVV1b1U9DFwLnDPqwUlOB/4x8IkxtU+SNKJxhsWxwH1927u7skHnJrk9yfVJjgdI8gzgvwGbx9g+SdKIxhkWGVJWA9s3Aeur6mXArcDVXflFwM1VdR/zSHJhkukk0zMzM0+7wZKk4cZ2zYLemcTxfdvHAXv7K1TV/X2bVwL/pXv+g8CPJLkIeDZwWJIHq+rigeOvAK4AmJqaGgwiSdIhMs6w2AaclOREYA9wHvDm/gpJjq6qr3abZwN3A1TVT/fVOR+YGgwKSdLCGVtYVNWBJJuArcAq4KqqujPJpcB0VW0B3pnkbOAA8ABw/rjaI0l66lK1PEZvpqamanp6erGbIUkTJcn2qppq1fMb3JKkJsNCktRkWEiSmgwLSVKTYSFJahrn9ywkadm48bY9XLZ1J3v37eeYI9ayeePJvOHUYTMYLU+GhSQ13HjbHi654Q72P/IoAHv27eeSG+4AWDGB4TCUJDVctnXnY0Exa/8jj3LZ1p2L1KKFZ1hIUsPeffsPqnw5MiwkqeGYI9YeVPlyZFhIUsPmjSezds2qJ5StXbOKzRtPXqQWLTwvcEtSw+xFbO+GkiTN6w2nHruiwmGQw1CSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKanBtKkibUQi71alhI0gRa6KVeHYaSpAm00Eu9GhaSNIEWeqlXw0KSJtBCL/VqWEjSBFropV69wC1JE2ihl3o1LCRpQi3kUq8OQ0mSmgwLSVKTYSFJajIsJElNhoUkqWmsYZHkrCQ7k+xKcvGQ/ecnmUmyo3tc0JWfkuSzSe5McnuSN42znZKk+Y3t1tkkq4DLgTOA3cC2JFuq6q6BqtdV1aaBsm8Db6uqe5IcA2xPsrWq9o2rvZKkuY3zzGIDsKuq7q2qh4FrgXNGObCq/m9V3dM93wt8HVg3tpZKkuY1zrA4Frivb3t3Vzbo3G6o6fokxw/uTLIBOAz46/E0U5LUMs6wyJCyGti+CVhfVS8DbgWufsILJEcD1wDvqKrvPOkNkguTTCeZnpmZOUTNliQNGmdY7Ab6zxSOA/b2V6iq+6vqoW7zSuD02X1JngP8IfCuqvrcsDeoqiuqaqqqptatc5RKksZlnGGxDTgpyYlJDgPOA7b0V+jOHGadDdzdlR8GfAz4aFX9/hjbKEkawdjuhqqqA0k2AVuBVcBVVXVnkkuB6araArwzydnAAeAB4Pzu8DcCrwKOTDJbdn5V7RhXeyVJc0vV4GWEyTQ1NVXT09OL3QxJmihJtlfVVKue3+CWJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2pqsVuwyGRZAb4ygK81VHANxbgfRbacuzXcuwTLM9+2afF871Vta5VadmExUJJMl1VU4vdjkNtOfZrOfYJlme/7NPS5zCUJKnJsJAkNRkWB++KxW7AmCzHfi3HPsHy7Jd9WuK8ZiFJavLMQpLUZFgMSHJ4ki8k+cskdyZ5b1d+YpLPJ7knyXVJDuvKn9lt7+r2r1/M9g8zT582de2uJEf11U+SX+323Z7ktMVr/XDz9Om3k+xM8sUkVyVZ05Uv+T7BvP36cFd2e5Lrkzy7K5/Yz1/f/g8kebBve8n3Ceb9Wf1Wki8l2dE9TunKJ+IzOKeq8tH3AAI8u3u+Bvg88Arg94DzuvIPAj/bPb8I+GD3/DzgusXuw0H06VRgPfBl4Ki++q8D/qg77hXA5xe7DwfRp9d1+wL8bt/Pacn3qdGv5/TVeT9w8aR//rrtKeAa4MG++ku+T42f1W8BPzmk/kR8Bud6eGYxoHpm/8pZ0z0KeA1wfVd+NfCG7vk53Tbd/tcmyQI1dyRz9amqbquqLw855Bzgo91xnwOOSHL0AjV3JPP06eZuXwFfAI7r6iz5PsG8/foW9P46BdbS+0zCBH/+kqwCLgP+/cAhS75PMO/virlMxGdwLobFEElWJdkBfB24BfhrYF9VHeiq7AaO7Z4fC9wH0O3/W+DIhW1x22Cfqurz81R/rE+d/v4uGfP1qRt+eivwx13RRPQJ5u5Xko8AXwNeBHygqz7Jn79NwJaq+upA9YnoE8z7Gfzlbqjpvyd5Zlc2MZ/BYQyLIarq0ao6hd5fpRuAFw+r1v077C+eJXeL2WCfkrx0nurLoU+/Dvzvqvp0tz0RfYK5+1VV7wCOAe4G3tRVn4h+DenTq4B/weOh128i+gRz/qwuoRfoLweeB/yHrvrE9GsYw2IeVbUP+DN644tHJFnd7ToO2Ns93w0cD9Dt/27ggYVt6ej6+nTWPNUe61Onv79LzmCfkrwbWAf8275qE9UnGP6zqqpHgeuAc7uiSf38/RjwAmBXki8Dz0qyq6s2UX2CJ/6squqr3VDTQ8BH6P3BCRP4GexnWAxIsi7JEd3ztcCP0/tL7k+Bn+yqvR34ePd8S7dNt/9PuvHyJWOOPv3VPIdsAd7W3b3xCuBvhwwVLKq5+pTkAmAj8FNV9Z2+Q5Z8n2DOfu1M8oKuLMDrefznN6mfv+1V9fyqWl9V64FvV9ULukOWfJ9g3s/g0V1Z6F3b/GJ3yER8Bueyul1lxTkauLq7+PYM4Peq6n8luQu4Nsn7gNuAD3f1Pwxc0/1V9AC9uzeWmrn69E56FxefD9ye5OaqugC4md6dG7uAbwPvWKR2z2euPh2gN/vwZ7trojdU1aVMRp9gSL+APwQ+neQ59IYy/hL42a7+xH7+5qk/CX2CuT+Df5JkHb2f1Q7gX3f1J+UzOJTf4JYkNTkMJUlqMiwkSU2GhSSpybCQJDUZFpKkJm+d1bKX5Ejgk93m84FHgZlue0NVPbwoDZtHkn8J3FxVX1vstkjgrbNaYZK8h94Mp/91CbRlVfeN7GH7PgNsqqodB/F6q/vmL5MOKYehtKIleXt6axLsSPLrSZ6RZHWSfUkuS/IXSbYm+YEkn0pyb5LXdcdekORj3f6dSd414uu+L8kX6M0l9N4k29Jbf+OD3bd73wScAlzXHX9Ykt193xZ+RZJbu+fvS/KhJLcAH+ne4/3de9/efaNdetoMC61Y6U369hPAD3WTwa3m8W8Lfzfwiao6DXgYeA/wWnqT313a9zIbumNOA96c5JQRXvcvqmpDVX0W+B9V9XLgn3T7zqqq6+h98/dNVXXKCMNkpwKvr6q3AhcCX6+qDfQmsvu5JCc8lf8+Uj+vWWgl+3F6v1Cnu6lB1vL4FNL7q+qW7vkd9ObxOZDkDnoLRs3aWlXfBEhyI/DD9P6/mut1HwY+1nf8a5NsBg4HjgK201sg52B8vKr+oXt+JvDiJP3hdBLwNwf5mtITGBZayQJcVVX/8QmFvZlO+/+a/w7wUN/z/v9vBi/6VeN1989OipfkWcCvAadV1Z5u3rHD52jrAR4fCRis8/cDfbqoqj6JdAg5DKWV7FbgjenWH09y5FMYsjkzyRHdL/5zgD8/iNddSy98vpHku3h82nGAvwO+q2/7y8Dp3fP+eoO2Ahd1wUSSk7sZUaWnxTMLrVhVdUeS9wK3JnkG8Ai9GUIPZo2BzwC/A3w/cM3s3UujvG5V3Z/kanpTWH+F3hrOsz4C/GaS/fSui7wHuDLJ1+gtFzuXDwEnADu6IbCv0wsx6Wnx1lnpKeruNHppVf38YrdFGjeHoSRJTZ5ZSJKaPLOQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJavr/ApSegFNjBOQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.scatter(temps, results)\n",
    "\n",
    "ax.set_ylabel('Density')\n",
    "ax.set_xlabel('Temperature')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating datreant objects\n",
    "\n",
    "datreant works by creating `Treant` objects in directories."
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 21,
=======
   "execution_count": 116,
>>>>>>> 0c5cef6... started datreant notebook
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
<<<<<<< HEAD
       "True"
      ]
     },
     "execution_count": 21,
=======
       "<Treant: 'foo'>"
      ]
     },
     "execution_count": 116,
>>>>>>> 0c5cef6... started datreant notebook
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
<<<<<<< HEAD
    "dtr.Treant('foo').exists"
=======
    "t = dtr.Treant('foo')\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then assign tags and categories to this object.\n",
    "This data is stored permanently in the directory, allowing us to later search and filter by this metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "t.tags.add('awesome')\n",
    "\n",
    "t.categories['color'] = 'green'\n",
    "t.categories['shape'] = 'round'"
>>>>>>> 0c5cef6... started datreant notebook
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How does it actually work?\n",
    "\n",
    "Datreant works by storing small files to keep track of directories that you have tagged.\n",
    "These are kept in a \"hidden\" directory called `.datreant`."
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 3,
=======
   "execution_count": 120,
>>>>>>> 0c5cef6... started datreant notebook
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
<<<<<<< HEAD
      "{\"temperature\": 100}\n",
=======
      "{\"temperature\": 298, \"density\": 0.5844066}\n",
>>>>>>> 0c5cef6... started datreant notebook
      "[\"Simulation\"]\n"
     ]
    }
   ],
   "source": [
<<<<<<< HEAD
    "with open('./sim_T100/.datreant/categories.json', 'r') as inf:\n",
    "    print(inf.read())\n",
    "with open('./sim_T100/.datreant/tags.json', 'r') as inf:\n",
=======
    "with open('./datreant_data/sim_T298/.datreant/categories.json', 'r') as inf:\n",
    "    print(inf.read())\n",
    "with open('./datreant_data/sim_T298/.datreant/tags.json', 'r') as inf:\n",
>>>>>>> 0c5cef6... started datreant notebook
    "    print(inf.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This simple implementation means that you can freely rearrange your results directories (eg move them from your HPC cluster to long term storage) without destroying the `Treant` objects you have created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Future work\n",
    "\n",
    "datreant is being actively developed and a command line interface (cli) is currently being added.\n",
    "\n",
    "For more information on this, talk to David or Richard and check out the repository during the hackathon on Tuesday!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}