tutorials/emdfile_package_walkthrough.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c319b309",
   "metadata": {},
   "source": [
    "# `emdfile` package walkthrough\n",
    "\n",
    "Here we demo the user-facing functionality in `emdfile`.\n",
    "\n",
    "\n",
    "`emdfile` is a Python package providing an interface between Python runtimes and EMD 1.0 (HDF5) files.  It does so by defining\n",
    "1. write and read functions, and \n",
    "2. a set of classes which hold data as well as tree-like relationships between data elements, intended to simplify logical grouping and reading/writing of related sets of data.\n",
    "\n",
    "\n",
    "The focus here is on `emdfile` and thus on the Python half of the interface being described, however some discussion of the EMD 1.0 file format is also included, beginning with description of it's basic elements in [Section 2](#emd).  The complete EMD 1.0 file specification is [available here](https://emdatasets.com/format/).\n",
    "\n",
    "\n",
    "\n",
    "## See also\n",
    "\n",
    "\n",
    "- [emdfile_intro_example.ipynb](./emdfile_intro_example.ipynb) for a motivating example\n",
    "- [test_custom_class.py](./test_custom_class.py) and [sample_custom_class_module/](./sample_custom_class_module) for an example of a Python module or package built on top of `emdfile`, using it to handle read and write functionality through class inheritance\n",
    "- [the EMD file specification](https://emdatasets.com/format/)\n",
    "\n",
    "\n",
    "## Authorship\n",
    "\n",
    "Benjamin H. Savitzky\n",
    "\n",
    "April, 2023\n",
    "\n",
    "<hr style=\"border:2px solid gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edcf0f4a",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "- 1. [Simple Python objects](#simplePython)\n",
    "    - 1.1 [numpy arrays](#nparrays)\n",
    "    - 1.2 [Python dictionaries](#dicts)\n",
    "    - 1.3 [lists](#lists)\n",
    "\n",
    "- 2. [EMD Trees](#emdtrees)\n",
    "    - 2.1 [Trees](#trees)\n",
    "    - 2.2 [Writing and reading EMD trees](#writingtrees)\n",
    "        - 2.2.1. [Write/read a tree](#write_one_tree)\n",
    "        - 2.2.2. [Read a subset of a tree](#read_a_tree_subset)\n",
    "            - 2.2.2.1 [Read a single node](#read_a_single_node)\n",
    "            - 2.2.2.2 [Read a branch](#read_a_branch)\n",
    "            - 2.2.2.3 [Read a branch, excluding its source node](#read_a_branch_exclude_source_node)\n",
    "        - 2.2.3. [Write a subset of a tree](#write_a_tree_subset)\n",
    "            - 2.2.3.1 [Write a single node](#write_a_single_node)\n",
    "            - 2.2.3.2 [Write a branch](#write_a_branch)\n",
    "            - 2.2.3.3 [Write a branch, excluding its source node](#write_a_branch_exclude_source_node)            \n",
    "        - 2.2.4. [Multiple trees](#write_and_read_multiple_trees)\n",
    "        - 2.2.5. [Unrooted nodes](#unrooted_nodes)\n",
    "\n",
    "- 3. [EMD Nodes](#emdnodes)\n",
    "    - 3.1 [Node](#node)\n",
    "    - 3.2 [Array](#array)\n",
    "        - 3.2.1 [single array](#singlearray)\n",
    "        - 3.2.2 [array stacks](#arraystacks)\n",
    "    - 3.3 [PointList](#pointlist)\n",
    "    - 3.4 [PointListArray](#pointlistarray)\n",
    "    - 3.5 [Root](#rootnode)\n",
    "    - 3.6 [Custom](#customnode)\n",
    "\n",
    "- 4. [EMD Metadata](#emdmetadata)\n",
    "    - 4.1 [The Metadata class](#metadataclass)\n",
    "    - 4.2 [The .metadata property](#metadataproperty)\n",
    "    - 4.3 [Root metadata](#rootmetadata)\n",
    "\n",
    "- 5. [More trees, branches, cutting, grafting, and appending](#moretrees)\n",
    "    - 5.1 [An example](#buildingtrees)\n",
    "    - 5.2 [Cut and graft branches](#cutgraft)\n",
    "        - 5.2.1 [Cut/graft root metadata options](#cutgraft_root_metadata)\n",
    "            - 5.2.1.1 [Cutting](#cut_rootmetadata)\n",
    "            - 5.2.1.2 [Grafting](#graft_rootmetadata)\n",
    "    - 5.3 [Overwrite mode](#overwrite)\n",
    "    - 5.4 [Appending to EMD files](#append)\n",
    "        - 5.4.1 [Append a new tree](#appendtree)\n",
    "        - 5.4.2 [Append to an existing tree](#appendbranch)\n",
    "        - 5.4.3 [Merge a runtime tree and an EMD tree](#append_diffmerge)\n",
    "        - 5.4.4 [Append conflict resolution and 'appendover' mode](#appendover)\n",
    "        - 5.4.5 [Appending and root metadata](#appendrootmetadata)\n",
    "    \n",
    "\n",
    "\n",
    "<hr style=\"border:2px solid gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8178d5e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Imports\n",
    "\n",
    "import emdfile as emd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "78ef5ef4",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Filepath utilities\n",
    "\n",
    "# update to a valid location on your filesystem!\n",
    "filepath = \"/Users/Ben/Desktop/test.h5\"\n",
    "\n",
    "\n",
    "from os.path import exists\n",
    "from os import remove\n",
    "def clean():\n",
    "    if exists(filepath):\n",
    "        remove(filepath)\n",
    "clean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc186360",
   "metadata": {},
   "source": [
    "<a id='simplePython'></a>\n",
    "\n",
    "\n",
    "# 1. Writing simple Python objects\n",
    "\n",
    "Write \n",
    "- a numpy array,\n",
    "- a python dictionary, or \n",
    "- a list of arrays/dictionaries\n",
    "\n",
    "into an HDF5 file.  Then read them.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed0267c",
   "metadata": {},
   "source": [
    "<a id='nparrays'></a>\n",
    "\n",
    "## 1.1 numpy arrays\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fd4b8ab4",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0c7721ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  2,  3],\n",
       "       [ 4,  5,  6,  7],\n",
       "       [ 8,  9, 10, 11]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a numpy array\n",
    "\n",
    "ar = np.arange(12).reshape((3,4))\n",
    "ar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1b901dc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save it\n",
    "\n",
    "emd.save(filepath, ar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e00a2e1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read it\n",
    "\n",
    "loaded_data = emd.read(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a1b6e974",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array( A 2-dimensional array of shape (3, 4) called 'np.array',\n",
       "       with dimensions:\n",
       "\n",
       "       dim0 = [0,1,...] pixels\n",
       "       dim1 = [0,1,...] pixels\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The result is an emdfile Array class instance\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a882270c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  2,  3],\n",
       "       [ 4,  5,  6,  7],\n",
       "       [ 8,  9, 10, 11]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The original numpy array is stored in the `.data` attribute\n",
    "\n",
    "loaded_data.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9091c03c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Confirm the save and loaded arrays are identical\n",
    "\n",
    "assert(np.array_equal( ar , loaded_data.data ))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47dd2548",
   "metadata": {},
   "source": [
    "<a id='dicts'></a>\n",
    "\n",
    "## 1.2 Python dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "061fe6b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6d099624",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'hey': 1, 'diddlediddle': 2, 'thecat': 3, 'andthe': 4, 'fiddle': 5}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a dictionary\n",
    "\n",
    "very_important_knowledge = {\n",
    "    'hey' : 1,\n",
    "    'diddlediddle' : 2,\n",
    "    'thecat' : 3,\n",
    "    'andthe' : 4,\n",
    "    'fiddle' : 5\n",
    "}\n",
    "\n",
    "very_important_knowledge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0a7e3682",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save it\n",
    "\n",
    "emd.save(filepath, very_important_knowledge)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0c6f53e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load it\n",
    "\n",
    "loaded_data = emd.read(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "65df3a85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Metadata( A Metadata instance called 'dictionary', containing the following fields:\n",
       "\n",
       "          andthe:         4\n",
       "          diddlediddle:   2\n",
       "          fiddle:         5\n",
       "          hey:            1\n",
       "          thecat:         3\n",
       ")"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The loaded data is an emdfile Metadata class instance\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c477ab34",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can retrieve and assign values from/to it like a normal Python dictionary\n",
    "\n",
    "loaded_data['andthe']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "262e7bc5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Metadata( A Metadata instance called 'dictionary', containing the following fields:\n",
       "\n",
       "          andthe:         4\n",
       "          diddlediddle:   2\n",
       "          fiddle:         5\n",
       "          hey:            1\n",
       "          thecat:         3\n",
       "          thecow:         1000000.0\n",
       ")"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_data['thecow'] = 1e6\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "55382973",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['andthe', 'diddlediddle', 'fiddle', 'hey', 'thecat', 'thecow'])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_data.keys"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3384ae96",
   "metadata": {},
   "source": [
    "<a id='lists'></a>\n",
    "\n",
    "## 1.3 `list`s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2fcac94b",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5d3b9995",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make some data: 2 arrays, and one dictionary\n",
    "\n",
    "ar1 = np.zeros((4,5))\n",
    "ar2 = np.eye(3)\n",
    "dic = {'cow':'moo','tuple':(1,2,3),'array':np.arange(9).reshape((3,3))}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c4767b04",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save it\n",
    "\n",
    "emd.save(filepath, [ar1,ar2,dic])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "9313b630",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load it\n",
    "\n",
    "loaded_data = emd.read(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0ccbee36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Root( A Node called 'root', containing the following top-level objects in its tree:\n",
       "\n",
       "          np.array_0               \t (Array)\n",
       "          np.array_1               \t (Array)\n",
       ")"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This time, since we saved multiple pieces of data, they were all saved together in a single EMD tree.\n",
    "# More on EMD trees in the next section - for now, that means that when we load the data we get an\n",
    "# emdfile Root class instance which contains some data\n",
    "\n",
    "# The printout indicates that the Root contains two arrays, but doesn't say anything about our dictionary...\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c10ec11c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Array( A 2-dimensional array of shape (4, 5) called 'np.array_0',\n",
      "       with dimensions:\n",
      "\n",
      "       dim0 = [0,1,...] pixels\n",
      "       dim1 = [0,1,...] pixels\n",
      ")\n",
      "Array( A 2-dimensional array of shape (3, 3) called 'np.array_1',\n",
      "       with dimensions:\n",
      "\n",
      "       dim0 = [0,1,...] pixels\n",
      "       dim1 = [0,1,...] pixels\n",
      ")\n",
      "[[0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]]\n",
      "[[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]]\n"
     ]
    }
   ],
   "source": [
    "# The arrays are accessible via  the `.tree` method by passing it a string key:\n",
    "\n",
    "l_ar1 = loaded_data.tree('np.array_0')\n",
    "l_ar2 = loaded_data.tree('np.array_1')\n",
    "\n",
    "print(l_ar1)\n",
    "print(l_ar2)\n",
    "print(l_ar1.data)\n",
    "print(l_ar2.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7746df36",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert(np.array_equal( ar1, l_ar1.data ))\n",
    "assert(np.array_equal( ar2, l_ar2.data ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d9441468",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dictionary_0': Metadata( A Metadata instance called 'dictionary_0', containing the following fields:\n",
       " \n",
       "           array:   2D-array\n",
       "           cow:     moo\n",
       "           tuple:   (1, 2, 3)\n",
       " )}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In emdfile, dictionaries correspond to Metadata class objects, and every data-like class instance can contain\n",
    "# any number of Metadata instances, stored in the `.metadata` property.  For instance, here's what the `.metadata`\n",
    "# property of the Root object we loaded looks like:\n",
    "\n",
    "loaded_data.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "62473011",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Metadata( A Metadata instance called 'dictionary_0', containing the following fields:\n",
       "\n",
       "          array:   2D-array\n",
       "          cow:     moo\n",
       "          tuple:   (1, 2, 3)\n",
       ")"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can retrieve our dictionary by passing the appropriate key to the Root's .metadata property, like this:\n",
    "\n",
    "md = loaded_data.metadata['dictionary_0']\n",
    "\n",
    "md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "be1fd64c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 2],\n",
       "       [3, 4, 5],\n",
       "       [6, 7, 8]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md['array']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "fa825603",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Confirm that everything worked by comparing our original and loaded dictionaries item-by-item\n",
    "\n",
    "for k,v in dic.items():\n",
    "    assert(k in md.keys)\n",
    "    if isinstance(v,np.ndarray): assert(np.array_equal(v,md[k]))\n",
    "    else: assert(v == md[k])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70912ca8",
   "metadata": {},
   "source": [
    "<a id='emdtrees'></a>\n",
    "\n",
    "# 2 EMD Trees\n",
    "\n",
    "This section defines the basic elements of EMD 1.0 files - trees and nodes - and demonstrates the interface to create, write, and read these elements in Python using `emdfile`.\n",
    "\n",
    "Note that this section does **not** cover the data itself.  Here the focus is the structures that relate and group different pieces of data.  In [Section 3: EMD Nodes](#classes) we look at the how to put data into these structures, and the kinds of data that can be stored."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf66a2ec",
   "metadata": {},
   "source": [
    "<a id='trees'></a>\n",
    "\n",
    "## 2.1 Trees\n",
    "\n",
    "The core structure of the EMD 1.0 file is the EMD tree.  An EMD tree is a tree in the usual computer-science sense - think of directories on a filesystem: they are nested containers, with a single highest level container plus any number of additional containers and data nested to any depth inside.  The highest level container is called the \"root directory\" on a filesystem, and an \"EMD root\" or (if it's clear from the context) just a \"root\" in an EMD 1.0 file.  The lower level containers and data are called \"files and folders\" or \"files and directories\" on a filesystem, and are generically referred to as \"nodes\" in an EMD 1.0 file.  In this analogy, EMD nodes do the job of both files and folders: they may both contain data (like a file) and also other nested nodes (like a folder).\n",
    "\n",
    "The image below shows two EMD trees.  On the left is a simple tree, comprised of a root and a single downstream node, and on the right is a complex tree, comprised of a root and many nested downstream nodes.\n",
    "\n",
    "<img src=\"./pngs/tree.png\" alt=\"Trees\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "70152581",
   "metadata": {},
   "outputs": [],
   "source": [
    "# EMD trees must have roots, so to build one from scratch we start with a Root object\n",
    "\n",
    "root = emd.Root( name='an_EMD_root' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "01c7250e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Root( A Node called 'an_EMD_root', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "53dce651",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n"
     ]
    }
   ],
   "source": [
    "# `emdfile` objects have a `.tree` method, which serves as an interface to the EMD tree it is a part of.\n",
    "# Calling `.tree()` with no arguments prints the tree to the screen.  Right now, there's nothing in the tree of\n",
    "# our new root:\n",
    "\n",
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "37ebb406",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Root( A Node called 'an_EMD_root', containing the following top-level objects in its tree:\n",
       "\n",
       "          node                     \t (Node)\n",
       "          lode                     \t (Node)\n",
       "          abode                    \t (Node)\n",
       ")"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate some nodes, and add them to the tree.\n",
    "# More on nodes in section 3.\n",
    "\n",
    "node1 = emd.Node( name='node' )\n",
    "node2 = emd.Node( name='lode' )\n",
    "node3 = emd.Node( name='abode' )\n",
    "\n",
    "\n",
    "# Passing the `.tree` method an EMD object as its argument, like the nodes we just made,\n",
    "# adds that object to the tree\n",
    "\n",
    "root.tree(node1)\n",
    "root.tree(node2)\n",
    "root.tree(node3)\n",
    "\n",
    "root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "94eba88c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node\n",
      "|---lode\n",
      "|---abode\n"
     ]
    }
   ],
   "source": [
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e0f17275",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'abode', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Passing a string argument to `.tree` returns the node of that name, if it exists\n",
    "\n",
    "root.tree('abode')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "d5afe8c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node\n",
      "|   |---snowed\n",
      "|   |---flowed\n",
      "|   |---crowed\n",
      "|---lode\n",
      "|   |---sewed\n",
      "|       |---glowed\n",
      "|           |---reload\n",
      "|---abode\n"
     ]
    }
   ],
   "source": [
    "# The location that new nodes are added to the tree depends on which existing node is used\n",
    "# to call `.tree`.  Here we make some new nodes, and add them to different places in the tree.\n",
    "\n",
    "node4 = emd.Node( name='snowed' )\n",
    "node5 = emd.Node( name='flowed' )\n",
    "node6 = emd.Node( name='crowed' )\n",
    "\n",
    "node7 = emd.Node( name='sewed' )\n",
    "node8 = emd.Node( name='glowed' )\n",
    "node9 = emd.Node( name='reload' )\n",
    "\n",
    "\n",
    "node1.tree(node4)    # these nodes are all added onto `node1` (which we named \"node\")\n",
    "node1.tree(node5)\n",
    "node1.tree(node6)\n",
    "\n",
    "node2.tree(node7)    # these nodes are added to different nodes in the tree\n",
    "node7.tree(node8)\n",
    "node8.tree(node9)\n",
    "\n",
    "\n",
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "423ab02f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---sewed\n",
      "    |---glowed\n",
      "        |---reload\n"
     ]
    }
   ],
   "source": [
    "# from a non-root node, we can display the branch below this node...\n",
    "\n",
    "node2.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "02322a57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node\n",
      "|   |---snowed\n",
      "|   |---flowed\n",
      "|   |---crowed\n",
      "|---lode\n",
      "|   |---sewed\n",
      "|       |---glowed\n",
      "|           |---reload\n",
      "|---abode\n"
     ]
    }
   ],
   "source": [
    "# ...or the whole tree from the root\n",
    "\n",
    "node2.tree( show=True )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "fcb9e449",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'sewed', containing the following top-level objects in its tree:\n",
       "\n",
       "          glowed                   \t (Node)\n",
       ")"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can retrieve objects at nested points in the tree using '/' delimiters\n",
    "\n",
    "root.tree('lode/sewed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "2446e150",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---sewed\n",
      "    |---glowed\n",
      "        |---reload\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'reload', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This works from any node\n",
    "\n",
    "node2.tree()\n",
    "\n",
    "print()\n",
    "\n",
    "node2.tree('sewed/glowed/reload')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "950947b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node\n",
      "|   |---snowed\n",
      "|   |---flowed\n",
      "|   |---crowed\n",
      "|---lode\n",
      "|   |---sewed\n",
      "|       |---glowed\n",
      "|           |---reload\n",
      "|---abode\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'snowed', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also retrieve objects using a root path even if \n",
    "# we're calling .tree from a downstream node, by using a leading '/'\n",
    "\n",
    "node2.tree(show=True)\n",
    "\n",
    "print()\n",
    "\n",
    "node2.tree('/node/snowed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "7e9da828",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root( A Node called 'an_EMD_root', containing the following top-level objects in its tree:\n",
      "\n",
      "          node                     \t (Node)\n",
      "          lode                     \t (Node)\n",
      "          abode                    \t (Node)\n",
      ")\n",
      "Root( A Node called 'an_EMD_root', containing the following top-level objects in its tree:\n",
      "\n",
      "          node                     \t (Node)\n",
      "          lode                     \t (Node)\n",
      "          abode                    \t (Node)\n",
      ")\n",
      "Root( A Node called 'an_EMD_root', containing the following top-level objects in its tree:\n",
      "\n",
      "          node                     \t (Node)\n",
      "          lode                     \t (Node)\n",
      "          abode                    \t (Node)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# The root node of a tree is accessible from any node with the .root property\n",
    "\n",
    "print(root)\n",
    "print(node1.root)\n",
    "print(node8.root)\n",
    "\n",
    "assert(root == root.root == node1.root == node2.root == node3.root == node4.root == node5.root)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a08ffa8c",
   "metadata": {},
   "source": [
    "<a id='writingtrees'></a>\n",
    "\n",
    "## 2.2 Writing and reading EMD trees\n",
    "\n",
    "In this section we\n",
    "- 2.2.1. [write an EMD tree to a file, and read it](#write_one_tree)\n",
    "- 2.2.2. [read a subset of an EMD tree](#read_a_branch)\n",
    "- 2.2.3. [write subsets of an `emdfile` tree to a file](#write_a_node_or_branch)\n",
    "- 2.2.4. [write several trees to one file, then read from them](#write_and_read_multiple_trees)\n",
    "\n",
    "Note that these sections use the tree built in section 2.1, so that should be run before this!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ebe25c2",
   "metadata": {},
   "source": [
    "<a id='write_one_tree'></a>\n",
    "\n",
    "### 2.2.1 Write an EMD tree to a file, and read it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "3c020e5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "ec421570",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node\n",
      "|   |---snowed\n",
      "|   |---flowed\n",
      "|   |---crowed\n",
      "|---lode\n",
      "|   |---sewed\n",
      "|       |---glowed\n",
      "|           |---reload\n",
      "|---abode\n"
     ]
    }
   ],
   "source": [
    "# Inspect the tree we want to write\n",
    "\n",
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "33572d87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write it to a file\n",
    "\n",
    "emd.save(filepath, root)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "88c38322",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---an_EMD_root\n",
      "    |---abode\n",
      "    |---lode\n",
      "    |   |---sewed\n",
      "    |       |---glowed\n",
      "    |           |---reload\n",
      "    |---node\n",
      "        |---crowed\n",
      "        |---flowed\n",
      "        |---snowed\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the resulting file\n",
    "\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "bbcba0f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the tree\n",
    "\n",
    "loaded_data = emd.read(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "fef44b07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root( A Node called 'an_EMD_root', containing the following top-level objects in its tree:\n",
      "\n",
      "          abode                    \t (Node)\n",
      "          lode                     \t (Node)\n",
      "          node                     \t (Node)\n",
      ")\n",
      "\n",
      "/\n",
      "|---abode\n",
      "|---lode\n",
      "|   |---sewed\n",
      "|       |---glowed\n",
      "|           |---reload\n",
      "|---node\n",
      "    |---crowed\n",
      "    |---flowed\n",
      "    |---snowed\n"
     ]
    }
   ],
   "source": [
    "# Inspect the loaded tree\n",
    "\n",
    "print(loaded_data)\n",
    "\n",
    "print()\n",
    "\n",
    "loaded_data.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1c24cf9",
   "metadata": {},
   "source": [
    "<a id='read_a_tree_subset'></a>\n",
    "\n",
    "### 2.2.2 Read a subset of an EMD tree\n",
    "\n",
    "There are three different subsets of an EMD tree that can be specified:\n",
    "- a single node\n",
    "- a branch, including its source node\n",
    "- a branch, excluding its source node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "be1dabde",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---an_EMD_root\n",
      "    |---abode\n",
      "    |---lode\n",
      "    |   |---sewed\n",
      "    |       |---glowed\n",
      "    |           |---reload\n",
      "    |---node\n",
      "        |---crowed\n",
      "        |---flowed\n",
      "        |---snowed\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the file we want to read from\n",
    "\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fa770fc",
   "metadata": {},
   "source": [
    "<a id='read_a_single_node'></a>\n",
    "\n",
    "#### 2.2.2.1 Read a single node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "b2aed491",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To read a subset of an EMD tree, we need to specify where it is in the file.\n",
    "# This is done with the `emdpath` argument\n",
    "\n",
    "# The type of subset that will be read is specified with the `tree` argument.\n",
    "# Here we set `tree=False`, which means: read the selected node only\n",
    "\n",
    "loaded_data = emd.read(\n",
    "    filepath,\n",
    "    emdpath = '/an_EMD_root/lode/sewed',\n",
    "    tree = False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "a7dbbddc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'sewed', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the data we loaded.  As expected, it's the node called \"sewed\"...\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "a919ab2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n"
     ]
    }
   ],
   "source": [
    "# ...and the tree underneath it is empty, i.e. we've loaded this node only\n",
    "\n",
    "loaded_data.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0291b0fa",
   "metadata": {},
   "source": [
    "<a id='read_a_branch'></a>\n",
    "\n",
    "#### 2.2.2.2 Read a branch of a tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "d7828d46",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To read an entire branch of a tree, starting at the selected node, we set `tree=True`\n",
    "\n",
    "loaded_data = emd.read(\n",
    "    filepath,\n",
    "    emdpath = '/an_EMD_root/lode/sewed',\n",
    "    tree = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "30f156e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'sewed', containing the following top-level objects in its tree:\n",
       "\n",
       "          glowed                   \t (Node)\n",
       ")"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The loaded data is again the node called \"sewed\"...\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "34a80317",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---glowed\n",
      "    |---reload\n"
     ]
    }
   ],
   "source": [
    "# ...but this time it contains any nodes that were downstream of itself in its runtime tree\n",
    "\n",
    "loaded_data.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b989742",
   "metadata": {},
   "source": [
    "<a id='read_a_branch_exclude_source_node'></a>\n",
    "\n",
    "#### 2.2.2.3 Read a branch of a tree, excluding the selected starting node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "5b353c2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To read a branch of a tree but exclude the selected starting node, we set `tree='branch'`\n",
    "\n",
    "loaded_data = emd.read(\n",
    "    filepath,\n",
    "    emdpath = '/an_EMD_root/lode/sewed',\n",
    "    tree = 'branch'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "c76e50b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Root( A Node called 'an_EMD_root', containing the following top-level objects in its tree:\n",
       "\n",
       "          glowed                   \t (Node)\n",
       ")"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This time the loaded data is not the selected node - we didn't want to read that node! - and\n",
    "# instead is a Root, with its name coming from the root of the whole tree.\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "73503f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Incidentally, because an EMD tree - whether in an EMD 1.0 HDF5 file, or in an `emdfile` Python\n",
    "# representation - must always have a root, the root was also present in `loaded_data` in the\n",
    "# previous examples, it just wasn't what was returned by the read function.  You can check that it\n",
    "# was there each time by running `loaded_data.root` in the previous examples.\n",
    "\n",
    "# Roots are discussed further in Section 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "9327e441",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---glowed\n",
      "    |---reload\n"
     ]
    }
   ],
   "source": [
    "# The root that's been returned carries the data we asked for - the tree branch downstream of the node \"sewed\"\n",
    "\n",
    "loaded_data.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcfbc2cf",
   "metadata": {},
   "source": [
    "<a id='write_a_tree_subset'></a>\n",
    "\n",
    "### 2.2.3 Write a subset of an `emdfile` tree to a file\n",
    "\n",
    "Writing a subset of an `emdfile` tree parallels reading a subset of an EMD tree.  Here we'll write:\n",
    "- a single node\n",
    "- a branch, including its source node\n",
    "- a branch, excluding its source node"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d962ec7",
   "metadata": {},
   "source": [
    "<a id='write_a_single_node'></a>\n",
    "\n",
    "#### 2.2.3.1 Write a single node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "188e533b",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "2038e4b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node\n",
      "|   |---snowed\n",
      "|   |---flowed\n",
      "|   |---crowed\n",
      "|---lode\n",
      "|   |---sewed\n",
      "|       |---glowed\n",
      "|           |---reload\n",
      "|---abode\n"
     ]
    }
   ],
   "source": [
    "# Inspect the tree we're writing from\n",
    "\n",
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "6b2cd1cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'sewed', containing the following top-level objects in its tree:\n",
       "\n",
       "          glowed                   \t (Node)\n",
       ")"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's say we want to save the node 'sewed'.  First let's retrieve the node:\n",
    "\n",
    "my_favorite_node = root.tree('lode/sewed')\n",
    "\n",
    "my_favorite_node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "23d79ad3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To save this node, we pass it to the save function with `tree=False`\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    my_favorite_node,\n",
    "    tree = False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "d8e7829d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---an_EMD_root\n",
      "    |---sewed\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the resulting file\n",
    "\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "6aff8db6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node( A Node called 'sewed', containing the following top-level objects in its tree:\n",
      "\n",
      ")\n",
      "\n",
      "/\n"
     ]
    }
   ],
   "source": [
    "# Because there's only one node under one tree in this file, we can load it with no arguments to read\n",
    "# other than the filepath\n",
    "\n",
    "loaded_data = emd.read(filepath)\n",
    "\n",
    "print(loaded_data)\n",
    "print()\n",
    "loaded_data.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f44dd784",
   "metadata": {},
   "source": [
    "<a id='write_a_branch'></a>\n",
    "\n",
    "#### 2.2.3.2 Write a branch of a tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "f5590d33",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "7cc42715",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node( A Node called 'sewed', containing the following top-level objects in its tree:\n",
      "\n",
      "          glowed                   \t (Node)\n",
      ")\n",
      "\n",
      "/\n",
      "|---glowed\n",
      "    |---reload\n"
     ]
    }
   ],
   "source": [
    "# Let's write the branch starting at our favorite node \"sewed\"\n",
    "\n",
    "print(my_favorite_node)\n",
    "print()\n",
    "my_favorite_node.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "c0cd629c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the data\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    my_favorite_node,\n",
    "    #tree = True         # this line can be added in or left out - i.e. this is the default setting\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "e5a34ede",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---an_EMD_root\n",
      "    |---sewed\n",
      "        |---glowed\n",
      "            |---reload\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the resulting file\n",
    "\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d1cb555",
   "metadata": {},
   "source": [
    "<a id='write_a_branch_exclude_source_node'></a>\n",
    "\n",
    "#### 2.2.3.3 Write a branch of a tree, excluding its source node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "d3c974fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "d5d22b34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll use node 'sewed' again.  All we need to do is modify the 'tree' argument\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    my_favorite_node,\n",
    "    tree = None\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "eb6d437d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---an_EMD_root\n",
      "    |---glowed\n",
      "        |---reload\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the resulting file\n",
    "\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "397befab",
   "metadata": {},
   "source": [
    "<a id='write_and_read_multiple_trees'></a>\n",
    "\n",
    "### 2.2.4 Write/read several EMD trees to a single file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "9663a6d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "ff707885",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---hello\n",
      "|   |---hellohello\n",
      "|---yousay\n",
      "    |---goodbye\n",
      "        |---eyesay\n",
      "            |---HELLOWORLD\n"
     ]
    }
   ],
   "source": [
    "# Make a second `emdfile` tree\n",
    "\n",
    "# Note that this syntax is condensed but not very nice...\n",
    "# for real data, I wouldn't recommend doing it like this!\n",
    "root2 = emd.Root('another_root_to_boot')\n",
    "root2.tree( emd.Node('hello') )\n",
    "root2.tree('hello').tree( emd.Node('hellohello') )\n",
    "root2.tree( emd.Node('yousay') )\n",
    "root2.tree('yousay').tree( emd.Node('goodbye') )\n",
    "root2.tree('yousay/goodbye').tree( emd.Node('eyesay') )\n",
    "root2.tree('yousay/goodbye/eyesay').tree( emd.Node('HELLOWORLD') )\n",
    "\n",
    "root2.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "ed8f9499",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save both trees to a file\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    [root,root2]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "2de7c644",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---an_EMD_root\n",
      "|   |---abode\n",
      "|   |---lode\n",
      "|   |   |---sewed\n",
      "|   |       |---glowed\n",
      "|   |           |---reload\n",
      "|   |---node\n",
      "|       |---crowed\n",
      "|       |---flowed\n",
      "|       |---snowed\n",
      "|---another_root_to_boot\n",
      "    |---hello\n",
      "    |   |---hellohello\n",
      "    |---yousay\n",
      "        |---goodbye\n",
      "            |---eyesay\n",
      "                |---HELLOWORLD\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show the results\n",
    "\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "3a47d264",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Alternatively, we can save one tree first, then\n",
    "# come back and save the second later by invoking append mode\n",
    "\n",
    "clean()\n",
    "\n",
    "# save the first tree\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root\n",
    ")\n",
    "\n",
    "# append the second tree\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root2,\n",
    "    mode = 'a'         # aliases for 'a' include 'append' or '+'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "4a7b89b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---an_EMD_root\n",
      "|   |---abode\n",
      "|   |---lode\n",
      "|   |   |---sewed\n",
      "|   |       |---glowed\n",
      "|   |           |---reload\n",
      "|   |---node\n",
      "|       |---crowed\n",
      "|       |---flowed\n",
      "|       |---snowed\n",
      "|---another_root_to_boot\n",
      "    |---hello\n",
      "    |   |---hellohello\n",
      "    |---yousay\n",
      "        |---goodbye\n",
      "            |---eyesay\n",
      "                |---HELLOWORLD\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show the file\n",
    "\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "d583dfe0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Multiple root groups detected - returning root names. Please specify the `emdpath` argument. Returning the list of rootgroups.\n",
      "\n",
      "['an_EMD_root', 'another_root_to_boot']\n"
     ]
    }
   ],
   "source": [
    "# If a file contains multiple trees, trying to read it without\n",
    "# specifying the data we want returns a list of the available tree roots\n",
    "\n",
    "loaded_data = emd.read(filepath)\n",
    "\n",
    "print()\n",
    "print(loaded_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "8013adae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root( A Node called 'an_EMD_root', containing the following top-level objects in its tree:\n",
      "\n",
      "          abode                    \t (Node)\n",
      "          lode                     \t (Node)\n",
      "          node                     \t (Node)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# We can load a tree by specifying it with the `emdpath` argument\n",
    "\n",
    "loaded_data = emd.read(\n",
    "    filepath,\n",
    "    emdpath = 'an_EMD_root'\n",
    ")\n",
    "\n",
    "print(loaded_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3465a6a7",
   "metadata": {},
   "source": [
    "<a id='unrooted_nodes'></a>\n",
    "\n",
    "### 2.2.5 Unrooted nodes\n",
    "\n",
    "EMD trees and `emdfile` trees must have a root node.  It is possible to create and use nodes that don't have roots; it is **not** possible to use their `.tree` methods until they are added to a root or an existing (rooted) tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "b8dbc8c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'unrooted_node', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make an unrooted node\n",
    "# Making and calling on an unrooted node is ok\n",
    "\n",
    "unrooted_node = emd.Node( name='unrooted_node')\n",
    "\n",
    "unrooted_node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "beafce38",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hit the road, Jack\n"
     ]
    }
   ],
   "source": [
    "# However, trying to build a tree with an unrooted node will throw an error\n",
    "\n",
    "another_node = emd.Node( name='another_node' )\n",
    "\n",
    "try:\n",
    "    unrooted_node.tree( another_node )   # raises an Error!\n",
    "\n",
    "except AssertionError:\n",
    "    print('hit the road, Jack')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25a283dd",
   "metadata": {},
   "source": [
    "<a id='emdnodes'></a>\n",
    "\n",
    "# 3. EMD Nodes\n",
    "\n",
    "Each EMD node can store both data and metadata.  The kind of data stored, and how it's stored and retrieved, depends on the node type.  Metadata is stored and retrieved identically in all node types.  This section covers data; [Section 4](#emdmetadata) covers metadata.\n",
    "\n",
    "\n",
    "The generic node structure as stored in an EMD 1.0 H5 file is shown below.  For the purpose of using `emdfile` to write and read data, the important part here is the yellow block called \"Data\", and which will vary from one node type to the next.\n",
    "\n",
    "\n",
    "<img src=\"./pngs/node.png\" alt=\"Nodes\" width=\"1000\"/>\n",
    "\n",
    "\n",
    "The `emdfile` node types, and the kind of data stored in their \"Data\" block, are:\n",
    "\n",
    "- [`Node`](#node): no \"Data\" block\n",
    "- [`Array`](#array): array-like data\n",
    "- [`PointList`](#pointlist): point-like data, representing N points in M string-labelled dimensions\n",
    "- [`PointListArray`](#pointlistarray): ragged-array-like, representing 2D grids of variable length pointlist-like data\n",
    "- [`Root`](#root): no \"Data\" block; a `Root` is like a `Node` plus a few special properties\n",
    "- [`Custom`](#custom): container for arbitrary flat or tree-like combinations of the prior data types"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ab3d31",
   "metadata": {},
   "source": [
    "<a id='node'></a>\n",
    "\n",
    "## 3.1 `Node`\n",
    "\n",
    "The `Node` class does not contain a \"Data\" block.\n",
    "\n",
    "It does include the structure for\n",
    "- the `.tree` method, for building and interacting with `emdfile` trees ([Section 2](#emdtrees)), and\n",
    "- the `.metadata` property, for storing and retrieving arbitrarily many dictionary like `Metadata` objects ([Section 4](#emdmetadata))\n",
    "\n",
    "All the other public `emdfile` classes aside from `Metadata` inherit from `Node`, and therefore also have these routines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "948eba70",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'some_node', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Nodes need names, so they can be referenced and retrieved later!\n",
    "\n",
    "node = emd.Node( name='some_node' )\n",
    "\n",
    "node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "a2c00746",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n"
     ]
    }
   ],
   "source": [
    "# Nodes possess the `.tree` method.  See Section 2.\n",
    "\n",
    "node.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "f137ef62",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Nodes possess the `.metadata` property. See Section 4.\n",
    "\n",
    "node.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f80dd19a",
   "metadata": {},
   "source": [
    "<a id='array'></a>\n",
    "\n",
    "## 3.2 `Array`\n",
    "\n",
    "In an `Array` the data block is array-like.\n",
    "\n",
    "In addition to the `.metadata` properties which stores general/arbitrary metadata, each of the data-containing node types contain some self-descriptive metadata representing the space/dimensions its data quantifies. In an `Array` instance these are the units of the pixel values and, for an N-dimensional data array, N vectors specifying the names, units, and interpolation points of the coordinate axes sampled by the array.\n",
    "\n",
    "In addition to holding a single array, `Array` supports stacks of arrays, in which the data is (N+1)-dimensional and represents M distinct N dimensional arrays, each with a string name and associated with the N-dimensional space defined by the N dim vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d317bc72",
   "metadata": {},
   "source": [
    "<a id='singlearray'></a>\n",
    "\n",
    "### 3.2.1 Single arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "68ef0098",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 0,  1,  2,  3,  4],\n",
       "        [ 5,  6,  7,  8,  9],\n",
       "        [10, 11, 12, 13, 14],\n",
       "        [15, 16, 17, 18, 19]],\n",
       "\n",
       "       [[20, 21, 22, 23, 24],\n",
       "        [25, 26, 27, 28, 29],\n",
       "        [30, 31, 32, 33, 34],\n",
       "        [35, 36, 37, 38, 39]],\n",
       "\n",
       "       [[40, 41, 42, 43, 44],\n",
       "        [45, 46, 47, 48, 49],\n",
       "        [50, 51, 52, 53, 54],\n",
       "        [55, 56, 57, 58, 59]]])"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generate some data\n",
    "\n",
    "data = np.arange(60).reshape((3,4,5))\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "0ba5f4d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array( A 3-dimensional array of shape (3, 4, 5) called 'arrrrrr',\n",
       "       with dimensions:\n",
       "\n",
       "       dim0 = [0,1,...] pixels\n",
       "       dim1 = [0,1,...] pixels\n",
       "       dim2 = [0,1,...] pixels\n",
       ")"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make the Array\n",
    "\n",
    "ar = emd.Array(\n",
    "    name = 'arrrrrr',\n",
    "    data = data\n",
    ")\n",
    "\n",
    "ar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "965cce2d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 0,  1,  2,  3,  4],\n",
       "        [ 5,  6,  7,  8,  9],\n",
       "        [10, 11, 12, 13, 14],\n",
       "        [15, 16, 17, 18, 19]],\n",
       "\n",
       "       [[20, 21, 22, 23, 24],\n",
       "        [25, 26, 27, 28, 29],\n",
       "        [30, 31, 32, 33, 34],\n",
       "        [35, 36, 37, 38, 39]],\n",
       "\n",
       "       [[40, 41, 42, 43, 44],\n",
       "        [45, 46, 47, 48, 49],\n",
       "        [50, 51, 52, 53, 54],\n",
       "        [55, 56, 57, 58, 59]]])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "24749ec5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1,  2,  3,  4],\n",
       "       [ 6,  7,  8,  9],\n",
       "       [11, 12, 13, 14],\n",
       "       [16, 17, 18, 19]])"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can slice into the array using numpy-like syntax on Array directly\n",
    "\n",
    "ar[0,:,1:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "1b5d7c9a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arrrrrr\n",
      "\n",
      "\n",
      "[0 1 2]\n",
      "[0 1 2 3]\n",
      "[0 1 2 3 4]\n",
      "\n",
      "dim0\n",
      "dim1\n",
      "dim2\n",
      "\n",
      "pixels\n",
      "pixels\n",
      "pixels\n"
     ]
    }
   ],
   "source": [
    "# Self-descriptive metadata\n",
    "\n",
    "print(ar.name)\n",
    "print(ar.units)\n",
    "print()\n",
    "print(ar.dims[0])\n",
    "print(ar.dims[1])\n",
    "print(ar.dims[2])\n",
    "print()\n",
    "print(ar.dim_names[0])\n",
    "print(ar.dim_names[1])\n",
    "print(ar.dim_names[2])\n",
    "print()\n",
    "print(ar.dim_units[0])\n",
    "print(ar.dim_units[1])\n",
    "print(ar.dim_units[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "9c8f3868",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The units attribute can be modified directly\n",
    "\n",
    "ar.units = 'cows'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "4063f601",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The dims should be set using the .set_dim method\n",
    "\n",
    "ar.set_dim?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "3ab55257",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  5, 10])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.set_dim(\n",
    "    0,                   # which dimension\n",
    "    dim = [0,5],         # when two numbers are passed the vector is extrapolated linearly\n",
    ")\n",
    "\n",
    "ar.dims[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "96befe52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the name and units can be set with \n",
    "# their own method calls...\n",
    "\n",
    "ar.set_dim_name(\n",
    "    0,\n",
    "    'x-axis'\n",
    ")\n",
    "ar.set_dim_units(\n",
    "    0,\n",
    "    'pastures'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "fb76e6f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 2, 4, 6])"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ...or inside a call to `.set_dim`\n",
    "\n",
    "ar.set_dim(\n",
    "    1,\n",
    "    dim = 2,             # when one number x is passed, the vect is extraplated linearly from [0,x] \n",
    "    name = 'y-axis',\n",
    "    units = 'fields'\n",
    ")\n",
    "\n",
    "ar.dims[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "147c8910",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.e-02, 1.e-01, 1.e+00, 1.e+01, 1.e+02])"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.set_dim(\n",
    "    2,\n",
    "    dim = np.logspace(-2,2,5),   # when a 1D array is passed, its length must match the array dim length \n",
    "    name = 'z-axis',\n",
    "    units = 'tracts'\n",
    ")\n",
    "\n",
    "ar.dims[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "09836a24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arrrrrr\n",
      "cows\n",
      "\n",
      "[ 0  5 10]\n",
      "[0 2 4 6]\n",
      "[1.e-02 1.e-01 1.e+00 1.e+01 1.e+02]\n",
      "\n",
      "x-axis\n",
      "y-axis\n",
      "z-axis\n",
      "\n",
      "pastures\n",
      "fields\n",
      "tracts\n"
     ]
    }
   ],
   "source": [
    "print(ar.name)\n",
    "print(ar.units)\n",
    "print()\n",
    "print(ar.dims[0])\n",
    "print(ar.dims[1])\n",
    "print(ar.dims[2])\n",
    "print()\n",
    "print(ar.dim_names[0])\n",
    "print(ar.dim_names[1])\n",
    "print(ar.dim_names[2])\n",
    "print()\n",
    "print(ar.dim_units[0])\n",
    "print(ar.dim_units[1])\n",
    "print(ar.dim_units[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "4d04f1cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Alternatively, the Array can be initialized with the name and dims specified\n",
    "\n",
    "ar = emd.Array(\n",
    "    data = np.array(\n",
    "        [[1,2,3],\n",
    "         [4,5,6]]\n",
    "    ),\n",
    "    name = 'my_array',\n",
    "    units = 'intensity',\n",
    "    dims = [[0,3],\n",
    "            [0,0.6]],\n",
    "    dim_names = ['x','y'],\n",
    "    dim_units = ['nm','km']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "0daaa6cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array( A 2-dimensional array of shape (2, 3) called 'my_array',\n",
       "       with dimensions:\n",
       "\n",
       "       x = [0,3,...] nm\n",
       "       y = [0.0,0.6,...] km\n",
       ")"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "910a32c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0, 3], array([0. , 0.6, 1.2]))"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.dims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "e64bce98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [4, 5, 6]])"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "171bc548",
   "metadata": {},
   "source": [
    "<a id='arraystacks'></a>\n",
    "\n",
    "### 3.2.2 Stack arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "4edd8b89",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[    1,     2,     3,     4],\n",
       "        [    5,     6,     7,     8],\n",
       "        [    9,    10,    11,    12]],\n",
       "\n",
       "       [[   10,    11,    12,    13],\n",
       "        [   14,    15,    16,    17],\n",
       "        [   18,    19,    20,    21]],\n",
       "\n",
       "       [[  100,   101,   102,   103],\n",
       "        [  104,   105,   106,   107],\n",
       "        [  108,   109,   110,   111]],\n",
       "\n",
       "       [[ 1000,  1001,  1002,  1003],\n",
       "        [ 1004,  1005,  1006,  1007],\n",
       "        [ 1008,  1009,  1010,  1011]],\n",
       "\n",
       "       [[10000, 10001, 10002, 10003],\n",
       "        [10004, 10005, 10006, 10007],\n",
       "        [10008, 10009, 10010, 10011]]])"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make some data\n",
    "# The code below makes 5 3x4 arrays, then combines them into a single array\n",
    "\n",
    "data = np.stack([ np.arange(12).reshape(3,4) + 10**x for x in range(5)])\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "8fc97be5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make the Array\n",
    "\n",
    "ar = emd.Array(\n",
    "    data = data,\n",
    "    name = 'my_stack_array',\n",
    "    units = 'intensity',\n",
    "    dims = [[0,3],                  # we want only two dim vectors!\n",
    "            [0,0.6]],\n",
    "    dim_names = ['x','y'],\n",
    "    dim_units = ['nm','km'],\n",
    "    slicelabels = ['a','b','c','d','e']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "6c55c17c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array( A stack of 5 Arrays with 2-dimensions and shape (3, 4), called 'my_stack_array'\n",
       "\n",
       "       The labels are:\n",
       "           a\n",
       "           b\n",
       "           c\n",
       "           d\n",
       "           e\n",
       "\n",
       "\n",
       "       The Array dimensions are:\n",
       "           x = [0,3,...] nm\n",
       "           y = [0.0,0.6,...] km\n",
       ")"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "9dfe9c1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[    1,     2,     3,     4],\n",
       "        [    5,     6,     7,     8],\n",
       "        [    9,    10,    11,    12]],\n",
       "\n",
       "       [[   10,    11,    12,    13],\n",
       "        [   14,    15,    16,    17],\n",
       "        [   18,    19,    20,    21]],\n",
       "\n",
       "       [[  100,   101,   102,   103],\n",
       "        [  104,   105,   106,   107],\n",
       "        [  108,   109,   110,   111]],\n",
       "\n",
       "       [[ 1000,  1001,  1002,  1003],\n",
       "        [ 1004,  1005,  1006,  1007],\n",
       "        [ 1008,  1009,  1010,  1011]],\n",
       "\n",
       "       [[10000, 10001, 10002, 10003],\n",
       "        [10004, 10005, 10006, 10007],\n",
       "        [10008, 10009, 10010, 10011]]])"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# `.data` still points to the full 3D data stack\n",
    "\n",
    "ar.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "04a03762",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5, 3, 4)\n",
      "(3, 4)\n"
     ]
    }
   ],
   "source": [
    "# note the difference between `.shape` and `.data.shape` - in a non-stacked array,\n",
    "# these would be the same, while here, `.shape` refers to the shared N-D space\n",
    "# spanned by the dim vectors\n",
    "\n",
    "print(ar.data.shape)\n",
    "print(ar.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "3b443753",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0, 3, 6]), array([0. , 0.6, 1.2, 1.8]))"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.dims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "20dc6aba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array( A 2-dimensional array of shape (3, 4) called 'my_stack_array_a',\n",
       "       with dimensions:\n",
       "\n",
       "       x = [0,3,...] nm\n",
       "       y = [0.0,0.6,...] km\n",
       ")"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can slice into the array with strings to get the labelled subarrays back\n",
    "# Note that when accessed in this way, the return value is a new Array instances\n",
    "\n",
    "ar['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "0d654300",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1,  2,  3,  4],\n",
       "       [ 5,  6,  7,  8],\n",
       "       [ 9, 10, 11, 12]])"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar['a'].data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "cd3bf0ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 7])"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# labelled subarrays can be sliced in the same [...] call\n",
    "\n",
    "ar['a',:2,2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "5b8a38d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 7])"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar['a'].data[:2,2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "9d385c33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1,  2,  3,  4],\n",
       "       [ 5,  6,  7,  8],\n",
       "       [ 9, 10, 11, 12]])"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The order of the labels is fixed, and when slicing we can always replace the labels\n",
    "# with their corresponding index.  Note that in this case, a regular numpy array is\n",
    "# returned and no new Array instances are created\n",
    "\n",
    "ar[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "8051acbc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 7])"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar[0,:2,2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "01aadfd9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "# the `.rank` attribute returns N and the `.data.ndim` attribute gives N+1 for\n",
    "# an N+1 dimensional stack array. For non-stack arrays, `.rank` and `data.ndim` are identical\n",
    "\n",
    "print(ar.rank)\n",
    "print(ar.data.ndim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "a3172d74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the `.depth` property gives the number of slices. For non-stack arrays, it's 0\n",
    "\n",
    "ar.depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "c6123018",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array( A 2-dimensional array of shape (3, 4) called 'dingo',\n",
       "       with dimensions:\n",
       "\n",
       "       x = [0,3,...] nm\n",
       "       y = [0.0,0.6,...] km\n",
       ")"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# When accessing subarrays with ar['label'] syntax, we got an Array back named \"name_label\".\n",
    "# We can grab a subarray and give the new Array some other name\n",
    "\n",
    "ar.get_slice('a','dingo')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3785f7b4",
   "metadata": {},
   "source": [
    "<a id='pointlist'></a>\n",
    "\n",
    "\n",
    "## 3.3 `PointList`\n",
    "\n",
    "A `PointList`s data attribute can have any length, with any number of string-named fields, and each field may have its own data type.  PointList wraps numpy structured arrays.\n",
    "\n",
    "PointLists have variable length that can change at runtime.  Routines are provided to add points, remove points, sort by field, copy a pointlist, or copy a pointlist with additional dimensions (fields) added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "70d3f90c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0, 0.), (0, 0.), (0, 0.)], dtype=[('x', '<i8'), ('y', '<f8')])"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make some data\n",
    "# we define the fields by specifying a custom `dtype` for numpy\n",
    "\n",
    "data = np.zeros(\n",
    "    3,\n",
    "    dtype = [\n",
    "        ('x',int),\n",
    "        ('y',float)\n",
    "    ]\n",
    ")\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "fe1bff50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PointList( A length 3 PointList called 'my_pointlist',\n",
       "           with 2 fields:\n",
       "\n",
       "           x   (int64)\n",
       "           y   (float64)\n",
       ")"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make a PointList\n",
    "\n",
    "pointlist = emd.PointList(\n",
    "    name = 'my_pointlist',\n",
    "    data = data\n",
    ")\n",
    "\n",
    "pointlist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "6d2668e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0, 0.), (0, 0.), (0, 0.)], dtype=[('x', '<i8'), ('y', '<f8')])"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pointlist.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "23d8e29f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0]\n",
      "[0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "# the fields can be accessed by slicing directly into the PointList\n",
    "\n",
    "print(pointlist['x'])\n",
    "print(pointlist['y'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "c8e06e6a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0, 0.), (0, 0.), (0, 0.), (1, 1.), (1, 1.), (1, 1.)],\n",
       "      dtype=[('x', '<i8'), ('y', '<f8')])"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add points\n",
    "\n",
    "# the new data can be either a numpy structured array, or another PointList\n",
    "# either way, it must have the same dtype as the existing data\n",
    "\n",
    "# make the new data\n",
    "new_data = np.ones(\n",
    "    3,\n",
    "    dtype = [\n",
    "        ('x',int),\n",
    "        ('y',float)\n",
    "    ]\n",
    ")\n",
    "\n",
    "# add it to the array\n",
    "pointlist.add(new_data)\n",
    "\n",
    "# show\n",
    "pointlist.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "65068e5e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0, 0.), (0, 0.), (0, 0.), (1, 1.), (1, 1.), (1, 1.), (2, 2.),\n",
       "       (2, 2.), (2, 2.)], dtype=[('x', '<i8'), ('y', '<f8')])"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add points using another PointList\n",
    "\n",
    "# make the new data\n",
    "more_new_data = emd.PointList(\n",
    "    name = 'a_pointlist',\n",
    "    data = np.full(3,2,\n",
    "        dtype = [\n",
    "            ('x',int),\n",
    "            ('y',float)\n",
    "        ])\n",
    ")\n",
    "\n",
    "# add it to the array\n",
    "pointlist.add(more_new_data)\n",
    "\n",
    "# show\n",
    "pointlist.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "3868373f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0, 0.  ), (0, 0.  ), (0, 0.  ), (1, 1.  ), (1, 1.  ), (1, 1.  ),\n",
       "       (2, 2.  ), (2, 2.  ), (2, 2.  ), (5, 5.  ), (6, 5.25), (7, 5.5 ),\n",
       "       (8, 5.75), (9, 6.  )], dtype=[('x', '<i8'), ('y', '<f8')])"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add data as 1D vectors corresponding to the fields\n",
    "\n",
    "pointlist.add_data_by_field(\n",
    "    data = (np.linspace(5,6,num=5),np.arange(5,10)),\n",
    "    fields = ('y','x')\n",
    ")\n",
    "\n",
    "pointlist.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "39e2f000",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(2, 2.  ), (2, 2.  ), (2, 2.  ), (5, 5.  ), (6, 5.25), (7, 5.5 ),\n",
       "       (8, 5.75), (9, 6.  )], dtype=[('x', '<i8'), ('y', '<f8')])"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove points\n",
    "\n",
    "# this is a little clunky! uses a boolean mask...\n",
    "\n",
    "# remove the last two points\n",
    "rm = np.zeros(pointlist.length,dtype = bool)\n",
    "rm[:6] = True                   # flag points for removal\n",
    "pointlist.remove(rm)\n",
    "\n",
    "# show\n",
    "pointlist.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "ed626dae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(2, 2.  , False), (2, 2.  , False), (2, 2.  , False),\n",
       "       (5, 5.  , False), (6, 5.25, False), (7, 5.5 , False),\n",
       "       (8, 5.75, False), (9, 6.  , False)],\n",
       "      dtype=[('x', '<i8'), ('y', '<f8'), ('z', '?')])"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make a new pointlist like this one, with some additional fields added\n",
    "\n",
    "pointlist_copy = pointlist.add_fields(\n",
    "    [('z',bool)],\n",
    "    name = 'another_pointlist',\n",
    ")\n",
    "\n",
    "pointlist_copy.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "e3873c8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(2, 2.  , False), (2, 2.  , False), (2, 2.  , False),\n",
       "       (5, 5.  , False), (6, 5.25,  True), (7, 5.5 ,  True),\n",
       "       (8, 5.75,  True), (9, 6.  ,  True)],\n",
       "      dtype=[('x', '<i8'), ('y', '<f8'), ('z', '?')])"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# modify values in an existing field\n",
    "\n",
    "pointlist_copy['z'][4:] = True\n",
    "\n",
    "pointlist_copy.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "61070e7f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(9, 6.  ,  True), (8, 5.75,  True), (7, 5.5 ,  True),\n",
       "       (6, 5.25,  True), (5, 5.  , False), (2, 2.  , False),\n",
       "       (2, 2.  , False), (2, 2.  , False)],\n",
       "      dtype=[('x', '<i8'), ('y', '<f8'), ('z', '?')])"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sort the pointlist by one of its fields\n",
    "\n",
    "pointlist_copy.sort('y', order='descending')\n",
    "\n",
    "pointlist_copy.data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffb0872f",
   "metadata": {},
   "source": [
    "<a id='pointlistarray'></a>\n",
    "\n",
    "\n",
    "## 3.4 `PointListArray`\n",
    "\n",
    "`emd.PointListArray` represents 2D grids of PointList instances with the same data fields.  It stores 2D ragged arrays of vectors of any length with string-accessible fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "2b2aa1fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PointListArray( A shape (5, 6) PointListArray called 'my_pointlistarray',\n",
       "                with 2 fields:\n",
       "\n",
       "                x   (int64)\n",
       "                y   (int64)\n",
       ")"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make a PointListArray\n",
    "\n",
    "shape = (5,6)\n",
    "dtype = [('x',int),('y',int)]\n",
    "\n",
    "pointlistarray = emd.PointListArray(\n",
    "    name = 'my_pointlistarray',\n",
    "    shape = shape,\n",
    "    dtype = dtype\n",
    ")\n",
    "\n",
    "pointlistarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "816a7193",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PointList( A length 0 PointList called '0,0',\n",
       "           with 2 fields:\n",
       "\n",
       "           x   (int64)\n",
       "           y   (int64)\n",
       ")"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the pointlists can be accessed by slicing into the pointlistarray\n",
    "\n",
    "pointlistarray[0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "05040a12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([], dtype=[('x', '<i8'), ('y', '<i8')])"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and are instantiated empty\n",
    "\n",
    "pointlistarray[3,4].data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "1d585cf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can populate the pointlists with the `add` method\n",
    "\n",
    "for ii in range(pointlistarray.shape[0]):\n",
    "    for jj in range(pointlistarray.shape[1]):\n",
    "        \n",
    "        # set an integer value that varies sinusoidally from 0 to 8\n",
    "        val = int(np.round((np.sin((ii*shape[1]+jj) * 2*np.pi / np.prod(shape)) + 1) * 4))\n",
    "        \n",
    "        # add to the pointlist\n",
    "        pointlistarray[ii,jj].add(\n",
    "            np.full(\n",
    "                shape = val,\n",
    "                fill_value= val,\n",
    "                dtype = dtype\n",
    "            )\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "460b082e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(4, 4), (4, 4), (4, 4), (4, 4)], dtype=[('x', '<i8'), ('y', '<i8')])"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pointlistarray[0,0].data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "1b9122aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(4, 4) (4, 4) (4, 4) (4, 4)]\n",
      "[(5, 5) (5, 5) (5, 5) (5, 5) (5, 5)]\n",
      "[(6, 6) (6, 6) (6, 6) (6, 6) (6, 6) (6, 6)]\n",
      "[(6, 6) (6, 6) (6, 6) (6, 6) (6, 6) (6, 6)]\n",
      "[(7, 7) (7, 7) (7, 7) (7, 7) (7, 7) (7, 7) (7, 7)]\n",
      "[(7, 7) (7, 7) (7, 7) (7, 7) (7, 7) (7, 7) (7, 7)]\n",
      "[(8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8)]\n",
      "[(8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8)]\n",
      "[(8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8)]\n",
      "[(8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8) (8, 8)]\n",
      "[(7, 7) (7, 7) (7, 7) (7, 7) (7, 7) (7, 7) (7, 7)]\n",
      "[(7, 7) (7, 7) (7, 7) (7, 7) (7, 7) (7, 7) (7, 7)]\n",
      "[(6, 6) (6, 6) (6, 6) (6, 6) (6, 6) (6, 6)]\n",
      "[(6, 6) (6, 6) (6, 6) (6, 6) (6, 6) (6, 6)]\n",
      "[(5, 5) (5, 5) (5, 5) (5, 5) (5, 5)]\n",
      "[(4, 4) (4, 4) (4, 4) (4, 4)]\n",
      "[(3, 3) (3, 3) (3, 3)]\n",
      "[(2, 2) (2, 2)]\n",
      "[(2, 2) (2, 2)]\n",
      "[(1, 1)]\n",
      "[(1, 1)]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[(1, 1)]\n",
      "[(1, 1)]\n",
      "[(2, 2) (2, 2)]\n",
      "[(2, 2) (2, 2)]\n",
      "[(3, 3) (3, 3) (3, 3)]\n"
     ]
    }
   ],
   "source": [
    "for x in range(pointlistarray.shape[0]):\n",
    "    for y in range(pointlistarray.shape[1]):\n",
    "        print(pointlistarray[x,y].data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1ee41ca",
   "metadata": {},
   "source": [
    "<a id='rootnode'></a>\n",
    "\n",
    "\n",
    "## 3.5 `Root`\n",
    "\n",
    "`emd.Root` represents the root node of an `emdfile` tree, and contains no \"Data\" block.  A root node is required to build a tree structure (by calling `.tree(node)`) with `emdfile` objects.  The root node of a tree is accessible from any node in the tree with `.root`.\n",
    "\n",
    "Loaded data will always come with the root, and its metadata, of the EMD tree the loaded data came from.  If a single node is loaded, the node itself is returned and the root can be accessed by calling `.root`.  If a larger tree is loaded, the root node is returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "decd633c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Root( A Node called 'groot', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "root = emd.Root( name='groot' )\n",
    "\n",
    "root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "769e8a40",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I need a root!\n",
      "\n",
      "/\n"
     ]
    }
   ],
   "source": [
    "# Without a root, we can't build a tree\n",
    "\n",
    "node1 = emd.Node( name='nodoubt' )\n",
    "node2 = emd.Node( name='depechenode' )\n",
    "\n",
    "try:\n",
    "    node1.tree(node2)\n",
    "    \n",
    "except AssertionError:\n",
    "    print('I need a root!')\n",
    "    \n",
    "print()\n",
    "node1.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "f8fbc804",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "/\n",
      "|---depechenode\n"
     ]
    }
   ],
   "source": [
    "# With a root, we can\n",
    "\n",
    "root.tree(node1)\n",
    "\n",
    "try:\n",
    "    node1.tree(node2)\n",
    "    \n",
    "except AssertionError:\n",
    "    print('I need a root!')\n",
    "    \n",
    "print()\n",
    "node1.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "02562172",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root( A Node called 'groot', containing the following top-level objects in its tree:\n",
      "\n",
      "          nodoubt                  \t (Node)\n",
      ")\n",
      "Root( A Node called 'groot', containing the following top-level objects in its tree:\n",
      "\n",
      "          nodoubt                  \t (Node)\n",
      ")\n",
      "Root( A Node called 'groot', containing the following top-level objects in its tree:\n",
      "\n",
      "          nodoubt                  \t (Node)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(node1.root)\n",
    "print(node2.root)\n",
    "print(root.root)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba378f11",
   "metadata": {},
   "source": [
    "<a id='customnode'></a>\n",
    "\n",
    "\n",
    "## 3.6 `Custom`\n",
    "\n",
    "`emd.Custom` represents data corresponding to an arbitrary group or tree hierarchies of the data blocks of the other data carrying node types (Array, PointList, and PointListArray).\n",
    "\n",
    "An example implementation can be found in [test_custom_class.py](./test_custom_class.py) and [sample_custom_class_module/](./sample_custom_class_module)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "981f4635",
   "metadata": {},
   "source": [
    "<a id='emdmetadata'></a>\n",
    "\n",
    "\n",
    "## 4 Metadata\n",
    "\n",
    "Each `emdfile` node can store any number of Metadata objects, each of which wraps a Python dictionary of various kinds of data including strings, numbers, booleans, None, arrays, and lists or tuples of the aforementioned types.  A full list of the supported value types are described in the EMD 1.0 file specification.\n",
    "\n",
    "Note that at present, when populating a Metadata instance the class does not check that the value types are read/write supported.  If an unsupported value type is placed in Metadata instance and a save operation is subsequently attempted on it, the save call will fail by throwing a TypeError."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b4b9dbe",
   "metadata": {},
   "source": [
    "<a id='metadataclass'></a>\n",
    "\n",
    "\n",
    "## 4.1 The `Metadata` class\n",
    "\n",
    "`emd.Metadata` represents many (typically small) pieces of data, each of which is accessibly using a string key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "944a92f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Metadata( A Metadata instance called 'i_never_metadata_i_didnt_like', containing the following fields:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a new Metadata instance\n",
    "\n",
    "metadata = emd.Metadata( name='i_never_metadata_i_didnt_like' )\n",
    "\n",
    "metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "185c0ff9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "value\n",
      "42\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# These work like Python dictionaries -\n",
    "# we can slice into them to either get or set items with square brackets and string keys\n",
    "\n",
    "metadata['key'] = 'value'\n",
    "metadata['answer'] = 42\n",
    "metadata['bool'] = True\n",
    "\n",
    "print(metadata['key'])\n",
    "print(metadata['answer'])\n",
    "print(metadata['bool'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "fb9010cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['key', 'answer', 'bool'])"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.keys"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3731e04d",
   "metadata": {},
   "source": [
    "<a id='metadataproperty'></a>\n",
    "\n",
    "\n",
    "## 4.2 The `.metadata` property\n",
    "\n",
    "Each `emdfile` node type has the `.metadata` property, which provides an interface to store any number of Metadata instances with the node, and retrieve them again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "688dbb15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'i_never_metadata_i_didnt_like': Metadata( A Metadata instance called 'i_never_metadata_i_didnt_like', containing the following fields:\n",
       " \n",
       "           key:      value\n",
       "           answer:   42\n",
       "           bool:     True\n",
       " )}"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To add Metadata to a node, assign it to the node.metadata property\n",
    "\n",
    "# Note that this does *not* overwrite the value of `.metadata` - instead, it checks if the value\n",
    "# it's been passed is a Metadata instance, and as long as it is, it adds it to a dictionary where\n",
    "# it can be accessed again using the Metadata instances name as a key\n",
    "\n",
    "node2.metadata = metadata\n",
    "\n",
    "node2.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "14b276ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Metadata( A Metadata instance called 'i_never_metadata_i_didnt_like', containing the following fields:\n",
       "\n",
       "          key:      value\n",
       "          answer:   42\n",
       "          bool:     True\n",
       ")"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "node2.metadata['i_never_metadata_i_didnt_like']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "73ae364c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "no dice!\n"
     ]
    }
   ],
   "source": [
    "# Trying to assign a non-Metadata value to `.metadata` fails\n",
    "\n",
    "try:\n",
    "    node2.metadata = 5\n",
    "except AssertionError:\n",
    "    print('no dice!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "47be7996",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'i_never_metadata_i_didnt_like': Metadata( A Metadata instance called 'i_never_metadata_i_didnt_like', containing the following fields:\n",
       " \n",
       "           key:      value\n",
       "           answer:   42\n",
       "           bool:     True\n",
       " ),\n",
       " 'more_metadata': Metadata( A Metadata instance called 'more_metadata', containing the following fields:\n",
       " \n",
       "           an_array:   2D-array\n",
       "           none:       None\n",
       "           tup:        (1, 2, 3)\n",
       "           list:       ['a', 'b', 'c']\n",
       " )}"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assigning a second Metadata instance to the same node's `.metadata` property will\n",
    "# simply add the new one to a running dictionary, such that both are now stored and accessible\n",
    "\n",
    "\n",
    "# make a new Metadata instance\n",
    "more_metadata = emd.Metadata( name='more_metadata' )\n",
    "\n",
    "# add info to it\n",
    "more_metadata['an_array'] = np.arange(12).reshape((3,4))\n",
    "more_metadata['none'] = None\n",
    "more_metadata['tup'] = (1,2,3)\n",
    "more_metadata['list'] = ['a','b','c']\n",
    "\n",
    "# add it to node2\n",
    "node2.metadata = more_metadata\n",
    "\n",
    "# show the metadata in node2\n",
    "node2.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "ec60e5b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Metadata( A Metadata instance called 'more_metadata', containing the following fields:\n",
       "\n",
       "          an_array:   2D-array\n",
       "          none:       None\n",
       "          tup:        (1, 2, 3)\n",
       "          list:       ['a', 'b', 'c']\n",
       ")"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "node2.metadata['more_metadata']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "7ec76ff1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  2,  3],\n",
       "       [ 4,  5,  6,  7],\n",
       "       [ 8,  9, 10, 11]])"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "node2.metadata['more_metadata']['an_array']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "a629c4ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'i_never_metadata_i_didnt_like': Metadata( A Metadata instance called 'i_never_metadata_i_didnt_like', containing the following fields:\n",
       " \n",
       "           key:      value\n",
       "           answer:   42\n",
       "           bool:     True\n",
       " ),\n",
       " 'more_metadata': Metadata( A Metadata instance called 'more_metadata', containing the following fields:\n",
       " \n",
       "           an_array:   2D-array\n",
       "           none:       None\n",
       "           tup:        (1, 2, 3)\n",
       "           list:       ['a', 'b', 'c']\n",
       " ),\n",
       " 'even_more_metadata': Metadata( A Metadata instance called 'even_more_metadata', containing the following fields:\n",
       " \n",
       "           x:   True\n",
       "           y:   False\n",
       " )}"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can build and populate a new Metadata instance and add it to a node all in one command\n",
    "# by passing a dictionary to the `data` argument in the `Metadata` constructor\n",
    "\n",
    "\n",
    "node2.metadata = emd.Metadata(\n",
    "    name = 'even_more_metadata',\n",
    "    data = {\n",
    "        'x' : True,\n",
    "        'y' : False\n",
    "    }\n",
    ")\n",
    "\n",
    "node2.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "352e8b1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---nodoubt\n",
      "    |---depechenode\n"
     ]
    }
   ],
   "source": [
    "# metadata does not show up in the emd tree - you have to get it from the nodes\n",
    "\n",
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "fe8857d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "root.tree('nodoubt').metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "id": "29c066b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'i_never_metadata_i_didnt_like': Metadata( A Metadata instance called 'i_never_metadata_i_didnt_like', containing the following fields:\n",
       " \n",
       "           key:      value\n",
       "           answer:   42\n",
       "           bool:     True\n",
       " ),\n",
       " 'more_metadata': Metadata( A Metadata instance called 'more_metadata', containing the following fields:\n",
       " \n",
       "           an_array:   2D-array\n",
       "           none:       None\n",
       "           tup:        (1, 2, 3)\n",
       "           list:       ['a', 'b', 'c']\n",
       " ),\n",
       " 'even_more_metadata': Metadata( A Metadata instance called 'even_more_metadata', containing the following fields:\n",
       " \n",
       "           x:   True\n",
       "           y:   False\n",
       " )}"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "root.tree('nodoubt/depechenode').metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "a4f602a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'even_more_metadata': Metadata( A Metadata instance called 'even_more_metadata', containing the following fields:\n",
       " \n",
       "           x:   True\n",
       "           y:   False\n",
       " ),\n",
       " 'i_never_metadata_i_didnt_like': Metadata( A Metadata instance called 'i_never_metadata_i_didnt_like', containing the following fields:\n",
       " \n",
       "           answer:   42\n",
       "           bool:     True\n",
       "           key:      value\n",
       " ),\n",
       " 'more_metadata': Metadata( A Metadata instance called 'more_metadata', containing the following fields:\n",
       " \n",
       "           an_array:   2D-array\n",
       "           list:       ['a', 'b', 'c']\n",
       "           none:       None\n",
       "           tup:        (1, 2, 3)\n",
       " )}"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Metadata is taken along for the ride during write/read\n",
    "\n",
    "clean()\n",
    "emd.save(filepath, root)\n",
    "loaded_data = emd.read(filepath)\n",
    "loaded_data.tree('depechenode').metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fc78ab7",
   "metadata": {},
   "source": [
    "<a id='rootmetadata'></a>\n",
    "\n",
    "\n",
    "## 4.3 Root metadata\n",
    "\n",
    "The root of an EMD tree is always loaded along with nodes being read from that tree, regardless of where they are in the tree.  This means that by default root metadata always stays attached to each piece of data inside that tree as well.  This can be useful if, for instance, some calibrations apply to many related pieces of data all living in one tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "id": "8d8f19b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Metadata( A Metadata instance called 'i_am_root_metadata', containing the following fields:\n",
       "\n",
       "          thelittledog:   laughed\n",
       "          thedish:        thespoon\n",
       ")"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add some root metadata\n",
    "\n",
    "root.metadata = emd.Metadata(\n",
    "    name = 'i_am_root_metadata',\n",
    "    data = {\n",
    "        'thelittledog' : 'laughed',\n",
    "        'thedish' : 'thespoon'\n",
    "    }\n",
    ")\n",
    "\n",
    "root.metadata['i_am_root_metadata']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "id": "ebbb7b8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---nodoubt\n",
      "    |---depechenode\n"
     ]
    }
   ],
   "source": [
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "a71671de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node( A Node called 'depechenode', containing the following top-level objects in its tree:\n",
       "\n",
       ")"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's save the whole tree then load only the last node, 'depechenode'\n",
    "\n",
    "clean()\n",
    "emd.save(filepath, root)\n",
    "loaded_data = emd.read(\n",
    "    filepath,\n",
    "    emdpath = 'groot/nodoubt/depechenode',\n",
    "    tree = False\n",
    ")\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "73cf19ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Root( A Node called 'groot', containing the following top-level objects in its tree:\n",
       "\n",
       "          depechenode              \t (Node)\n",
       ")"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The root comes attached to the node...\n",
    "\n",
    "loaded_data.root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "c45fd23d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'i_am_root_metadata': Metadata( A Metadata instance called 'i_am_root_metadata', containing the following fields:\n",
       " \n",
       "           thedish:        thespoon\n",
       "           thelittledog:   laughed\n",
       " )}"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ...and its metadata comes with it\n",
    "\n",
    "loaded_data.root.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "a10d71dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Metadata( A Metadata instance called 'i_am_root_metadata', containing the following fields:\n",
       "\n",
       "          thedish:        thespoon\n",
       "          thelittledog:   laughed\n",
       ")"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_data.root.metadata['i_am_root_metadata']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "387d80fa",
   "metadata": {},
   "source": [
    "<a id='moretrees'></a>\n",
    "\n",
    "\n",
    "# 5. More trees, branches, cutting, grafting, and appending\n",
    "\n",
    "This section covers the remaining, more advanced functionality not described in the prior sections, including\n",
    "- building, saving, and reading a tree comprised of various data-containing node types ([section 5.1](#buildingtrees))\n",
    "- removing a branch from a tree to create a new tree, or removing a branch from a tree and attaching it somewhere on another tree ([section 5.2](#cutgraft))\n",
    "- appending new data to an EMD tree in an existing EMD 1.0 file ([section 5.3](#append))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59a52af4",
   "metadata": {},
   "source": [
    "<a id='buildingtrees'></a>\n",
    "\n",
    "\n",
    "## 5.1. An example\n",
    "\n",
    "Here we building a tree comprised of various data-containing node types, save it, and load it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "5f28755b",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "0061ac39",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make some data\n",
    "\n",
    "ar1 = emd.Array(\n",
    "    name = 'ar1',\n",
    "    data = np.arange(12).reshape((3,4))\n",
    ")\n",
    "ar2 = emd.Array(\n",
    "    name = 'ar2',\n",
    "    data = np.arange(24).reshape((3,4,2)),\n",
    "    slicelabels = ('a','b')\n",
    ")\n",
    "node = emd.Node(\n",
    "    name = 'immanode'\n",
    ")\n",
    "pointlist1 = emd.PointList(\n",
    "    name = 'pointlist1',\n",
    "    data = np.ones(\n",
    "        5,\n",
    "        dtype = [('rx',int),('ry',int)]\n",
    "    )\n",
    ")\n",
    "pointlist2 = emd.PointList(\n",
    "    name = 'pointlist2',\n",
    "    data = np.zeros(\n",
    "        6,\n",
    "        dtype = [('qx',float),('qy',float)]\n",
    "    )\n",
    ")\n",
    "pointlistarray = emd.PointListArray(\n",
    "    name = 'pointlistarray',\n",
    "    shape = (3,4),\n",
    "    dtype = [('yes',bool),('no',bool)]\n",
    ")\n",
    "for rx in range(pointlistarray.shape[0]):\n",
    "    for ry in range(pointlistarray.shape[1]):\n",
    "        pointlistarray[rx,ry].add(\n",
    "            np.ones(\n",
    "                int(ry + rx*pointlistarray.shape[1]),\n",
    "                dtype = [('yes',bool),('no',bool)]\n",
    "            )\n",
    "        )\n",
    "        \n",
    "# add some metadata\n",
    "pointlist1.metadata = emd.Metadata(\n",
    "    name = 'evolution',\n",
    "    data = {\n",
    "        'pikachu' : 'raichi',\n",
    "        'thunderstone' : True\n",
    "    }\n",
    ")\n",
    "pointlistarray.metadata = emd.Metadata(\n",
    "    name = 'is_rodent',\n",
    "    data = {\n",
    "        'gerbil' : True,\n",
    "        'mouse' : True,\n",
    "        'pikachu' : True,\n",
    "        'bulbasaur' : 'False'\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "id": "e8dd53b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---immanode\n",
      "|   |---pointlistarray\n",
      "|   |   |---pointlist1\n",
      "|   |---ar2\n",
      "|---ar1\n",
      "|---pointlist2\n"
     ]
    }
   ],
   "source": [
    "# Make a tree\n",
    "\n",
    "# start with a Root\n",
    "root = emd.Root( name='worldtree' )\n",
    "\n",
    "# and add data\n",
    "root.tree(node)\n",
    "node.tree(pointlistarray)\n",
    "pointlistarray.tree(pointlist1)\n",
    "root.tree(ar1)\n",
    "root.tree(pointlist2)\n",
    "node.tree(ar2)\n",
    "\n",
    "# show the tree\n",
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "id": "3c3030e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 1988.76it/s]\n"
     ]
    }
   ],
   "source": [
    "# save\n",
    "\n",
    "emd.save(filepath,root)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "id": "bfd5d73b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reading PointListArray: 100%|█████████████████████████████████████| 12/12 [00:00<00:00, 2770.80PointList/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Root( A Node called 'worldtree', containing the following top-level objects in its tree:\n",
       "\n",
       "          ar1                      \t (Array)\n",
       "          immanode                 \t (Node)\n",
       "          pointlist2               \t (PointList)\n",
       ")"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load\n",
    "\n",
    "loaded_data = emd.read(filepath)\n",
    "\n",
    "loaded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "95506311",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---ar1\n",
      "|---immanode\n",
      "|   |---ar2\n",
      "|   |---pointlistarray\n",
      "|       |---pointlist1\n",
      "|---pointlist2\n"
     ]
    }
   ],
   "source": [
    "loaded_data.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "id": "081596d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check that the data is the same\n",
    "\n",
    "assert(np.array_equal( loaded_data.tree('ar1').data, ar1.data ))\n",
    "assert(np.array_equal( loaded_data.tree('immanode/ar2').data, ar2.data ))\n",
    "assert(np.array_equal( loaded_data.tree('immanode/pointlistarray/pointlist1').data, pointlist1.data ))\n",
    "assert(np.array_equal( loaded_data.tree('pointlist2').data, pointlist2.data ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "id": "9701a542",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check that the metadata is the same\n",
    "\n",
    "def check_metadata(obj1, obj2):\n",
    "    \"\"\" asserts equivalence of the metadata in obj1 to obj2. Fails for array-like metadata\n",
    "    \"\"\"\n",
    "    for k in obj1.metadata.keys():\n",
    "        md_i,md_f = obj1.metadata[k],obj2.metadata[k]\n",
    "        for k in md_i.keys:\n",
    "            assert( md_i[k] == md_f[k] )\n",
    "            \n",
    "check_metadata(\n",
    "    loaded_data.tree('immanode/pointlistarray/pointlist1'),\n",
    "    pointlist1\n",
    ")\n",
    "check_metadata(\n",
    "    loaded_data.tree('immanode/pointlistarray'),\n",
    "    pointlistarray\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "id": "cbfd53ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PointList( A length 5 PointList called 'pointlist1',\n",
       "           with 2 fields:\n",
       "\n",
       "           rx   (int64)\n",
       "           ry   (int64)\n",
       ")"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get some node from root\n",
    "\n",
    "data = root.tree('immanode/pointlistarray/pointlist1')\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "id": "acdfd56f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PointList( A length 5 PointList called 'pointlist1',\n",
       "           with 2 fields:\n",
       "\n",
       "           rx   (int64)\n",
       "           ry   (int64)\n",
       ")"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get some node from another, upstream node\n",
    "\n",
    "data = pointlistarray.tree('pointlist1')\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "id": "6144e78c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PointList( A length 5 PointList called 'pointlist1',\n",
       "           with 2 fields:\n",
       "\n",
       "           rx   (int64)\n",
       "           ry   (int64)\n",
       ")"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get some node from another node, using a path referenced to the root\n",
    "\n",
    "data = pointlistarray.tree('/immanode/pointlistarray/pointlist1')\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "124b5263",
   "metadata": {},
   "source": [
    "<a id='cutgraft'></a>\n",
    "\n",
    "\n",
    "## 5.2 Cutting and grafting branches\n",
    "\n",
    "\n",
    "In this section we'll make a new tree and use it to demonstrate\n",
    "- cutting branches off a parent tree to yield some new, smaller tree, and\n",
    "- grafting a branch from one tree to another"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "id": "891c4d19",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make some data\n",
    "\n",
    "ar3 = emd.Array(\n",
    "    name = 'ar3',\n",
    "    data = np.arange(12,22).reshape((5,2))\n",
    ")\n",
    "node2 = emd.Node(\n",
    "    name = 'node2'\n",
    ")\n",
    "node3 = emd.Node(\n",
    "    name = 'node3'\n",
    ")\n",
    "pointlist3 = emd.PointList(\n",
    "    name = 'pointlist3',\n",
    "    data = np.ones(\n",
    "        3,\n",
    "        dtype = [('rx',int),('ry',int)]\n",
    "    )\n",
    ")\n",
    "pointlist4 = emd.PointList(\n",
    "    name = 'pointlist4',\n",
    "    data = np.zeros(\n",
    "        7,\n",
    "        dtype = [('qx',float),('qy',float)]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "id": "e3932645",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node2\n",
      "|   |---ar3\n",
      "|   |---pointlist3\n",
      "|       |---node3\n",
      "|---pointlist4\n"
     ]
    }
   ],
   "source": [
    "# make a tree\n",
    "\n",
    "root2 = emd.Root( name='treeofknowledge')\n",
    "root2.tree(node2)\n",
    "node2.tree(ar3)\n",
    "node2.tree(pointlist3)\n",
    "pointlist3.tree(node3)\n",
    "root2.tree(pointlist4)\n",
    "\n",
    "root2.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "id": "3b88cd94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Root( A Node called 'treeofknowledge_cut_pointlist3', containing the following top-level objects in its tree:\n",
       "\n",
       "          pointlist3               \t (PointList)\n",
       ")"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Cut a branch off of the tree\n",
    "\n",
    "new_root = pointlist3.tree(cut=True)\n",
    "\n",
    "new_root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "id": "5ddbfe34",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node2\n",
      "|   |---ar3\n",
      "|---pointlist4\n",
      "\n",
      "/\n",
      "|---pointlist3\n",
      "    |---node3\n"
     ]
    }
   ],
   "source": [
    "# Show the original tree and cut off branch\n",
    "\n",
    "root2.tree()\n",
    "print()\n",
    "new_root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "id": "dd18c9b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node2\n",
      "|   |---ar3\n",
      "|---pointlist4\n",
      "\n",
      "/\n",
      "|---immanode\n",
      "|   |---pointlistarray\n",
      "|   |   |---pointlist1\n",
      "|   |---ar2\n",
      "|---ar1\n",
      "|---pointlist2\n"
     ]
    }
   ],
   "source": [
    "# Show the two trees\n",
    "root2.tree()\n",
    "print()\n",
    "root.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "id": "be227d36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---pointlist4\n",
      "\n",
      "/\n",
      "|---immanode\n",
      "|   |---pointlistarray\n",
      "|   |   |---pointlist1\n",
      "|   |       |---node2\n",
      "|   |           |---ar3\n",
      "|   |---ar2\n",
      "|---ar1\n",
      "|---pointlist2\n"
     ]
    }
   ],
   "source": [
    "# Graft a branch from root2 at node2 onto root at pointlist1\n",
    "\n",
    "\n",
    "# perform the graft\n",
    "\n",
    "node2.graft(pointlist1)               # Note that these two lines are just different syntax\n",
    "#node2.tree( graft=pointlist1 )         # They perform an identical graft operation\n",
    "\n",
    "\n",
    "# showing the two trees\n",
    "root2.tree()\n",
    "print()\n",
    "root.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31577101",
   "metadata": {},
   "source": [
    "<a id='cutgraft_rootmetadata'></a>\n",
    "\n",
    "\n",
    "### 4.2.1 Cut/graft root metadata options\n",
    "\n",
    "`emdfile` tries to keeps roots and their metadata with all the associated data in it's tree.  When cutting a branch, a new root is created.  When grafting a branch, the root of the target tree becomes the new root of the branch being moved.  However, a question arises regarding the original root's metadata: should this metadata be carried along to the new tree's root, or not?\n",
    "\n",
    "Three options are available.  They are:\n",
    "- no, leave the old root metadata behind\n",
    "- yes, copy the old root metadata to the new root\n",
    "- yes, create a pointer in the new root to the old root metadata\n",
    "\n",
    "In the last case, the new and old roots will both point to the same Metadata instances, and changing values in one will change both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "id": "5ed11049",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make two data trees, each containing some containing root metadata\n",
    "\n",
    "def make_trees():\n",
    "\n",
    "    # roots\n",
    "    root1 = emd.Root( name='root1' )\n",
    "    root2 = emd.Root( name='root2' )\n",
    "\n",
    "    # nodes\n",
    "    node1 = emd.Node( name = 'node1' )\n",
    "    node2 = emd.Node( name = 'node2' )\n",
    "    node3 = emd.Node( name = 'node3' )\n",
    "    node4 = emd.Node( name = 'node4' )\n",
    "\n",
    "    # tree 1\n",
    "    root1.tree(node1)\n",
    "    node1.tree(node2)\n",
    "    node2.tree(node3)\n",
    "    # add root metadata\n",
    "    root1.metadata = emd.Metadata(\n",
    "        name = 'metadata1',\n",
    "        data = {'x':1}\n",
    "    )\n",
    "    root1.metadata = emd.Metadata(\n",
    "        name = 'metadata2',\n",
    "        data = {'y':2}\n",
    "    )\n",
    "    \n",
    "    # tree 2\n",
    "    root2.tree(node4)\n",
    "    # add root metadata\n",
    "    root2.metadata = emd.Metadata(\n",
    "        name = 'metadata3',\n",
    "        data = {'z':3}\n",
    "    )\n",
    "    \n",
    "    return root1,root2\n",
    "\n",
    "root1,root2 = make_trees()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "id": "83f92db7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---node1\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "/\n",
      "|---node4\n"
     ]
    }
   ],
   "source": [
    "# show the trees\n",
    "\n",
    "root1.tree()\n",
    "print()\n",
    "root2.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "id": "b585983f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'metadata1': Metadata( A Metadata instance called 'metadata1', containing the following fields:\n",
      "\n",
      "          x:   1\n",
      "), 'metadata2': Metadata( A Metadata instance called 'metadata2', containing the following fields:\n",
      "\n",
      "          y:   2\n",
      ")}\n",
      "\n",
      "{'metadata3': Metadata( A Metadata instance called 'metadata3', containing the following fields:\n",
      "\n",
      "          z:   3\n",
      ")}\n"
     ]
    }
   ],
   "source": [
    "# show the root metadata\n",
    "\n",
    "print(root1.metadata)\n",
    "print()\n",
    "print(root2.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6ba1e18",
   "metadata": {},
   "source": [
    "<a id='cut_rootmetadata'></a>\n",
    "\n",
    "\n",
    "#### 4.2.1.1 Cutting branches and root metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "id": "b76d2dbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORIGINAL TREE\n",
      "/\n",
      "|---node1\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "TARGET BRANCH\n",
      "/\n",
      "|---node3\n",
      "\n",
      "OLD/NEW TREES\n",
      "/\n",
      "|---node1\n",
      "\n",
      "/\n",
      "|---node2\n",
      "    |---node3\n",
      "\n",
      "NEW ROOT METADATA\n",
      "{'metadata1_copy': Metadata( A Metadata instance called 'metadata1_copy', containing the following fields:\n",
      "\n",
      "          x:   1\n",
      "), 'metadata2_copy': Metadata( A Metadata instance called 'metadata2_copy', containing the following fields:\n",
      "\n",
      "          y:   2\n",
      ")}\n",
      "\n",
      "COMPARE OLD/NEW ROOT METADATA\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# Case 1: cut a branch, copy root metadata\n",
    "\n",
    "\n",
    "\n",
    "# make new trees\n",
    "root1,_ = make_trees()\n",
    "\n",
    "# show the original trees\n",
    "print(\"ORIGINAL TREE\")\n",
    "root1.tree()\n",
    "\n",
    "# get the target node\n",
    "target_node = root1.tree('node1/node2')\n",
    "\n",
    "# show the tree under this node\n",
    "print()\n",
    "print(\"TARGET BRANCH\")\n",
    "target_node.tree()\n",
    "\n",
    "# cut off the branch\n",
    "new_root = target_node.tree( cut='copy' )   # specify copying the root metadata\n",
    "\n",
    "\n",
    "\n",
    "# Show results\n",
    "\n",
    "# show the old and new trees\n",
    "print()\n",
    "print(\"OLD/NEW TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "new_root.tree()\n",
    "\n",
    "# show the metadata in the new tree\n",
    "print()\n",
    "print(\"NEW ROOT METADATA\")\n",
    "print(new_root.metadata)\n",
    "\n",
    "# check if the new tree's Metadata is the same or different objects\n",
    "# from the old tree's Metadata of the same name\n",
    "print()\n",
    "print(\"COMPARE OLD/NEW ROOT METADATA\")\n",
    "print(root1.metadata['metadata1'] is new_root.metadata['metadata1_copy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "id": "32ebab88",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORIGINAL TREE\n",
      "/\n",
      "|---node1\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "TARGET BRANCH\n",
      "/\n",
      "|---node3\n",
      "\n",
      "OLD/NEW TREES\n",
      "/\n",
      "|---node1\n",
      "\n",
      "/\n",
      "|---node2\n",
      "    |---node3\n",
      "\n",
      "NEW ROOT METADATA\n",
      "{'metadata1': Metadata( A Metadata instance called 'metadata1', containing the following fields:\n",
      "\n",
      "          x:   1\n",
      "), 'metadata2': Metadata( A Metadata instance called 'metadata2', containing the following fields:\n",
      "\n",
      "          y:   2\n",
      ")}\n",
      "\n",
      "COMPARE OLD/NEW ROOT METADATA\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Case 2: cut a branch, add pointers to root metadata\n",
    "\n",
    "\n",
    "\n",
    "# make new trees\n",
    "root1,_ = make_trees()\n",
    "\n",
    "# show the original trees\n",
    "print(\"ORIGINAL TREE\")\n",
    "root1.tree()\n",
    "\n",
    "# get the target node\n",
    "target_node = root1.tree('node1/node2')\n",
    "\n",
    "# show the tree under this node\n",
    "print()\n",
    "print(\"TARGET BRANCH\")\n",
    "target_node.tree()\n",
    "\n",
    "# cut off the branch\n",
    "new_root = target_node.tree( cut=True )   # specify adding pointers to the root metadata\n",
    "\n",
    "\n",
    "\n",
    "# Show results\n",
    "\n",
    "# show the old and new trees\n",
    "print()\n",
    "print(\"OLD/NEW TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "new_root.tree()\n",
    "\n",
    "# show the metadata in the new tree\n",
    "print()\n",
    "print(\"NEW ROOT METADATA\")\n",
    "print(new_root.metadata)\n",
    "\n",
    "# check if the new tree's Metadata is the same or different objects\n",
    "# from the old tree's Metadata of the same name\n",
    "print()\n",
    "print(\"COMPARE OLD/NEW ROOT METADATA\")\n",
    "print(root1.metadata['metadata1'] is new_root.metadata['metadata1'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "id": "7b05075e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORIGINAL TREE\n",
      "/\n",
      "|---node1\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "TARGET BRANCH\n",
      "/\n",
      "|---node3\n",
      "\n",
      "OLD/NEW TREES\n",
      "/\n",
      "|---node1\n",
      "\n",
      "/\n",
      "|---node2\n",
      "    |---node3\n",
      "\n",
      "NEW ROOT METADATA\n",
      "{}\n"
     ]
    }
   ],
   "source": [
    "# Case 3: cut a branch, leave root metadata behind\n",
    "\n",
    "\n",
    "\n",
    "# make new trees\n",
    "root1,_ = make_trees()\n",
    "\n",
    "# show the original trees\n",
    "print(\"ORIGINAL TREE\")\n",
    "root1.tree()\n",
    "\n",
    "# get the target node\n",
    "target_node = root1.tree('node1/node2')\n",
    "\n",
    "# show the tree under this node\n",
    "print()\n",
    "print(\"TARGET BRANCH\")\n",
    "target_node.tree()\n",
    "\n",
    "# cut off the branch\n",
    "new_root = target_node.tree( cut=False )   # specify *not* carrying the root metadata to the new tree\n",
    "\n",
    "\n",
    "\n",
    "# Show results\n",
    "\n",
    "# show the old and new trees\n",
    "print()\n",
    "print(\"OLD/NEW TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "new_root.tree()\n",
    "\n",
    "# show the metadata in the new tree\n",
    "print()\n",
    "print(\"NEW ROOT METADATA\")\n",
    "print(new_root.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce4b22b9",
   "metadata": {},
   "source": [
    "<a id='graft_rootmetadata'></a>\n",
    "\n",
    "\n",
    "#### 4.2.1.2 Grafting branches and root metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "id": "d813dbac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORIGINAL TREES\n",
      "/\n",
      "|---node1\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "/\n",
      "|---node4\n",
      "\n",
      "SOURCE AND TARGET BRANCHES\n",
      "/\n",
      "|---node3\n",
      "/\n",
      "\n",
      "OLD/NEW TREES\n",
      "/\n",
      "|---node1\n",
      "\n",
      "/\n",
      "|---node4\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "NEW ROOT METADATA\n",
      "{'metadata3': Metadata( A Metadata instance called 'metadata3', containing the following fields:\n",
      "\n",
      "          z:   3\n",
      "), 'metadata1_copy': Metadata( A Metadata instance called 'metadata1_copy', containing the following fields:\n",
      "\n",
      "          x:   1\n",
      "), 'metadata2_copy': Metadata( A Metadata instance called 'metadata2_copy', containing the following fields:\n",
      "\n",
      "          y:   2\n",
      ")}\n",
      "\n",
      "COMPARE OLD/NEW ROOT METADATA\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# Case 1: graft a branch, copy root metadata\n",
    "\n",
    "\n",
    "\n",
    "# make new trees\n",
    "root1,root2 = make_trees()\n",
    "\n",
    "# show the original trees\n",
    "print(\"ORIGINAL TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "root2.tree()\n",
    "\n",
    "# get the souce and target nodes\n",
    "source_node = root1.tree('node1/node2')\n",
    "target_node = root2.tree('node4')\n",
    "\n",
    "# show the tree under these nodes\n",
    "print()\n",
    "print(\"SOURCE AND TARGET BRANCHES\")\n",
    "source_node.tree()\n",
    "target_node.tree()\n",
    "\n",
    "# perform the graft\n",
    "source_node.tree(graft = (target_node,'copy'))   # This syntax is a little clunky :|\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Show results\n",
    "\n",
    "# show the old and new trees\n",
    "print()\n",
    "print(\"OLD/NEW TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "root2.tree()\n",
    "\n",
    "# show the metadata in the new tree\n",
    "print()\n",
    "print(\"NEW ROOT METADATA\")\n",
    "print(root2.metadata)\n",
    "\n",
    "# check if the new tree's Metadata is the same or different objects\n",
    "# from the old tree's Metadata of the same name\n",
    "print()\n",
    "print(\"COMPARE OLD/NEW ROOT METADATA\")\n",
    "print(root1.metadata['metadata1'] is root2.metadata['metadata1_copy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "id": "0a589907",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORIGINAL TREES\n",
      "/\n",
      "|---node1\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "/\n",
      "|---node4\n",
      "\n",
      "SOURCE AND TARGET BRANCHES\n",
      "/\n",
      "|---node3\n",
      "/\n",
      "\n",
      "OLD/NEW TREES\n",
      "/\n",
      "|---node1\n",
      "\n",
      "/\n",
      "|---node4\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "NEW ROOT METADATA\n",
      "{'metadata3': Metadata( A Metadata instance called 'metadata3', containing the following fields:\n",
      "\n",
      "          z:   3\n",
      "), 'metadata1': Metadata( A Metadata instance called 'metadata1', containing the following fields:\n",
      "\n",
      "          x:   1\n",
      "), 'metadata2': Metadata( A Metadata instance called 'metadata2', containing the following fields:\n",
      "\n",
      "          y:   2\n",
      ")}\n",
      "\n",
      "COMPARE OLD/NEW ROOT METADATA\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Case 2: graft a branch, add pointers to root metadata\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# make new trees\n",
    "root1,root2 = make_trees()\n",
    "\n",
    "# show the original trees\n",
    "print(\"ORIGINAL TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "root2.tree()\n",
    "\n",
    "# get the souce and target nodes\n",
    "source_node = root1.tree('node1/node2')\n",
    "target_node = root2.tree('node4')\n",
    "\n",
    "# show the tree under these nodes\n",
    "print()\n",
    "print(\"SOURCE AND TARGET BRANCHES\")\n",
    "source_node.tree()\n",
    "target_node.tree()\n",
    "\n",
    "# perform the graft\n",
    "source_node.tree(graft = (target_node,True))   # This syntax is a little clunky :|\n",
    "#source_node.tree(graft = target_node)          # This is equivalent to the above line\n",
    "                                               # (i.e. this is the default behavior)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Show results\n",
    "\n",
    "# show the old and new trees\n",
    "print()\n",
    "print(\"OLD/NEW TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "root2.tree()\n",
    "\n",
    "# show the metadata in the new tree\n",
    "print()\n",
    "print(\"NEW ROOT METADATA\")\n",
    "print(root2.metadata)\n",
    "\n",
    "# check if the new tree's Metadata is the same or different objects\n",
    "# from the old tree's Metadata of the same name\n",
    "print()\n",
    "print(\"COMPARE OLD/NEW ROOT METADATA\")\n",
    "print(root1.metadata['metadata1'] is root2.metadata['metadata1'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "e3d4b4a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORIGINAL TREES\n",
      "/\n",
      "|---node1\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "/\n",
      "|---node4\n",
      "\n",
      "SOURCE AND TARGET BRANCHES\n",
      "/\n",
      "|---node3\n",
      "/\n",
      "\n",
      "OLD/NEW TREES\n",
      "/\n",
      "|---node1\n",
      "\n",
      "/\n",
      "|---node4\n",
      "    |---node2\n",
      "        |---node3\n",
      "\n",
      "NEW ROOT METADATA\n",
      "{'metadata3': Metadata( A Metadata instance called 'metadata3', containing the following fields:\n",
      "\n",
      "          z:   3\n",
      ")}\n"
     ]
    }
   ],
   "source": [
    "# Case 3: graft a branch, leave root metadata behind\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# make new trees\n",
    "root1,root2 = make_trees()\n",
    "\n",
    "# show the original trees\n",
    "print(\"ORIGINAL TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "root2.tree()\n",
    "\n",
    "# get the souce and target nodes\n",
    "source_node = root1.tree('node1/node2')\n",
    "target_node = root2.tree('node4')\n",
    "\n",
    "# show the tree under these nodes\n",
    "print()\n",
    "print(\"SOURCE AND TARGET BRANCHES\")\n",
    "source_node.tree()\n",
    "target_node.tree()\n",
    "\n",
    "# perform the graft\n",
    "source_node.tree(graft = (target_node,False))   # This syntax is a little clunky :|\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Show results\n",
    "\n",
    "# show the old and new trees\n",
    "print()\n",
    "print(\"OLD/NEW TREES\")\n",
    "root1.tree()\n",
    "print()\n",
    "root2.tree()\n",
    "\n",
    "# show the metadata in the new tree\n",
    "print()\n",
    "print(\"NEW ROOT METADATA\")\n",
    "print(root2.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e93da54d",
   "metadata": {},
   "source": [
    "<a id='overwrite'></a>\n",
    "\n",
    "\n",
    "## 5.3 Overwrite mode\n",
    "\n",
    "Attempting to save to a path where a file already exists will throw an error, unless overwrite mode is invoked, in which case the existing file is deleted and the new file written in its place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "id": "c2d1cc13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove files currently at `filepath` \n",
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "id": "ec560532",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make new trees\n",
    "root1,root2 = make_trees()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "id": "c2bac31c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---root1\n",
      "    |---node1\n",
      "        |---node2\n",
      "            |---node3\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# write a new file\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1\n",
    ")\n",
    "\n",
    "# examine the file\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "id": "b3e8a47e",
   "metadata": {},
   "outputs": [
    {
     "ename": "AssertionError",
     "evalue": "A file already exists at this destination; use append or overwrite mode, or choose a new file path.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[186], line 4\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# attempt to write to the same path without overwrite mode,\u001b[39;00m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;66;03m# raising an error\u001b[39;00m\n\u001b[0;32m----> 4\u001b[0m \u001b[43memd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msave\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m      5\u001b[0m \u001b[43m    \u001b[49m\u001b[43mfilepath\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m      6\u001b[0m \u001b[43m    \u001b[49m\u001b[43mroot2\u001b[49m\n\u001b[1;32m      7\u001b[0m \u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/work/ncem_projects/self/emd/emdfile/src/emdfile/write.py:152\u001b[0m, in \u001b[0;36mwrite\u001b[0;34m(filepath, data, mode, emdpath, tree)\u001b[0m\n\u001b[1;32m    150\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m(tree \u001b[38;5;129;01min\u001b[39;00m (\u001b[38;5;28;01mTrue\u001b[39;00m,\u001b[38;5;28;01mFalse\u001b[39;00m,\u001b[38;5;28;01mNone\u001b[39;00m)), \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124minvalid value \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mtree\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m passed for `tree`\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    151\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m mode \u001b[38;5;129;01min\u001b[39;00m writemode:\n\u001b[0;32m--> 152\u001b[0m     \u001b[38;5;28;01massert\u001b[39;00m(\u001b[38;5;129;01mnot\u001b[39;00m(exists(filepath))), \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mA file already exists at this destination; use append or overwrite mode, or choose a new file path.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    155\u001b[0m \u001b[38;5;66;03m# validate `data` inputs, and\u001b[39;00m\n\u001b[1;32m    156\u001b[0m \u001b[38;5;66;03m# handle non-Node `data` inputs\u001b[39;00m\n\u001b[1;32m    157\u001b[0m \n\u001b[1;32m    158\u001b[0m \u001b[38;5;66;03m# numpy array -> Array\u001b[39;00m\n\u001b[1;32m    159\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(data, np\u001b[38;5;241m.\u001b[39mndarray):\n",
      "\u001b[0;31mAssertionError\u001b[0m: A file already exists at this destination; use append or overwrite mode, or choose a new file path."
     ]
    }
   ],
   "source": [
    "# attempt to write to the same path without overwrite mode,\n",
    "# raising an error\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7150f023",
   "metadata": {},
   "outputs": [],
   "source": [
    "# write the new tree to the same filepath, using overwrite mode\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root2,\n",
    "    mode = 'o'\n",
    "    #mode = 'overwrite'    # identical to the line above\n",
    ")\n",
    "\n",
    "# examine the file\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f98b08f7",
   "metadata": {},
   "source": [
    "<a id='append'></a>\n",
    "\n",
    "\n",
    "## 5.4 Appending to EMD files\n",
    "\n",
    "\n",
    "The simplest case of file appending is adding a new tree to an existing EMD file.\n",
    "\n",
    "We can also append new nodes or branches to existing trees in EMD files.  In the case of appending an entire branch to an existing tree, the runtime branch may be entirely new nodes, in which case no conflict resolution is necessary.  Alternatively, the new branch may contain some new nodes and some nodes already present in the target H5 branch.  In this case we can either skip or overwrite the conflicting nodes.\n",
    "\n",
    "When appending branches to existing trees, the root metadata in the new branch may not match the root metadata of the existing EMD tree.  New root metadata is added to the existing root.  In the case of conflicting root metadata, we can either ignore the new root metadata, or overwrite the old root metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "id": "e62e4b59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---x\n",
      "    |---y\n",
      "    |---z\n",
      "        |---w\n",
      "\n",
      "/\n",
      "|---alpha\n",
      "    |---beta\n"
     ]
    }
   ],
   "source": [
    "# Define some trees for append examples\n",
    "\n",
    "def make_append_trees():\n",
    "    \n",
    "    # tree 1\n",
    "    root1 = emd.Root( name = 'tree1' )\n",
    "    x1 = emd.Array( name='x', data=np.full((2,2),1) )\n",
    "    y1 = emd.Array( name='y', data=np.full((2,2),1) )\n",
    "    z1 = emd.Array( name='z', data=np.full((2,2),1) )\n",
    "    w1 = emd.Array( name='w', data=np.full((2,2),1) )\n",
    "    root1.tree(x1)\n",
    "    x1.tree(y1)\n",
    "    x1.tree(z1)\n",
    "    z1.tree(w1)\n",
    "    \n",
    "    # tree 2\n",
    "    root2 = emd.Root( name = 'tree2' )\n",
    "    alpha = emd.Node( name='alpha' )\n",
    "    beta = emd.Node( name='beta' )\n",
    "    root2.tree(alpha)\n",
    "    alpha.tree(beta)\n",
    "    \n",
    "    return root1,root2\n",
    "\n",
    "\n",
    "# make the trees\n",
    "root1,root2 = make_append_trees()\n",
    "\n",
    "# show the trees\n",
    "root1.tree()\n",
    "print()\n",
    "root2.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "718f3c7b",
   "metadata": {},
   "source": [
    "<a id='appendtree'></a>\n",
    "\n",
    "\n",
    "### 5.4.1 Append a new tree\n",
    "\n",
    "If we try to append a tree to an existing EMD file, and no tree with this name (i.e. the root name) exists, a new tree will be added to the tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "id": "913a36aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "b845de25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---y\n",
      "        |---z\n",
      "            |---w\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Write a new file with only tree1\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "id": "719b19a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "|   |---x\n",
      "|       |---y\n",
      "|       |---z\n",
      "|           |---w\n",
      "|---tree2\n",
      "    |---alpha\n",
      "        |---beta\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Append tree2\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root2,\n",
    "    mode = 'a'\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "439bf73b",
   "metadata": {},
   "source": [
    "<a id='appendbranch'></a>\n",
    "\n",
    "\n",
    "### 5.4.2 Append from a runtime tree to a different EMD tree\n",
    "\n",
    "The `emdpath` argument specifies a target node inside the existing EMD file that we'd like to append onto.  If we pass as data a runtime tree which does not match the existing EMD tree, and we specify a node in the existing tree with `emdpath`, the data is appended onto the target node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "id": "93c97927",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "id": "fce15bef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---y\n",
      "        |---z\n",
      "            |---w\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Write a new file\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "id": "0c59012d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---alpha\n",
      "    |---beta\n"
     ]
    }
   ],
   "source": [
    "# Show the runtime tree we want to append\n",
    "\n",
    "root2.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "id": "544e1de3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---y\n",
      "        |   |---alpha\n",
      "        |       |---beta\n",
      "        |---z\n",
      "            |---w\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Append to a target node in an existing EMD tree\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root2,\n",
    "    #root2.tree('alpha'),\n",
    "    mode = 'a',\n",
    "    emdpath = '/tree1/x/y',\n",
    "    #tree = False\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d399bdef",
   "metadata": {},
   "source": [
    "<a id='append_diffmerge'></a>\n",
    "\n",
    "\n",
    "### 5.4.3 Merge a runtime tree and an EMD tree\n",
    "\n",
    "If we pass a runtime tree with the same name as an existing EMD tree, append mode will attempt to match the runtime tree to the existing EMD tree, then performs a diffmerge - that is, it looks for nodes in the source (runtime) tree which are not present in the target (EMD) tree, and adds them.\n",
    "\n",
    "If `emdpath` is set, then only nodes at or past the specified target node will be modified in the diffmerge.  New nodes in the runtime tree which are not at or downstream of the target node (e.g. exist on some other branch) will not be written.  Similarly, if the node passed as data is not the root node, only data at or downstream of this source node will be included in the append.\n",
    "\n",
    "If \"appendover\" mode has been set, it will also overwrite any nodes present in both the source and target trees.  Note that overwriting in this way unlinks the overwritten data but **does not** free the associated storage space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "47149a25",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "id": "4faed462",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---x\n",
      "    |---y\n",
      "    |---z\n",
      "        |---w\n"
     ]
    }
   ],
   "source": [
    "# make a tree\n",
    "root1,_ = make_append_trees()\n",
    "\n",
    "# show the tree\n",
    "root1.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "id": "d61891fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---y\n",
      "        |---z\n",
      "            |---w\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Write a new file\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "id": "769a4f1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---x\n",
      "|   |---y\n",
      "|   |   |---gamma\n",
      "|   |---z\n",
      "|   |   |---w\n",
      "|   |       |---delta\n",
      "|   |---beta\n",
      "|---alpha\n"
     ]
    }
   ],
   "source": [
    "# Modify the tree\n",
    "\n",
    "alpha = emd.Node( name='alpha' )\n",
    "beta = emd.Node( name='beta' )\n",
    "gamma = emd.Node( name='gamma' )\n",
    "delta = emd.Node( name='delta' )\n",
    "\n",
    "root1.tree(alpha)\n",
    "root1.tree('x').tree(beta)\n",
    "root1.tree('x/y').tree(gamma)\n",
    "root1.tree('x/z/w').tree(delta)\n",
    "\n",
    "root1.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "id": "107783d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---alpha\n",
      "    |---x\n",
      "        |---beta\n",
      "        |---y\n",
      "        |   |---gamma\n",
      "        |---z\n",
      "            |---w\n",
      "                |---delta\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Append\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1,\n",
    "    mode = 'a'\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "id": "c832671d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---y\n",
      "        |---z\n",
      "            |---w\n",
      "\n",
      "\n",
      "/\n",
      "|---x\n",
      "|   |---y\n",
      "|   |   |---gamma\n",
      "|   |---z\n",
      "|   |   |---w\n",
      "|   |       |---delta\n",
      "|   |---beta\n",
      "|---alpha\n"
     ]
    }
   ],
   "source": [
    "# Do the same, but this time specify an `emdpath`\n",
    "\n",
    "clean()\n",
    "\n",
    "# make a tree\n",
    "root1,_ = make_append_trees()\n",
    "\n",
    "# Write a new file\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1\n",
    ")\n",
    "\n",
    "# Modify the tree\n",
    "alpha = emd.Node( name='alpha' )\n",
    "beta = emd.Node( name='beta' )\n",
    "gamma = emd.Node( name='gamma' )\n",
    "delta = emd.Node( name='delta' )\n",
    "\n",
    "root1.tree(alpha)\n",
    "root1.tree('x').tree(beta)\n",
    "root1.tree('x/y').tree(gamma)\n",
    "root1.tree('x/z/w').tree(delta)\n",
    "\n",
    "\n",
    "# Show the current state of the EMD file and the runtime tree\n",
    "emd.print_h5_tree(filepath)\n",
    "root1.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "id": "37378a4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---beta\n",
      "        |---y\n",
      "        |   |---gamma\n",
      "        |---z\n",
      "            |---w\n",
      "                |---delta\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Append\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1,\n",
    "    mode = 'a',\n",
    "    emdpath = 'tree1/x'\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "id": "e723c8b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---y\n",
      "        |---z\n",
      "            |---w\n",
      "\n",
      "\n",
      "/\n",
      "|---x\n",
      "|   |---y\n",
      "|   |   |---gamma\n",
      "|   |---z\n",
      "|   |   |---w\n",
      "|   |       |---delta\n",
      "|   |---beta\n",
      "|---alpha\n"
     ]
    }
   ],
   "source": [
    "# Do the same, but this time specify an `emdpath` and use a non-root runtime node\n",
    "\n",
    "clean()\n",
    "\n",
    "# make a tree\n",
    "root1,_ = make_append_trees()\n",
    "\n",
    "# Write a new file\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1\n",
    ")\n",
    "\n",
    "# Modify the tree\n",
    "alpha = emd.Node( name='alpha' )\n",
    "beta = emd.Node( name='beta' )\n",
    "gamma = emd.Node( name='gamma' )\n",
    "delta = emd.Node( name='delta' )\n",
    "\n",
    "root1.tree(alpha)\n",
    "root1.tree('x').tree(beta)\n",
    "root1.tree('x/y').tree(gamma)\n",
    "root1.tree('x/z/w').tree(delta)\n",
    "\n",
    "\n",
    "# Show the current state of the EMD file and the runtime tree\n",
    "emd.print_h5_tree(filepath)\n",
    "root1.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "id": "2c146212",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---beta\n",
      "        |---y\n",
      "        |   |---gamma\n",
      "        |---z\n",
      "            |---w\n",
      "                |---delta\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Append\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1.tree('x'),\n",
    "    mode = 'a',\n",
    "    emdpath = 'tree1/x'\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b73ed40",
   "metadata": {},
   "source": [
    "<a id='appendover'></a>\n",
    "\n",
    "\n",
    "### 5.4.4 Append from a runtime tree to the same EMD tree\n",
    "\n",
    "If we pass a runtime tree with the same name as an existing EMD tree, append mode will attempt to match the runtime tree to the existing EMD tree, then performs a diffmerge - that is, it looks for nodes in the source (runtime) tree which are not present in the target (EMD) tree, and adds them.\n",
    "\n",
    "If `emdpath` is set, then only nodes at or past the specified target node will be modified in the diffmerge.  New nodes in the runtime tree which are not at or downstream of the target node (e.g. exist on some other branch) will not be written.  Similarly, if the node passed as data is not the root node, only data at or downstream of this source node will be included in the append.\n",
    "\n",
    "If \"appendover\" mode has been set, it will also overwrite any nodes present in both the source and target trees.  Note that overwriting in this way unlinks the overwritten data but **does not** free the associated storage space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "id": "18886512",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "id": "ac27582c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---x\n",
      "    |---y\n",
      "    |---z\n",
      "        |---w\n"
     ]
    }
   ],
   "source": [
    "# make a tree\n",
    "root1,_ = make_append_trees()\n",
    "\n",
    "# show the tree\n",
    "root1.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "id": "1c2d4667",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---y\n",
      "        |---z\n",
      "            |---w\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Write a new file\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "id": "6a8bd9e0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 1]\n",
      " [1 1]]\n",
      "\n",
      "[[1 1]\n",
      " [1 1]]\n",
      "\n",
      "[[1 1]\n",
      " [1 1]]\n",
      "\n",
      "[[1 1]\n",
      " [1 1]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Currently all the nodes store the array ([[1,1],[1,1]])\n",
    "\n",
    "print(root1.tree('x').data)\n",
    "print()\n",
    "print(root1.tree('x/y').data)\n",
    "print()\n",
    "print(root1.tree('x/z').data)\n",
    "print()\n",
    "print(root1.tree('x/z/w').data)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "id": "07baadbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---x\n",
      "|   |---y\n",
      "|   |   |---gamma\n",
      "|   |---z\n",
      "|   |   |---w\n",
      "|   |       |---delta\n",
      "|   |---beta\n",
      "|---alpha\n"
     ]
    }
   ],
   "source": [
    "# Modify the tree\n",
    "\n",
    "# Here we both add new nodes, AND modify some existing nodes\n",
    "\n",
    "\n",
    "# new nodes\n",
    "alpha = emd.Node( name='alpha' )\n",
    "beta = emd.Node( name='beta' )\n",
    "gamma = emd.Node( name='gamma' )\n",
    "delta = emd.Node( name='delta' )\n",
    "root1.tree(alpha)\n",
    "root1.tree('x').tree(beta)\n",
    "root1.tree('x/y').tree(gamma)\n",
    "root1.tree('x/z/w').tree(delta)\n",
    "\n",
    "# modify existing nodes\n",
    "root1.tree('x').data = np.full((2,2),2)\n",
    "root1.tree('x/y').data = np.full((2,2),2)\n",
    "root1.tree('x/z').data = np.full((2,2),2)\n",
    "root1.tree('x/z/w').data = np.full((2,2),2)\n",
    "\n",
    "root1.tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "id": "4058973d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2 2]\n",
      " [2 2]]\n",
      "\n",
      "[[2 2]\n",
      " [2 2]]\n",
      "\n",
      "[[2 2]\n",
      " [2 2]]\n",
      "\n",
      "[[2 2]\n",
      " [2 2]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now we have new nodes, and all the original nodes store the array ([[2,2],[2,2]])\n",
    "\n",
    "print(root1.tree('x').data)\n",
    "print()\n",
    "print(root1.tree('x/y').data)\n",
    "print()\n",
    "print(root1.tree('x/z').data)\n",
    "print()\n",
    "print(root1.tree('x/z/w').data)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "id": "51fd8faf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---tree1\n",
      "    |---x\n",
      "        |---y\n",
      "        |   |---gamma\n",
      "        |---z\n",
      "            |---w\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Append and overwrite nodes\n",
    "\n",
    "emd.save(\n",
    "    filepath,\n",
    "    root1.tree('x/y'),\n",
    "    mode = 'ao',           # appendover mode\n",
    "    emdpath = 'tree1/x/'\n",
    ")\n",
    "\n",
    "# show\n",
    "emd.print_h5_tree(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "id": "ad939955",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/\n",
      "|---x\n",
      "    |---y\n",
      "    |   |---gamma\n",
      "    |---z\n",
      "        |---w\n",
      "[[1 1]\n",
      " [1 1]]\n",
      "\n",
      "[[2 2]\n",
      " [2 2]]\n",
      "\n",
      "[[1 1]\n",
      " [1 1]]\n",
      "\n",
      "[[1 1]\n",
      " [1 1]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Load the results and check which nodes were overwritten\n",
    "\n",
    "loaded_data = emd.read(filepath).root\n",
    "\n",
    "loaded_data.tree()\n",
    "\n",
    "print(loaded_data.tree('x').data)\n",
    "print()\n",
    "print(loaded_data.tree('x/y').data)\n",
    "print()\n",
    "print(loaded_data.tree('x/z').data)\n",
    "print()\n",
    "print(loaded_data.tree('x/z/w').data)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa050f72",
   "metadata": {},
   "source": [
    "<a id='appendrootmetadata'></a>\n",
    "\n",
    "\n",
    "### 5.4.5 Appending and root metadata\n",
    "\n",
    "\n",
    "In append operations where an existing tree is being modified, any root metadata in the source tree root which is not in the EMD tree root will be added there.  If appendover mode is on, then any root Metadata instances in the EMD tree root with the same name as Metadata instances in the source tree root will be overwritten."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50eb4921",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}