Awkward-RDataFrame-status-evaluated.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14e4174c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"img/Princeton%20logo.png\" width=\"150\" align=\"right\"/>\n",
    "</div>\n",
    "\n",
    "<div>\n",
    "<img src=\"img/IRIS-HEP%20logo.png\" width=\"120\" align=\"left\"/>\n",
    "</div>\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "# Awkward RDataFrame Status\n",
    "### Ianna Osborne, Jim Pivarski \n",
    "*Princeton University*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff03eaf6",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"img/NSF%20logo.png\" width=\"50\" align=\"left\"/>\n",
    "</div>\n",
    "\n",
    "#### OAC-1836650"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19a4371e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Awkward Array and RDataFrame\n",
    "\n",
    "- Awkward Array and RDataFrame are two very different ways of performing calculations at scale\n",
    "\n",
    "    - [Awkward Array](https://github.com/scikit-hep/awkward) is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms\n",
    "\n",
    "    - [RDataFrame](https://root.cern.ch/doc/master/classROOT_1_1RDataFrame.html) - [ROOT](https://root.cern.ch/doc/master/namespaceROOT.html)'s declarative analysis interface \n",
    "        - supports many input formats\n",
    "\n",
    "- The Awkward-RDataFrame bridge provides users with more flexibility in mixing different packages and languages in their analyses. There are numerous benefits of combining both Python and C++. Physicists can mix analyses using Awkward Arrays, Numba, and ROOT C++ in memory, without saving to disk and without leaving their environment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f16cd97b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Awkward Array and other projects"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d6174a8",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"img/Awkward%20Array%20and%20other%20projects.png\" width=\"800\" align=\"center\"/>\n",
    "</div>\n",
    "\n",
    "- See Jim's talk [Uproot, Awkward Array, hist, Vector: from basics to combinatorics](https://github.com/jpivarski-talks/2022-09-12-pyhep22-awkward-combinatorics/blob/main/pyhep2022-combinatorics.ipynb)\n",
    "- See Saransh's talk [Constructing HEP vectors and analyzing HEP data using Vector](https://github.com/Saransh-cpp/PyHEP22-Constructing-HEP-vectors-and-analyzing-HEP-data-using-Vector)\n",
    "- See Doug's [Dask Tutorial]() \n",
    "- See Kush's [Uproot + Dask]() \n",
    "- [This tutorial](https://github.com/ianna/PyHep2022) covers Awkward Arrays <--> RDataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f05e7a8a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## From Awkward Array to RDataFrame\n",
    "### View\n",
    "* Awkward Arrays are already JIT-compiled with Numba. Here we are reusing some of the Numba implementation for C++: no performance difference\n",
    "* The `ak.to_rdataframe` function presents a view of a Python Awkward Array as an RDataFrame source\n",
    "* This view is generated on demand and the data is not copied\n",
    "* The column readers are generated based on the run-time type of the views\n",
    "* The readers are passed to a generated source derived from ROOT::RDF::RDataSource\n",
    "\n",
    "```python\n",
    "            self.data_ptrs_list.append(self.lookups[key].arrayptrs.ctypes.data)\n",
    "\n",
    "            cpp_code_declare_slots = (\n",
    "                cpp_code_declare_slots\n",
    "                + f\"\"\"\n",
    "        ULong64_t fPtrs_{key} = 0;\n",
    "        std::vector<{self.entry_types[key]}>  slots_{key};\n",
    "        std::vector<{self.entry_types[key]}*> addrs_{key};\n",
    "    \"\"\"\n",
    "            )\n",
    "\n",
    "            cpp_code_define_readers = (\n",
    "                cpp_code_define_readers\n",
    "                + f\"\"\"\n",
    "        if (name == \"{key}\") {{\n",
    "            for (auto i : ROOT::TSeqU(fNSlots)) {{\n",
    "                addrs_{key}[i] = &slots_{key}[i];\n",
    "                reader.emplace_back((void *)(&addrs_{key}[i]));\n",
    "            }}\n",
    "        }}\n",
    "    \"\"\"\n",
    "\n",
    "```\n",
    "\n",
    "### Data Source\n",
    "* Generated AwkwardArray RDataSource takes pointers into the original array data: a 40-byte ArrayView object is allocated on the stack\n",
    "* Array view is a cursor:\n",
    "<div>\n",
    "<img src=\"img/cursor.png\" width=\"700\" align=\"center\"/>\n",
    "</div>\n",
    "\n",
    "\n",
    "* The large-scale array data are not copied\n",
    "* The views are transient, their lifetime is defined by the lifetime of their lookup Python object\n",
    "  * A base class of all Array Views holds a pointer to a lookup Python object\n",
    "\n",
    "```c++\n",
    "  class ArrayView {\n",
    "  public:\n",
    "    ArrayView(ssize_t start, ssize_t stop, ssize_t which, ssize_t* ptrs, PyObject* lookup)\n",
    "      : start_(start), stop_(stop), which_(which), ptrs_(ptrs), lookup_(lookup) {\n",
    "      }\n",
    "\n",
    "    size_t size() const noexcept {\n",
    "      return stop_ - start_;\n",
    "    }\n",
    "\n",
    "    bool empty() const noexcept {\n",
    "      return start_ == stop_;\n",
    "    }\n",
    "\n",
    "    PyObject* lookup() {\n",
    "        Py_INCREF(lookup_);\n",
    "        return lookup_;\n",
    "    }\n",
    "\n",
    "  protected:\n",
    "    ssize_t start_;\n",
    "    ssize_t stop_;\n",
    "    ssize_t which_;\n",
    "    ssize_t* ptrs_;\n",
    "    PyObject* lookup_;\n",
    "  };\n",
    "```\n",
    "\n",
    "* ArrayViews keep a pointer to their lookup Python object to facilitate a no-copy conversion back to an Awkward Array\n",
    "\n",
    "### From Python to C++\n",
    "#### from C++ back to Python and to C++ again\n",
    "* C++ code generated from Python and fed to Cling\n",
    "  * Let's look at an example of a generated `ListArray` view for an `ak.Array([[1.1, 2.2], [3.3], … []])` - e.g. nested `NumpyArray`s.\n",
    "    * Note, the `value_type` is requested to be `ROOT::VecOps::RVec<double>`, though native type is: \n",
    "    \n",
    "    ```c++\n",
    "typedef NumpyArray_float64_IeMUCd8nMJU value_type;\n",
    "    ```\n",
    "\n",
    "```c++\n",
    "namespace awkward {\n",
    "  class ListArray_Cj7Qec8BHPk: public ArrayView {\n",
    "  public:\n",
    "    ListArray_Cj7Qec8BHPk(ssize_t start, ssize_t stop, ssize_t which, ssize_t* ptrs, PyObject* lookup)\n",
    "      : ArrayView(start, stop, which, ptrs, lookup) { }\n",
    "    ListArray_Cj7Qec8BHPk() : ArrayView(0, 0, 0, 0, 0) { }\n",
    "\n",
    "    typedef ROOT::VecOps::RVec<double> value_type;\n",
    "\n",
    "    const std::string parameter(const std::string& parameter) const noexcept {\n",
    "      return \"null\";\n",
    "    }\n",
    "\n",
    "    bool operator==(ListArray_Cj7Qec8BHPk other) const noexcept {\n",
    "      return start_ == other.start_  &&\n",
    "             stop_ == other.stop_  &&\n",
    "             which_ == other.which_  &&\n",
    "             ptrs_ == other.ptrs_;\n",
    "    }\n",
    "\n",
    "    bool operator!=(ListArray_Cj7Qec8BHPk other) const noexcept {\n",
    "      return start_ != other.start_  ||\n",
    "             stop_ != other.stop_  ||\n",
    "             which_ != other.which_  ||\n",
    "             ptrs_ != other.ptrs_;\n",
    "    }\n",
    "\n",
    "    Iterator<ListArray_Cj7Qec8BHPk, value_type> begin() const noexcept {\n",
    "      return Iterator<ListArray_Cj7Qec8BHPk, value_type>(start_, stop_, which_, ptrs_, lookup_);\n",
    "    }\n",
    "\n",
    "    Iterator<ListArray_Cj7Qec8BHPk, value_type> end() const noexcept {\n",
    "      return Iterator<ListArray_Cj7Qec8BHPk, value_type>(stop_, stop_, which_, ptrs_, lookup_);\n",
    "    }\n",
    "\n",
    "    RIterator<ListArray_Cj7Qec8BHPk, value_type> rbegin() const noexcept {\n",
    "      return RIterator<ListArray_Cj7Qec8BHPk, value_type>(stop_ - 1, stop_, which_, ptrs_, lookup_);\n",
    "    }\n",
    "\n",
    "    RIterator<ListArray_Cj7Qec8BHPk, value_type> rend() const noexcept {\n",
    "      return RIterator<ListArray_Cj7Qec8BHPk, value_type>(start_ - 1, stop_, which_, ptrs_, lookup_);\n",
    "    }\n",
    "\n",
    "    value_type operator[](size_t at) const noexcept {\n",
    "      ssize_t start = reinterpret_cast<int64_t*>(ptrs_[which_ + 1])[start_ + at];\n",
    "      ssize_t stop = reinterpret_cast<int64_t*>(ptrs_[which_ + 2])[start_ + at];\n",
    "      ssize_t which = ptrs_[which_ + 3];\n",
    "      double* content = reinterpret_cast<double*>(ptrs_[which + 1]) + start;\n",
    "      return value_type(content, stop - start);\n",
    "    }\n",
    "  };\n",
    "}\n",
    "```\n",
    "\n",
    "* The Awkward Array <--> RDataFrame is a part of Awkward Array version 2, which is currently a master branch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8435cb67",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome to JupyROOT 6.26/06\n"
     ]
    }
   ],
   "source": [
    "import awkward as ak\n",
    "import ROOT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4d4a3a03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.0rc1'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ak.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7b625c80",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "array_x = ak.Array(\n",
    "        [\n",
    "            {\"x\": [1.1, 1.2, 1.3]},\n",
    "            {\"x\": [2.1, 2.2]},\n",
    "            {\"x\": [3.1]},\n",
    "            {\"x\": [4.1, 4.2, 4.3, 4.4]},\n",
    "            {\"x\": [5.1]},\n",
    "        ]\n",
    "    )\n",
    "array_y = ak.Array([1, 2, 3, 4, 5])\n",
    "array_z = ak.Array([[1.1], [2.1, 2.3, 2.4], [3.1], [4.1, 4.2, 4.3], [5.1]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e51f9fc0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: 5 * {\n",
      "    x: var * float64\n",
      "}\n",
      "[{x: [1.1, 1.2, 1.3]},\n",
      " {x: [2.1, 2.2]},\n",
      " {x: [3.1]},\n",
      " {x: [4.1, 4.2, 4.3, 4.4]},\n",
      " {x: [5.1]}]\n"
     ]
    }
   ],
   "source": [
    "array_x.show(type=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d9ae2e11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: 5 * int64\n",
      "[1,\n",
      " 2,\n",
      " 3,\n",
      " 4,\n",
      " 5]\n"
     ]
    }
   ],
   "source": [
    "array_y.show(type=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6b1b7eca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: 5 * var * float64\n",
      "[[1.1],\n",
      " [2.1, 2.3, 2.4],\n",
      " [3.1],\n",
      " [4.1, 4.2, 4.3],\n",
      " [5.1]]\n"
     ]
    }
   ],
   "source": [
    "array_z.show(type=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5d23780",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `ak.to_rdataframe` function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b862e70e",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- The awkward style `ak.to_rdataframe` function requires a dictionary: \n",
    "    - each - unique - key defines a column name in RDataFrame\n",
    "    - the arrays given for each column have to be equal length\n",
    "- There is a small overhead of generating [Awkward RDataSource C++ code](https://github.com/scikit-hep/awkward/blob/main/src/awkward/_v2/_connect/rdataframe/to_rdataframe.py#L152-L252)\n",
    "    - This operation does not execute the RDF event loop\n",
    "    - The array data are not copied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "536ada10",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(array_x) == len(array_y) == len(array_z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a3d074e7",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df = ak.to_rdataframe({\"x\": array_x, \"y\": array_y, \"z\": array_z})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3fd7a5c8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from datasource Custom Datasource\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            4\n",
      "Columns from defines        1\n",
      "Event loops run             0\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "awkward_index_  Long64_t                        Define\n",
      "x               awkward::Record_eTuI5KazfpM     Dataset\n",
      "y               int64_t                         Dataset\n",
      "z               ROOT::VecOps::RVec<double>      Dataset"
     ]
    }
   ],
   "source": [
    "df.Describe().Print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38125ab7",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "* The `x` column contains an Awkward Array with a made-up type; `awkward::Record_cKnX5DyNVM`.\n",
    "\n",
    "* Awkward Arrays are dynamically typed, so in a C++ context, the type name is hashed. In practice, there is no need to know the type. The C++ code should use a placeholder type specifier `auto`. The type of the variable that is being declared will be automatically deduced from its initializer.\n",
    "\n",
    "* The `awkward_index_` column is always defined; it is used internally. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb533371",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Columns type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bec21151",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'awkward::Record_eTuI5KazfpM'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.GetColumnType(\"x\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "eb57e3b2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'int64_t'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.GetColumnType(\"y\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0b9cf26d",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ROOT::VecOps::RVec<double>'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.GetColumnType(\"z\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d5c86d1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Operations on data in RDataFrame\n",
    "* Scheduling a filtering operation does not execute the event loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c9065b0c",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df = df.Filter(\"y > 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b78928d",
   "metadata": {},
   "source": [
    "* Let's check the state of the dataframe to make sure that the event loop was not triggered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f2290c06",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from datasource Custom Datasource\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            4\n",
      "Columns from defines        1\n",
      "Event loops run             0\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "awkward_index_  Long64_t                        Define\n",
      "x               awkward::Record_eTuI5KazfpM     Dataset\n",
      "y               int64_t                         Dataset\n",
      "z               ROOT::VecOps::RVec<double>      Dataset"
     ]
    }
   ],
   "source": [
    "df.Describe().Print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97f701ec",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Retrieve selected columns\n",
    "\n",
    "* The *`ak.from_rdataframe`* function converts selected columns to native Awkward Arrays.\n",
    "\n",
    "* The function takes a tuple of strings that are the RDF column names.\n",
    "\n",
    "* The event loop is triggered once to retrieve all selected columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "68d56865",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "out = ak.from_rdataframe(\n",
    "    df,\n",
    "    columns=(\"x\", \"y\", \"z\",),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "98baf009",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{y: 3, z: [3.1], x: {...}}, ..., {...}] type='3 * {y: int64, z: var...'>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34dcb7be",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Filtered array type\n",
    "Let's check the type of the Awkward Array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "652876e7",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: 3 * {\n",
      "    y: int64,\n",
      "    z: var * float64,\n",
      "    x: {\n",
      "        x: var * float64\n",
      "    }\n",
      "}\n",
      "[{y: 3, z: [3.1], x: {x: [3.1]}},\n",
      " {y: 4, z: [4.1, 4.2, 4.3], x: {x: [4.1, ...]}},\n",
      " {y: 5, z: [5.1], x: {x: [5.1]}}]\n"
     ]
    }
   ],
   "source": [
    "out.show(type=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b575c94",
   "metadata": {},
   "source": [
    "Let's check the filtered entries for `y > 2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "bbd83cf3",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3, 4, 5]\n",
      "[[3.1], [4.1, 4.2, 4.3], [5.1]]\n",
      "[{'x': [3.1]}, {'x': [4.1, 4.2, 4.3, 4.4]}, {'x': [5.1]}]\n"
     ]
    }
   ],
   "source": [
    "print(out[\"y\"].to_list())\n",
    "print(out[\"z\"].to_list())\n",
    "print(out[\"x\"].to_list())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "126cc0d9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Check to make sure we triggered the event loop only once:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fed3aca2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from datasource Custom Datasource\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            4\n",
      "Columns from defines        1\n",
      "Event loops run             1\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "awkward_index_  Long64_t                        Define\n",
      "x               awkward::Record_eTuI5KazfpM     Dataset\n",
      "y               int64_t                         Dataset\n",
      "z               ROOT::VecOps::RVec<double>      Dataset"
     ]
    }
   ],
   "source": [
    "df.Describe().Print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cf01db4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Layout details\n",
    "The `RecordArray` data: its content `NumpyArray` - is not copied, it is indexed: it is wrapped in an `IndexedArray` - because of the filter selection. The other two columns data are copied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d1f1c6bf",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<RecordArray is_tuple='false' len='3'>\n",
       "    <content index='0' field='y'>\n",
       "        <NumpyArray dtype='int64' len='3'>[3 4 5]</NumpyArray>\n",
       "    </content>\n",
       "    <content index='1' field='z'>\n",
       "        <ListOffsetArray len='3'>\n",
       "            <offsets><Index dtype='int64' len='4'>\n",
       "                [0 1 4 5]\n",
       "            </Index></offsets>\n",
       "            <content><NumpyArray dtype='float64' len='5'>[3.1 4.1 4.2 4.3 5.1]</NumpyArray></content>\n",
       "        </ListOffsetArray>\n",
       "    </content>\n",
       "    <content index='2' field='x'>\n",
       "        <IndexedArray len='3'>\n",
       "            <index><Index dtype='int64' len='3'>[2 3 4]</Index></index>\n",
       "            <content><RecordArray is_tuple='false' len='5'>\n",
       "                <content index='0' field='x'>\n",
       "                    <ListArray len='5'>\n",
       "                        <starts><Index dtype='int64' len='5'>\n",
       "                            [ 0  3  5  6 10]\n",
       "                        </Index></starts>\n",
       "                        <stops><Index dtype='int64' len='5'>\n",
       "                            [ 3  5  6 10 11]\n",
       "                        </Index></stops>\n",
       "                        <content><NumpyArray dtype='float64' len='11'>\n",
       "                            [1.1 1.2 1.3 2.1 2.2 3.1 4.1 4.2 4.3 4.4 5.1]\n",
       "                        </NumpyArray></content>\n",
       "                    </ListArray>\n",
       "                </content>\n",
       "            </RecordArray></content>\n",
       "        </IndexedArray>\n",
       "    </content>\n",
       "</RecordArray>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out.layout"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c1ab908",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The same operation on Awkward arrays in Python\n",
    "\n",
    "- Is the array type as expected? Yes.\n",
    "- Is its layout the same? No. Awkward arrays are immutable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1ddd6a8e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: 3 * {\n",
      "    y: int64,\n",
      "    z: var * float64,\n",
      "    x: {\n",
      "        x: var * float64\n",
      "    }\n",
      "}\n",
      "[{y: 3, z: [3.1], x: {x: [3.1]}},\n",
      " {y: 4, z: [4.1, 4.2, 4.3], x: {x: [4.1, ...]}},\n",
      " {y: 5, z: [5.1], x: {x: [5.1]}}]\n"
     ]
    }
   ],
   "source": [
    "array_yzx = ak.Array({\"y\": array_y, \"z\": array_z, \"x\": array_x})\n",
    "filtered_array = array_yzx[array_yzx[\"y\"] > 2]\n",
    "filtered_array.show(type=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2089ed5e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<IndexedArray len='3'>\n",
       "    <index><Index dtype='int64' len='3'>\n",
       "        [2 3 4]\n",
       "    </Index></index>\n",
       "    <content><RecordArray is_tuple='false' len='5'>\n",
       "        <content index='0' field='y'>\n",
       "            <NumpyArray dtype='int64' len='5'>[1 2 3 4 5]</NumpyArray>\n",
       "        </content>\n",
       "        <content index='1' field='z'>\n",
       "            <ListOffsetArray len='5'>\n",
       "                <offsets><Index dtype='int64' len='6'>\n",
       "                    [0 1 4 5 8 9]\n",
       "                </Index></offsets>\n",
       "                <content><NumpyArray dtype='float64' len='9'>\n",
       "                    [1.1 2.1 2.3 2.4 3.1 4.1 4.2 4.3 5.1]\n",
       "                </NumpyArray></content>\n",
       "            </ListOffsetArray>\n",
       "        </content>\n",
       "        <content index='2' field='x'>\n",
       "            <RecordArray is_tuple='false' len='5'>\n",
       "                <content index='0' field='x'>\n",
       "                    <ListOffsetArray len='5'>\n",
       "                        <offsets><Index dtype='int64' len='6'>\n",
       "                            [ 0  3  5  6 10 11]\n",
       "                        </Index></offsets>\n",
       "                        <content><NumpyArray dtype='float64' len='11'>\n",
       "                            [1.1 1.2 1.3 2.1 2.2 3.1 4.1 4.2 4.3 4.4 5.1]\n",
       "                        </NumpyArray></content>\n",
       "                    </ListOffsetArray>\n",
       "                </content>\n",
       "            </RecordArray>\n",
       "        </content>\n",
       "    </RecordArray></content>\n",
       "</IndexedArray>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_array.layout"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80195401",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## From RDataFrame to Awkward Array\n",
    "\n",
    "The *`ak.from_rdataframe`* function converts selected columns to native Awkward Arrays. The function takes a tuple of strings that are the RDF column names and recognizes the following column data types:\n",
    "\n",
    "- Primitive types: `integer`, `float`, `double`, `std::complex<double>`, etc.\n",
    "- Lists of primitive types and the arbitrary depth nested lists of primitive types: `std::vector<double>`, `RVec<int>`, etc.\n",
    "- Awkward types: run-time generated types derived from `awkward::ArrayView` or `awkward::RecordView`\n",
    "  - no copy required because Awkward Arrays are immutable\n",
    "<div>\n",
    "<img src=\"img/Awkward%20types.png\" width=\"600\" align=\"center\"/>\n",
    "</div>\n",
    "\n",
    "### Readers of an arbitrary depth nested data\n",
    "\n",
    "* A type-specialized nested array builder is used\n",
    "  * C++ template Layout builder, a header-only implementation\n",
    "* The function that iterates over the nested data is generated for each column based on the column type:\n",
    "\n",
    "```python\n",
    "    def cpp_builder_type(depth, data_type):\n",
    "        if depth == 1:\n",
    "            return f\"awkward::LayoutBuilder::Numpy<{data_type}>>\"\n",
    "        else:\n",
    "            return (\n",
    "                \"awkward::LayoutBuilder::ListOffset<int64_t, \"\n",
    "                + cpp_builder_type(depth - 1, data_type)\n",
    "                + \">\"\n",
    "            )\n",
    "\n",
    "    def cpp_fill_offsets_and_flatten(depth):\n",
    "        if depth == 1:\n",
    "            return \"\\nfor (auto it : vec1) {\\n\" + \"  builder1.append(it);\\n\" + \"}\\n\"\n",
    "        else:\n",
    "            return (\n",
    "                f\"for (auto const& vec{depth - 1} : vec{depth}) \"\n",
    "                + \"{\\n\"\n",
    "                + f\"  auto& builder{depth - 1} = builder{depth}.begin_list();\\n\"\n",
    "                + \"  \"\n",
    "                + cpp_fill_offsets_and_flatten(depth - 1)\n",
    "                + \"\\n\"\n",
    "                + f\"  builder{depth}.end_list();\\n\"\n",
    "                + \"}\\n\"\n",
    "            )\n",
    "\n",
    "    def cpp_fill_function(depth):\n",
    "        if depth == 1:\n",
    "            return (\n",
    "                \"template<class BUILDER, typename PRIMITIVE>\\n\"\n",
    "                + \"void\\n\"\n",
    "                + \"fill_from(BUILDER& builder, ROOT::RDF::RResultPtr<std::vector<PRIMITIVE>>& result) {\"\n",
    "                + \"  for (auto it : result) {\\n\"\n",
    "                + \"    builder.append(it);\\n\"\n",
    "                + \"  }\\n\"\n",
    "                + \"}\\n\"\n",
    "            )\n",
    "        else:\n",
    "            return (\n",
    "                \"template<class BUILDER, typename PRIMITIVE>\\n\"\n",
    "                + \"void\\n\"\n",
    "                + f\"fill_offsets_and_flatten{depth}(BUILDER& builder{depth}, ROOT::RDF::RResultPtr<std::vector<PRIMITIVE>>& result) \"\n",
    "                + \"{\\n\"\n",
    "                + f\"  for (auto const& vec{depth - 1} : result) \"\n",
    "                + \"{\\n\"\n",
    "                + f\"  auto& builder{depth - 1} = builder{depth}.begin_list();\\n\"\n",
    "                + \"  \"\n",
    "                + cpp_fill_offsets_and_flatten(depth - 1)\n",
    "                + \"\\n\"\n",
    "                + f\"  builder{depth}.end_list();\\n\"\n",
    "                + \"}\\n\"\n",
    "                + \"}\\n\"\n",
    "            )\n",
    "\n",
    "    is_indexed = True if \"awkward_index_\" in data_frame.GetColumnNames() else False\n",
    "```\n",
    "\n",
    "#### Discussion Point\n",
    "* This code can handle any iterable containers\n",
    "  * If we new the *provenance* the Awkward columns converted to `RVecs` we could avoid the copy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "061fa9fd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data analysis: from C++ to Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c1cd262d",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import awkward as ak\n",
    "import ROOT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "943787c4",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df = ROOT.RDataFrame('Events', 'root://eospublic.cern.ch//eos/opendata/cms/derived-data/AOD2NanoAODOutreachTool/Run2012BC_DoubleMuParked_Muons.root')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6172a26",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "CMS data from [CERN Open Data portal DOI:10.7483/OPENDATA.CMS.LVG5.QT81](http://opendata.web.cern.ch/record/12341) \n",
    "- This dataset contains about 60 mio. data events from the CMS detector taken in 2012 during Run B and C.\n",
    "- The original AOD dataset is converted to the NanoAOD format and reduced to the muon collections.\n",
    "- The dataset in the file is called <b>Events</b> and contains the following columns:\n",
    " - <b>nMuon</b>\t`unsigned int`\t*Number of muons in this event*\n",
    " - <b>Muon_pt</b>\t`float[nMuon]`\t*Transverse momentum of the muons (stored as an array of size nMuon)*\n",
    " - <b>Muon_eta</b>\t`float[nMuon]`\t*Pseudorapidity of the muons*\n",
    " - <b>Muon_phi</b>\t`float[nMuon]`\t*Azimuth of the muons*\n",
    " - <b>Muon_mass</b>\t`float[nMuon]`\t*Mass of the muons*\n",
    " - <b>Muon_charge</b>\t`int[nMuon]`\t*Charge of the muons (either 1 or -1)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0661e117",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from TChain Events in file root://eospublic.cern.ch//eos/opendata/cms/derived-data/AOD2NanoAODOutreachTool/Run2012BC_DoubleMuParked_Muons.root\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            6\n",
      "Columns from defines        0\n",
      "Event loops run             0\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "Muon_charge     ROOT::VecOps::RVec<Int_t>       Dataset\n",
      "Muon_eta        ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_mass       ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_phi        ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_pt         ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "nMuon           UInt_t                          Dataset"
     ]
    }
   ],
   "source": [
    "# Describe the state of the dataframe.\n",
    "# Note that this operation is not running the event loop.\n",
    "df.Describe().Print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb5adca0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Build a small analysis studying the invariant mass of dimuon systems.\n",
    "- See [ROOT tutorial](https://root.cern.ch/doc/master/df102__NanoAODDimuonAnalysis_8py.html) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b6098a90",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df = df.Filter('nMuon == 2')\\\n",
    "       .Filter('Muon_charge[0] != Muon_charge[1]')\\\n",
    "       .Define('Dimuon_mass', 'InvariantMass(Muon_pt, Muon_eta, Muon_phi, Muon_mass)')\\\n",
    "       .Filter('Dimuon_mass > 70')\\\n",
    "       .Range(1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bead5e6",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Trigger the event loop by asking for the mean of the dimuon mass:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "52a9922b",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Approximate mass of the Z boson: 91.44 GeV\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('\\nApproximate mass of the Z boson: {:.2f} GeV\\n'.format(\n",
    "        df.Mean('Dimuon_mass').GetValue()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5696c5c6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Check that this operation triggered the event loop once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4e975c37",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from TChain Events in file root://eospublic.cern.ch//eos/opendata/cms/derived-data/AOD2NanoAODOutreachTool/Run2012BC_DoubleMuParked_Muons.root\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            7\n",
      "Columns from defines        1\n",
      "Event loops run             1\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "Dimuon_mass     float                           Define\n",
      "Muon_charge     ROOT::VecOps::RVec<Int_t>       Dataset\n",
      "Muon_eta        ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_mass       ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_phi        ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_pt         ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "nMuon           UInt_t                          Dataset"
     ]
    }
   ],
   "source": [
    "df.Describe().Print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8559fd4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Convert selected columns to Awkward Array\n",
    "\n",
    "- The scheduled analysis executed the event loop once\n",
    "- A user can take the data out as an Awkward Array\n",
    "- If the columns type is not `awkward`, the `ROOT::VecOps::RVec`s content is copied to a Numpy buffer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ba5d09d5",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "array = ak.from_rdataframe(\n",
    "        df,\n",
    "        columns=(\n",
    "            \"Dimuon_mass\",\n",
    "        ),\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8fa8ae46",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{Dimuon_mass: 114}, {...}, ..., {...}] type='1000 * {Dimuon_mass: f...'>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e05bc830",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Array type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8e9adfea",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: 1000 * {\n",
      "    Dimuon_mass: float32\n",
      "}\n",
      "[{Dimuon_mass: 114},\n",
      " {Dimuon_mass: 88.3},\n",
      " {Dimuon_mass: 88.3},\n",
      " {Dimuon_mass: 91.3},\n",
      " {Dimuon_mass: 93.6},\n",
      " {Dimuon_mass: 90.9},\n",
      " {Dimuon_mass: 89.2},\n",
      " {Dimuon_mass: 82.3},\n",
      " {Dimuon_mass: 94.6},\n",
      " {Dimuon_mass: 89.2},\n",
      " ...,\n",
      " {Dimuon_mass: 86.4},\n",
      " {Dimuon_mass: 77.1},\n",
      " {Dimuon_mass: 89.8},\n",
      " {Dimuon_mass: 87.3},\n",
      " {Dimuon_mass: 92},\n",
      " {Dimuon_mass: 70.7},\n",
      " {Dimuon_mass: 89.6},\n",
      " {Dimuon_mass: 91.1},\n",
      " {Dimuon_mass: 90.1}]\n"
     ]
    }
   ],
   "source": [
    "array.show(type=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b588be5e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: 1000 * float32\n",
      "[114,\n",
      " 88.3,\n",
      " 88.3,\n",
      " 91.3,\n",
      " 93.6,\n",
      " 90.9,\n",
      " 89.2,\n",
      " 82.3,\n",
      " 94.6,\n",
      " 89.2,\n",
      " ...,\n",
      " 86.4,\n",
      " 77.1,\n",
      " 89.8,\n",
      " 87.3,\n",
      " 92,\n",
      " 70.7,\n",
      " 89.6,\n",
      " 91.1,\n",
      " 90.1]\n"
     ]
    }
   ],
   "source": [
    "array[\"Dimuon_mass\"].show(type=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "babe91f0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Check RDataFrame state\n",
    "- The first event loop was executed when the analysis was run\n",
    "- The second event loop was executed when the columns were retrieved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e1d39f44",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from TChain Events in file root://eospublic.cern.ch//eos/opendata/cms/derived-data/AOD2NanoAODOutreachTool/Run2012BC_DoubleMuParked_Muons.root\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            7\n",
      "Columns from defines        1\n",
      "Event loops run             2\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "Dimuon_mass     float                           Define\n",
      "Muon_charge     ROOT::VecOps::RVec<Int_t>       Dataset\n",
      "Muon_eta        ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_mass       ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_phi        ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "Muon_pt         ROOT::VecOps::RVec<Float_t>     Dataset\n",
      "nMuon           UInt_t                          Dataset"
     ]
    }
   ],
   "source": [
    "df.Describe().Print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13273822",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data as Awkward Arrays in Python\n",
    "* See Alexander's talk [End-to-end physics analysis with Open Data: the Analysis Grand Challenge](https://github.com/alexander-held/PyHEP-2022-AGC)\n",
    "* See Jim's talk [Uproot, Awkward Array, hist, Vector: from basics to combinatorics](https://github.com/jpivarski-talks/2022-09-12-pyhep22-awkward-combinatorics/blob/main/pyhep2022-combinatorics.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d6912c0",
   "metadata": {},
   "source": [
    "### Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "8898b732",
   "metadata": {},
   "outputs": [],
   "source": [
    "import hist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "46e7b0cf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAG0CAYAAAAYQdwgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0WklEQVR4nO3de1hVdd7//9fmtAUVEBSBxPOU4gFNjdt0EtNEKjvo3cExs3J0Ks2UyQONmVG/wezkaKZXXaV1j0732Kg1eqdjnhuRPDGWepOaZiVol96KgHJcvz/6sobNGdybvdg8H9e1rou9Pp+99nt9RHjxWSebYRiGAAAALMTL3QUAAACUR0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACW4+PuAuqjpKREZ8+eVcuWLWWz2dxdDgAAqAXDMHTlyhVFRkbKy6v6OZJGGVDOnj2rqKgod5cBAADq4YcfflC7du2q7dMoA0rLli0l/bKDgYGBbq4GAADURnZ2tqKioszf49VplAGl9LBOYGAgAQUAgEamNqdncJIsAACwHAIKAACwHAIKAACwHAIKAACwHAIKAACwHAIKAACwHAIKAACwHAIKAACwHAIKAACwHAIKAACwHAIKAACwnDoHlF27dmnUqFGKjIyUzWbT+vXrHdptNluly2uvvWb26dixY4X2BQsWXPfOAAAAz1DngJKbm6uYmBgtXbq00vbMzEyH5YMPPpDNZtOYMWMc+iUnJzv0e+aZZ+q3BwAAwOPU+WnGCQkJSkhIqLI9PDzc4fWnn36qoUOHqnPnzg7rW7ZsWaFvVfLz85Wfn2++zs7OrkPFAACgsXHpOSjnzp3Txo0bNXHixAptCxYsUGhoqPr27avXXntNRUVFVW4nJSVFQUFB5hIVFeXKsgEAgJvVeQalLj788EO1bNlSo0ePdlg/bdo03XzzzQoJCdGePXuUlJSkzMxMvfnmm5VuJykpSYmJiebr7OxsQkoDyisoUvS8zZKko8nxCvBz6bcNAACuDSgffPCBxo0bp2bNmjmsLxs2evfuLT8/P/3ud79TSkqK7HZ7he3Y7fZK1wMAAM/kskM8u3fvVkZGhn7729/W2Dc2NlZFRUU6ffq0q8oBAACNiMsCyvvvv69+/fopJiamxr7p6eny8vJSWFiYq8oBAACNSJ0P8eTk5OjEiRPm61OnTik9PV0hISFq3769pF/OEVmzZo3eeOONCu9PTU1VWlqahg4dqpYtWyo1NVUzZszQI488olatWl3HrgAAAE9R54Cyf/9+DR061Hxdej7JhAkTtHLlSknSxx9/LMMwNHbs2Arvt9vt+vjjjzV//nzl5+erU6dOmjFjhsN5KQAAoGmzGYZhuLuIusrOzlZQUJAuX76swMBAd5fj8biKBwDgDHX5/c2zeAAAgOUQUFBneQVF6jhnozrO2ai8gqpvsAcAQH0RUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOX4uLsAWJNhGLpaWCxJyisoMteX/RoAAFchoKBSVwuLFT1vc4X1/V/Z6oZqAABNDYd4AACA5RBQUKPds4a6uwQAQBNDQEGN/P34NgEANCx+8wAAAMshoAAAAMshoAAAAMvhMmNcl9L7ovj7estms7m5GgCAp2AGBdel/ytbFT1vs3lTNwAAnIGAAgAALIeAgjrZP3eY9s8d5u4yAAAejnNQUCcBfnzLAABcjxkUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOXUOKLt27dKoUaMUGRkpm82m9evXO7Q/9thjstlsDsvIkSMd+ly8eFHjxo1TYGCggoODNXHiROXk5FzXjgAAAM9R54CSm5urmJgYLV26tMo+I0eOVGZmprn85S9/cWgfN26cjhw5oi1btmjDhg3atWuXJk+eXPfqAQCAR6rzfcsTEhKUkJBQbR+73a7w8PBK244dO6ZNmzZp37596t+/vyRpyZIluvPOO/X6668rMjKyriUBAAAP45JzUHbs2KGwsDDddNNNeuqpp3ThwgWzLTU1VcHBwWY4kaThw4fLy8tLaWlplW4vPz9f2dnZDgsAAPBcTg8oI0eO1EcffaStW7fq1Vdf1c6dO5WQkKDi4mJJUlZWlsLCwhze4+Pjo5CQEGVlZVW6zZSUFAUFBZlLVFSUs8sGAAAW4vRH0z788MPm17169VLv3r3VpUsX7dixQ8OGDavXNpOSkpSYmGi+zs7OJqQAAODBXH6ZcefOndW6dWudOHFCkhQeHq7z58879CkqKtLFixerPG/FbrcrMDDQYQEAAJ7L6TMo5f3444+6cOGCIiIiJEkDBw7UpUuXdODAAfXr10+StG3bNpWUlCg2NtbV5aAeAvx8dHrBXebrvIIiN1YDAGgK6hxQcnJyzNkQSTp16pTS09MVEhKikJAQvfTSSxozZozCw8N18uRJzZo1S127dlV8fLwkqXv37ho5cqQmTZqk5cuXq7CwUFOnTtXDDz/MFTwAAEBSPQ7x7N+/X3379lXfvn0lSYmJierbt6/mzZsnb29vHT58WPfcc49uvPFGTZw4Uf369dPu3btlt9vNbaxatUrdunXTsGHDdOedd2rw4MF69913nbdXAACgUavzDEpcXJwMw6iyffPmzTVuIyQkRKtXr67rRwMAgCaCZ/EAAADLIaAAAADLIaAAAADLIaAAAADLIaCgzkrvi3I0Od7dpQAAPBQBBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBU6RV1CkjnM2quOcjcorKHJ3OQCARo6AAgAALKfOAWXXrl0aNWqUIiMjZbPZtH79erOtsLBQs2fPVq9evdS8eXNFRkbq0Ucf1dmzZx220bFjR9lsNodlwYIF170zAADAM9Q5oOTm5iomJkZLly6t0JaXl6eDBw/qhRde0MGDB7V27VplZGTonnvuqdA3OTlZmZmZ5vLMM8/Ubw8AAIDH8anrGxISEpSQkFBpW1BQkLZs2eKw7u2339Ytt9yiM2fOqH379ub6li1bKjw8vK4fDwAAmgCXn4Ny+fJl2Ww2BQcHO6xfsGCBQkND1bdvX7322msqKqr6xMr8/HxlZ2c7LAAAwHPVeQalLq5du6bZs2dr7NixCgwMNNdPmzZNN998s0JCQrRnzx4lJSUpMzNTb775ZqXbSUlJ0UsvveTKUgEAgIW4LKAUFhbqwQcflGEYWrZsmUNbYmKi+XXv3r3l5+en3/3ud0pJSZHdbq+wraSkJIf3ZGdnKyoqylWlAwAAN3NJQCkNJ99//722bdvmMHtSmdjYWBUVFen06dO66aabKrTb7fZKgwsAAPBMTg8opeHk+PHj2r59u0JDQ2t8T3p6ury8vBQWFubscgAAQCNU54CSk5OjEydOmK9PnTql9PR0hYSEKCIiQv/5n/+pgwcPasOGDSouLlZWVpYkKSQkRH5+fkpNTVVaWpqGDh2qli1bKjU1VTNmzNAjjzyiVq1aOW/PAABAo1XngLJ//34NHTrUfF16bsiECRM0f/58ffbZZ5KkPn36OLxv+/btiouLk91u18cff6z58+crPz9fnTp10owZMxzOMQEAAE1bnQNKXFycDMOosr26Nkm6+eabtXfv3rp+LAAAaEJ4Fg8AALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAclz3NGI1TXkGRoudtdncZAIAmjoACp7haUGJ+nVdQJEny9/WWzWZzV0kAgEaMQzxwil8v3G5+3f+VrYqet1lXC4vdWBEAoDEjoAAAAMvhEA/qzd/XW0eT4yX9clin/ytb3VwRAMBTEFBQbzabTQF+fAsBAJyPQzwAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByfNxdAKzBMAxdLSxWXkGRu0sBAICAgl9cLSxW9LzN7i4DAABJHOIBAAAWxAwKqrR/7jAF+PnI39fb3aUAAJoYZlBQpQA/HwX4+chms9Wq7+kFd+locnwDVAYA8HQEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDl1Dii7du3SqFGjFBkZKZvNpvXr1zu0G4ahefPmKSIiQv7+/ho+fLiOHz/u0OfixYsaN26cAgMDFRwcrIkTJyonJ+e6dgQAAHiOOgeU3NxcxcTEaOnSpZW2L1y4UIsXL9by5cuVlpam5s2bKz4+XteuXTP7jBs3TkeOHNGWLVu0YcMG7dq1S5MnT67/XgAAAI9S5zvJJiQkKCEhodI2wzC0aNEizZ07V/fee68k6aOPPlLbtm21fv16Pfzwwzp27Jg2bdqkffv2qX///pKkJUuW6M4779Trr7+uyMjICtvNz89Xfn6++To7O7uuZQMAgEbEqeegnDp1SllZWRo+fLi5LigoSLGxsUpNTZUkpaamKjg42AwnkjR8+HB5eXkpLS2t0u2mpKQoKCjIXKKiopxZNgAAsBinBpSsrCxJUtu2bR3Wt23b1mzLyspSWFiYQ7uPj49CQkLMPuUlJSXp8uXL5vLDDz84s2wAAGAxjeJhgXa7XXa73d1lAACABuLUGZTw8HBJ0rlz5xzWnzt3zmwLDw/X+fPnHdqLiop08eJFsw8AAGjanBpQOnXqpPDwcG3dutVcl52drbS0NA0cOFCSNHDgQF26dEkHDhww+2zbtk0lJSWKjY11ZjkAAKCRqvMhnpycHJ04ccJ8ferUKaWnpyskJETt27fX9OnT9corr+hXv/qVOnXqpBdeeEGRkZG67777JEndu3fXyJEjNWnSJC1fvlyFhYWaOnWqHn744Uqv4AEAAE1PnQPK/v37NXToUPN1YmKiJGnChAlauXKlZs2apdzcXE2ePFmXLl3S4MGDtWnTJjVr1sx8z6pVqzR16lQNGzZMXl5eGjNmjBYvXuyE3QEAAJ6gzgElLi5OhmFU2W6z2ZScnKzk5OQq+4SEhGj16tV1/WgAANBE8CweAABgOY3iMmM0rKPJ8Qrw41sDAOA+zKAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLIaAAAADLcXpA6dixo2w2W4VlypQpkqS4uLgKbU8++aSzywAAAI2Yj7M3uG/fPhUXF5uvv/nmG91xxx164IEHzHWTJk1ScnKy+TogIMDZZQAAgEbM6QGlTZs2Dq8XLFigLl26aMiQIea6gIAAhYeHO/ujAQCAh3DpOSgFBQX685//rCeeeEI2m81cv2rVKrVu3Vo9e/ZUUlKS8vLyqt1Ofn6+srOzHRYAAOC5nD6DUtb69et16dIlPfbYY+a63/zmN+rQoYMiIyN1+PBhzZ49WxkZGVq7dm2V20lJSdFLL73kylIBAICFuDSgvP/++0pISFBkZKS5bvLkyebXvXr1UkREhIYNG6aTJ0+qS5culW4nKSlJiYmJ5uvs7GxFRUW5rnAAAOBWLgso33//vb744otqZ0YkKTY2VpJ04sSJKgOK3W6X3W53eo0AAMCaXHYOyooVKxQWFqa77rqr2n7p6emSpIiICFeVAgAAGhmXzKCUlJRoxYoVmjBhgnx8/v0RJ0+e1OrVq3XnnXcqNDRUhw8f1owZM3Tbbbepd+/erigFAAA0Qi4JKF988YXOnDmjJ554wmG9n5+fvvjiCy1atEi5ubmKiorSmDFjNHfuXFeUAQAAGimXBJQRI0bIMIwK66OiorRz505XfCQAAPAgPIsHAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYjktvdQ/ryysoUvS8ze4uAwAAB8ygAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgAAAAyyGgwGWi521WxzkblVdQ5O5SAACNDAEFAABYDgEFAABYjo+7C0DDMwxDVwuLJYnDLwAASyKgNEFXC4sVPW+zu8sAAKBKHOIBAACWQ0Bp4nbPGmp+vX/uMB1Njpe/r7cbKwIAgEM8TZ6/378zaoCfjwL8+JYAALgfMygAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByuOkFXK70eT/+vt6y2WxurgYA0BgwgwKX6//KVkXP22w+oBAAgJowg9KE5BUU8ZBAAECjwAwKnMrf11tHk+O1f+4wd5cCAGjEmEGBU9lsNp7nAwC4bvwmaeIC/Hx0esFd7i4DAAAHTj/EM3/+fNlsNoelW7duZvu1a9c0ZcoUhYaGqkWLFhozZozOnTvn7DIAAEAj5pJzUHr06KHMzExz+fLLL822GTNm6O9//7vWrFmjnTt36uzZsxo9erQrygAAAI2USw7x+Pj4KDw8vML6y5cv6/3339fq1at1++23S5JWrFih7t27a+/evfqP//gPV5QDAAAaGZfMoBw/flyRkZHq3Lmzxo0bpzNnzkiSDhw4oMLCQg0fPtzs261bN7Vv316pqalVbi8/P1/Z2dkOCwAA8FxODyixsbFauXKlNm3apGXLlunUqVP69a9/rStXrigrK0t+fn4KDg52eE/btm2VlZVV5TZTUlIUFBRkLlFRUc4uGwAAWIjTD/EkJCSYX/fu3VuxsbHq0KGD/vrXv8rf379e20xKSlJiYqL5Ojs7m5ACAIAHc/mN2oKDg3XjjTfqxIkTCg8PV0FBgS5duuTQ59y5c5Wes1LKbrcrMDDQYQEAAJ7L5QElJydHJ0+eVEREhPr16ydfX19t3brVbM/IyNCZM2c0cOBAV5cCAAAaCacf4nnuuec0atQodejQQWfPntWLL74ob29vjR07VkFBQZo4caISExMVEhKiwMBAPfPMMxo4cCBX8AAAAJPTA8qPP/6osWPH6sKFC2rTpo0GDx6svXv3qk2bNpKkt956S15eXhozZozy8/MVHx+vd955x9llAACARszpAeXjjz+utr1Zs2ZaunSpli5d6uyPRhUMw9DVwmLlFRS5uxQAAGqFZ/E0AVcLixU9b3ODfmbpM37yCooa/LMBAI2fy0+SBQAAqCtmUJqo/XOHKcDPR/6+3u4uBQCACphBaaIC/HwU4Ocjm83WYJ+ZV1CkjnM2quOcjZwPAwCoFgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDg8LbGKOJscrwM89/+xXC0rMr0ufxePv692gzwMCADQOzKB4KCs+mO/XC7ebX/d/Zaui523W1cJiN1YEALAqAgoAALAcDvHApfx9vXU0OV7SL7M6/V/Z6uaKAACNAQEFLmWz2dx2zgsAoPHiEA8AALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcblDhYQzD0NXCYofb25d9Bg4AAI0BAcXDXC0sVvS8zQ7ryj4DBwCAxoBDPHCr6HmbLfVAQwCANTCD0gTsnzvMvN28v6+32+oI8PPR6QV3Ka+gqMIsDwAAZRFQmoAAPx+ehwMAaFQ4xAMAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHgAIAACyHm2PAEkrvJOvv6y2bzebmagAA7ub0GZSUlBQNGDBALVu2VFhYmO677z5lZGQ49ImLi5PNZnNYnnzySWeXgkak/ytbFT1vs64WFru7FACABTh9BmXnzp2aMmWKBgwYoKKiIj3//PMaMWKEjh49qubNm5v9Jk2apOTkZPN1QECAs0tp8o4mx3MHWQBAo+T0316bNm1yeL1y5UqFhYXpwIEDuu2228z1AQEBCg8Pr9U28/PzlZ+fb77Ozs52TrFwC39fbx1NjldeQZH6v7LV3eUAACzI5SfJXr58WZIUEhLisH7VqlVq3bq1evbsqaSkJOXl5VW5jZSUFAUFBZlLVFSUS2uGa9lsNp4PBAColkt/Q5SUlGj69OkaNGiQevbsaa7/zW9+ow4dOigyMlKHDx/W7NmzlZGRobVr11a6naSkJCUmJpqvs7Ozm3RIKfs0YA7jAAA8kUt/s02ZMkXffPONvvzyS4f1kydPNr/u1auXIiIiNGzYMJ08eVJdunSpsB273S673e7KUgEAgIW47BDP1KlTtWHDBm3fvl3t2rWrtm9sbKwk6cSJE64qBwAANCJOn0ExDEPPPPOM1q1bpx07dqhTp041vic9PV2SFBER4exymoyyh30AAGjsnB5QpkyZotWrV+vTTz9Vy5YtlZWVJUkKCgqSv7+/Tp48qdWrV+vOO+9UaGioDh8+rBkzZui2225T7969nV0OAABohJweUJYtWybpl5uxlbVixQo99thj8vPz0xdffKFFixYpNzdXUVFRGjNmjObOnevsUgAAQCPlkkM81YmKitLOnTud/bEAAMCD8LBAAABgOQQUuE2An49OL7hLR5Pj3V0KAMBiuMNXI2IYhq4WFptP/pX+/RTgsusAAGjsCCiNyNXC4gqXEvMsGwCAJ+IQDwAAsBwCSiOQV1CkjnM2Vnojtt2zhrqhItcp3deOczZy2AoAmjACSiPn78c/IQDA83AOSj1Z5YnC/r7eOpocr7yCIvN8lP1zhynAz0f+vt5uqQkAgOtFQGnkbDZbhXAU4OfjtsAEAIAzcHygAXF+BQAAtUNAAQAAlkNAAQAAlkNAqYSzDsVwSAcAgPrhTEoLq+zW9qXceeWQK10tKDG/Lt1vf19v2Ww2d5UEAHADz/sN18iUv1xZkvl6/9xhTe5W9r9euN38unTfy4cxq1ziDQBwHX6y11F1D+xz51/6pU8GBgDAExBQ6qi6B/Y1xF/znngTttKbzUlyuOEcAKDpIqA0kPLnkZS+ruvJs554E7bKbjYHAGja+K3QQMrPCjBLAABA1QgoZTjr/JLqrr4BAAA1I6CU4azzSyrbTnm7Zw11uGIFAAD8GwHFicpeHlwTf7+K98grP+NS9p4gTV3prJQky11BBQBwPgJKAykNLdWde1K+rewMS1O530fp5dJl73UiVT0r1ZBXUAEAGg4/0WupLrMjleGXJwAAtcdvzetQ/s6vdVF674+y9/0oP8tSes+T0v74BefvAIDnI6A0oJru9lp+lsUT73niDJWdvwMA8Cz8pAcAAJbDn+dVuJ7DN2WvvuHkTQAA6o7fnC7A+RHOVZuQ6IynG/OUZACwDn4C15Gz701S2XkpPJW4elVdigwA8BwElDoqPztS1dU4XH0DAED9EVCuU2VP4uXqG5TH4SMAqBt+SlaD2REAANyDgFINZkcan9Jn8xiGoR4v/kNSxSuySmcwys5qlL9DcH23U9vZEavNqFitnqaMfwvgF3znO0lNN2FDwyh/V966vu96twMAcA4CSi0QPlAfNT2BGQBQNQIKLKm6839Kz/2prI8V1HT5c1W1lp/al2q+v4u7Dwe46vNrs11X7rsnHLZrDBrjmDXGmhsrRhaWVJvzfyrrU6rs/WrKz1iUvq7NTEZdt1Of2ZGy57s0tNJZnqr2wd/XWzabrcHraor4twAcEVDgkcrer6aq80tcsZ2qtl3dE5jdeb7L1cLiCrM9ZfeBvxAbDv8WgCO3frcvXbpUr732mrKyshQTE6MlS5bolltucWdJsJjanP/jzDvLXs8zmKpTmycw12a2puxVRc668qg6VR1yOvLSCNlstgp/7Zc93FbVVU411VPVfpU/3FdTn9pedeWM/XJWPdWpyxjWd+wbso8zr5yjZtf0cTe3VfDf//3fSkxM1PLlyxUbG6tFixYpPj5eGRkZCgsLc1dZaMRKz0mRKv9lVv5clurOb6nvdqpSGqLKTuOXf09tZmvK/oB01ZVH1c32lLpaWHxdM1PVqc9MlbM+u7775ap6avNv0Vg1xivnGmPNjZnbAsqbb76pSZMm6fHHH5ckLV++XBs3btQHH3ygOXPmuKssNGJlz0mpzTOOqpudqc92avOXcGmN9Z31qc2zoOp7/k3Zv6zKK7+uNnXU5hydupwPVFf1OWeovvvlrHpK1eXforr9quvYN2SfqlzveV/U7Lw+7p5FsRluODOvoKBAAQEB+uSTT3TfffeZ6ydMmKBLly7p008/deifn5+v/Px88/Xly5fVvn17/fDDDwoMDHRaXXkFRbrl//slEX/1h2Fu/8dB41L2suLyqjrBsbr3lLpaUKwhr+2osn3nzDhJqrZPbZR+z5f9f1Abzvr8yrbr7+dd7f7Xps/1fL5Ut/1yVj31/bdojFz1/eNKjbHm+vjmpXinbzM7O1tRUVG6dOmSgoKCqu9suMFPP/1kSDL27NnjsH7mzJnGLbfcUqH/iy++aEhiYWFhYWFh8YDlhx9+qDErNIopgqSkJCUmJpqvS0pKdPHiRYWGhnrsZXelKdPZs0SNHeNSEWNSEWNSOcalIsakcq4aF8MwdOXKFUVGRtbY1y0BpXXr1vL29ta5c+cc1p87d07h4eEV+tvtdtntdod1wcHBrizRMgIDA/lPUwnGpSLGpCLGpHKMS0WMSeVcMS41Htr5f2q+9tEF/Pz81K9fP23d+u9jqyUlJdq6dasGDhzojpIAAICFuO0QT2JioiZMmKD+/fvrlltu0aJFi5Sbm2te1QMAAJoutwWUhx56SD///LPmzZunrKws9enTR5s2bVLbtm3dVZKl2O12vfjiixUObTV1jEtFjElFjEnlGJeKGJPKWWFc3HKZMQAAQHXccg4KAABAdQgoAADAcggoAADAcggoAADAcggoblZcXKwXXnhBnTp1kr+/v7p06aKXX35ZZc9dNgxD8+bNU0REhPz9/TV8+HAdP37cjVU7165duzRq1ChFRkbKZrNp/fr1Du212f+LFy9q3LhxCgwMVHBwsCZOnKicnJwG3Avnqm5MCgsLNXv2bPXq1UvNmzdXZGSkHn30UZ09e9ZhG542JlLN3ytlPfnkk7LZbFq0aJHDek8bl9qMybFjx3TPPfcoKChIzZs314ABA3TmzBmz/dq1a5oyZYpCQ0PVokULjRkzpsKNNBuTmsYkJydHU6dOVbt27eTv76/o6GgtX77coY+njUlKSooGDBigli1bKiwsTPfdd58yMjIc+tRmn8+cOaO77rpLAQEBCgsL08yZM1VU5PyHfUoEFLd79dVXtWzZMr399ts6duyYXn31VS1cuFBLliwx+yxcuFCLFy/W8uXLlZaWpubNmys+Pl7Xrl1zY+XOk5ubq5iYGC1durTS9trs/7hx43TkyBFt2bJFGzZs0K5duzR58uSG2gWnq25M8vLydPDgQb3wwgs6ePCg1q5dq4yMDN1zzz0O/TxtTKSav1dKrVu3Tnv37q30dtqeNi41jcnJkyc1ePBgdevWTTt27NDhw4f1wgsvqFmzZmafGTNm6O9//7vWrFmjnTt36uzZsxo9enRD7YLT1TQmiYmJ2rRpk/785z/r2LFjmj59uqZOnarPPvvM7ONpY7Jz505NmTJFe/fu1ZYtW1RYWKgRI0YoNzfX7FPTPhcXF+uuu+5SQUGB9uzZow8//FArV67UvHnzXFO0E579h+tw1113GU888YTDutGjRxvjxo0zDMMwSkpKjPDwcOO1114z2y9dumTY7XbjL3/5S4PW2hAkGevWrTNf12b/jx49akgy9u3bZ/b5/PPPDZvNZvz0008NVrurlB+Tynz11VeGJOP77783DMPzx8Qwqh6XH3/80bjhhhuMb775xujQoYPx1ltvmW2ePi6VjclDDz1kPPLII1W+59KlS4avr6+xZs0ac92xY8cMSUZqaqqrSm0wlY1Jjx49jOTkZId1N998s/GHP/zBMAzPHxPDMIzz588bkoydO3cahlG7ff6f//kfw8vLy8jKyjL7LFu2zAgMDDTy8/OdXiMzKG526623auvWrfr2228lSf/617/05ZdfKiEhQZJ06tQpZWVlafjw4eZ7goKCFBsbq9TUVLfU3JBqs/+pqakKDg5W//79zT7Dhw+Xl5eX0tLSGrxmd7h8+bJsNpv5jKqmOiYlJSUaP368Zs6cqR49elRob2rjUlJSoo0bN+rGG29UfHy8wsLCFBsb63DI48CBAyosLHT4P9atWze1b9/eY3/G3Hrrrfrss8/0008/yTAMbd++Xd9++61GjBghqWmMyeXLlyVJISEhkmq3z6mpqerVq5fDDVXj4+OVnZ2tI0eOOL1GAoqbzZkzRw8//LC6desmX19f9e3bV9OnT9e4ceMkSVlZWZJU4Q67bdu2Nds8WW32PysrS2FhYQ7tPj4+CgkJaRJjdO3aNc2ePVtjx441H+rVVMfk1VdflY+Pj6ZNm1Zpe1Mbl/PnzysnJ0cLFizQyJEj9Y9//EP333+/Ro8erZ07d0r6ZUz8/PwqPIDVk3/GLFmyRNHR0WrXrp38/Pw0cuRILV26VLfddpskzx+TkpISTZ8+XYMGDVLPnj0l1W6fs7KyKv1ZXNrmbG671T1+8de//lWrVq3S6tWr1aNHD6Wnp2v69OmKjIzUhAkT3F0eLK6wsFAPPvigDMPQsmXL3F2OWx04cEB/+tOfdPDgQdlsNneXYwklJSWSpHvvvVczZsyQJPXp00d79uzR8uXLNWTIEHeW5zZLlizR3r179dlnn6lDhw7atWuXpkyZosjISIcZBE81ZcoUffPNN/ryyy/dXUq1mEFxs5kzZ5qzKL169dL48eM1Y8YMpaSkSJLCw8MlqcKZ1OfOnTPbPFlt9j88PFznz593aC8qKtLFixc9eoxKw8n333+vLVu2ODwSvSmOye7du3X+/Hm1b99ePj4+8vHx0ffff6/f//736tixo6SmNy6tW7eWj4+PoqOjHdZ3797dvIonPDxcBQUFunTpkkMfT/0Zc/XqVT3//PN68803NWrUKPXu3VtTp07VQw89pNdff12SZ4/J1KlTtWHDBm3fvl3t2rUz19dmn8PDwyv9WVza5mwEFDfLy8uTl5fjP4O3t7f5l0+nTp0UHh6urVu3mu3Z2dlKS0vTwIEDG7RWd6jN/g8cOFCXLl3SgQMHzD7btm1TSUmJYmNjG7zmhlAaTo4fP64vvvhCoaGhDu1NcUzGjx+vw4cPKz093VwiIyM1c+ZMbd68WVLTGxc/Pz8NGDCgwuWk3377rTp06CBJ6tevn3x9fR3+j2VkZOjMmTMe+TOmsLBQhYWF1f7c9cQxMQxDU6dO1bp167Rt2zZ16tTJob02+zxw4EB9/fXXDiG/9I+j8iHYWUXDjSZMmGDccMMNxoYNG4xTp04Za9euNVq3bm3MmjXL7LNgwQIjODjY+PTTT43Dhw8b9957r9GpUyfj6tWrbqzcea5cuWIcOnTIOHTokCHJePPNN41Dhw6ZV6TUZv9Hjhxp9O3b10hLSzO+/PJL41e/+pUxduxYd+3SdatuTAoKCox77rnHaNeunZGenm5kZmaaS9kz6T1tTAyj5u+V8spfxWMYnjcuNY3J2rVrDV9fX+Pdd981jh8/bixZssTw9vY2du/ebW7jySefNNq3b29s27bN2L9/vzFw4EBj4MCB7tql61bTmAwZMsTo0aOHsX37duO7774zVqxYYTRr1sx45513zG142pg89dRTRlBQkLFjxw6Hnxl5eXlmn5r2uaioyOjZs6cxYsQIIz093di0aZPRpk0bIykpySU1E1DcLDs723j22WeN9u3bG82aNTM6d+5s/OEPf3D4RVNSUmK88MILRtu2bQ273W4MGzbMyMjIcGPVzrV9+3ZDUoVlwoQJhmHUbv8vXLhgjB071mjRooURGBhoPP7448aVK1fcsDfOUd2YnDp1qtI2Scb27dvNbXjamBhGzd8r5VUWUDxtXGozJu+//77RtWtXo1mzZkZMTIyxfv16h21cvXrVePrpp41WrVoZAQEBxv33329kZmY28J44T01jkpmZaTz22GNGZGSk0axZM+Omm24y3njjDaOkpMTchqeNSVU/M1asWGH2qc0+nz592khISDD8/f2N1q1bG7///e+NwsJCl9Rs+3+FAwAAWAbnoAAAAMshoAAAAMshoAAAAMshoAAAAMshoAAAAMshoAAAAMshoAAAAMshoAAAAMshoAAAAMshoAAAAMshoAAuFBcXp+nTp7u7DLhRXFycbDabbDab0tPT3VbHY489Ztaxfv16t9UB1BYBBaiHsj/sfX191bZtW91xxx364IMPzEe2S9LatWv18ssvu7FSWMGkSZOUmZmpnj17muuysrL07LPPqmvXrmrWrJnatm2rQYMGadmyZcrLy6v1tkeNGqWRI0dW2rZ7927ZbDYdPnxYf/rTn5SZmXnd+wI0FB93FwA0ViNHjtSKFStUXFysc+fOadOmTXr22Wf1ySef6LPPPpOPj49CQkLcXSYsICAgQOHh4ebr7777ToMGDVJwcLD++Mc/qlevXrLb7fr666/17rvv6oYbbtA999xTq21PnDhRY8aM0Y8//qh27do5tK1YsUL9+/dX7969JUlBQUHO2ynAxZhBAerJbrcrPDxcN9xwg26++WY9//zz+vTTT/X5559r5cqVkioe4omLi9Mzzzyj6dOnq1WrVmrbtq3ee+895ebm6vHHH1fLli3VtWtXff755w6f1bFjRy1atMhhXZ8+fTR//nxJUn5+vqZNm6awsDA1a9ZMgwcP1r59+xz6x8XFadq0aZo1a5ZCQkIUHh5uvr8q9a1306ZNGjx4sIKDgxUaGqq7775bJ0+eNNs/+eQT9erVS/7+/goNDdXw4cOVm5tbY5szayzvyy+/lK+vr65du2auO336tGw2m77//vta96mNp59+Wj4+Ptq/f78efPBBde/eXZ07d9a9996rjRs3atSoUWbfkpISpaSkqFOnTvL391dMTIw++eQTs/3uu+9WmzZtzO+5Ujk5OVqzZo0mTpxY67oAKyGgAE50++23KyYmRmvXrq2yz4cffqjWrVvrq6++0jPPPKOnnnpKDzzwgG699VYdPHhQI0aM0Pjx4+s0zT9r1iz97W9/04cffqiDBw+qa9euio+P18WLFyt8dvPmzZWWlqaFCxcqOTlZW7ZsqXbb9ak3NzdXiYmJ2r9/v7Zu3SovLy/df//9KikpUWZmpsaOHasnnnhCx44d044dOzR69GgZhlFtm7NrLC89PV3du3dXs2bNzHWHDh1Sq1at1KFDh1r3qcmFCxf0j3/8Q1OmTFHz5s0r7WOz2cyvU1JS9NFHH2n58uU6cuSIZsyYoUceeUQ7d+6UJPn4+OjRRx/VypUrHcZpzZo1Ki4u1tixY2tVF2A5BoA6mzBhgnHvvfdW2vbQQw8Z3bt3NwzDMIYMGWI8++yzZtuQIUOMwYMHm6+LioqM5s2bG+PHjzfXZWZmGpKM1NRUc12HDh2Mt956y+FzYmJijBdffNHIyckxfH19jVWrVpltBQUFRmRkpLFw4cIqP9swDGPAgAHG7Nmzq9zP+tZb3s8//2xIMr7++mvjwIEDhiTj9OnTFfpV1+bqGn/7298ajz76qMO6efPmGXFxcXXqU1l9Zb8H9u7da0gy1q5d69AvNDTUaN68udG8eXNj1qxZhmEYxrVr14yAgABjz549Dn0nTpxojB071nx97NgxQ5Kxfft2c92vf/1r45FHHqlQjyRj3bp1VdYLWAUzKICTGYbh8BdweaXnA0iSt7e3QkND1atXL3Nd27ZtJUnnz5+v1eedPHlShYWFGjRokLnO19dXt9xyi44dO1blZ0tSREREjZ9Tn3qPHz+usWPHqnPnzgoMDFTHjh0lSWfOnFFMTIyGDRumXr166YEHHtB7772n//u//5OkatucXWN56enp6tOnj8O6Q4cOOayrTZ/6+uqrr5Senq4ePXooPz9fknTixAnl5eXpjjvuUIsWLczlo48+cjhk1q1bN91666364IMPzPft3r2bwzto1AgogJMdO3ZMnTp1qrLd19fX4XXplUBlX0tyuBrIy8urwmGOwsLCOtdW2WeX/Rxn1Ttq1ChdvHhR7733ntLS0pSWliZJKigokLe3t7Zs2aLPP/9c0dHRWrJkiW666SadOnWq2jZn11hWcXGxvvnmG/Xt29dh/cGDB83wUZs+tdG1a1fZbDZlZGQ4rO/cubO6du0qf39/c11OTo4kaePGjUpPTzeXo0ePOpyHIv1ysuzf/vY3XblyRStWrFCXLl00ZMiQWtcFWA0BBXCibdu26euvv9aYMWOcut02bdo4XCKanZ1t/tLu0qWL/Pz89M9//tNsLyws1L59+xQdHe3UOmrjwoULysjI0Ny5czVs2DB17969wiyIzWbToEGD9NJLL+nQoUPy8/PTunXramxzlYyMDF27dk2RkZHmutTUVP30009m+KhNn9oIDQ3VHXfcobfffrvak38lKTo6Wna7XWfOnFHXrl0dlqioKIe+Dz74oLy8vLR69Wp99NFHeuKJJ6qdyQOsjsuMgXrKz89XVlaWw2XGKSkpuvvuu/Xoo4869bNuv/12rVy5UqNGjVJwcLDmzZsnb29vSVLz5s311FNPaebMmQoJCVH79u21cOFC5eXluWWKv1WrVgoNDdW7776riIgInTlzRnPmzDHb09LStHXrVo0YMUJhYWFKS0vTzz//rO7du1fb5kqlN1BbsmSJpk2bphMnTmjatGmSfpn1qW2f2nrnnXc0aNAg9e/fX/Pnz1fv3r3l5eWlffv26X//93/Vr18/SVLLli313HPPacaMGSopKdHgwYN1+fJl/fOf/1RgYKAmTJhgbrNFixZ66KGHlJSUpOzsbD322GPXMSKA+xFQgHratGmTIiIi5OPjo1atWikmJkaLFy/WhAkT5OXl3MnJpKQknTp1SnfffbeCgoL08ssvOxz2WLBggUpKSjR+/HhduXJF/fv31+bNm9WqVSun1lEbXl5e+vjjjzVt2jT17NlTN910kxYvXqy4uDhJUmBgoHbt2qVFixYpOztbHTp00BtvvKGEhAQdO3asyjZXSk9PV3x8vL777jv16tVL0dHReumll/TUU09p8eLF+q//+q9a9amtLl266NChQ/rjH/+opKQk/fjjj7Lb7YqOjtZzzz2np59+2uz78ssvq02bNkpJSdF3332n4OBg87L28iZOnKj3339fd955p8NMD9AY2YzyB7YBoImJj4/XgAED9Morr1xXn8rExcWpT58+Fe5j4y42m03r1q3Tfffd5+5SgGpxDgqAJu9f//qXw1U/9e1TlXfeeUctWrTQ119/Xa/3O8OTTz6pFi1auO3zgbpiBgVAk5aVlaWIiAgdOXKkypOKa9OnKj/99JOuXr0qSWrfvr38/Pyuu+b6OH/+vLKzsyX9cnl5VTeJA6yCgAIAACyHQzwAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMBy/n/8ZyclmbBHGgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h = (\n",
    "    hist.Hist.new.Regular(100, 70, 200, name=\"Dimuon mass m $\\mu$$\\mu$ [GeV]\")\n",
    "    .Double()\n",
    "    .fill(ak.flatten(array[\"Dimuon_mass\"], axis = -1))\n",
    "    .plot()\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "681e85b0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### NumPy and Awkward Arrays\n",
    "- Let's ask for the mean of the dimuon mass: and check that we get the same result using a NumPy function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "eb84a10d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f97fc534",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91.438578125"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(array[\"Dimuon_mass\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "806b2c5e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Awkward roundtrip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52fc3873",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "* The multi-threading can be enabled with:\n",
    "```python\n",
    "ROOT.ROOT.EnableImplicitMT()\n",
    "```\n",
    "* Create a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "0a11a7e2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df = ROOT.RDataFrame('Events', 'root://eospublic.cern.ch//eos/opendata/cms/derived-data/AOD2NanoAODOutreachTool/Run2012BC_DoubleMuParked_Muons.root')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea67f535",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## From RDataFrame to Awkward"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab2d871a",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's take out all 60 mio. data events as an Awkward Array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "190d5bf6",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "array_out = ak.from_rdataframe(\n",
    "        df,\n",
    "        columns=(\n",
    "            \"Muon_charge\",\n",
    "            \"Muon_eta\",\n",
    "            \"Muon_mass\",\n",
    "            \"Muon_phi\",\n",
    "            \"Muon_pt\",\n",
    "            \"nMuon\",\n",
    "        ),\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "be294f04",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{Muon_charge: [...], ...}, ..., {...}] type='61540413 * {Muon_charg...'>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "3f81d1cd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: 61540413 * {\n",
      "    Muon_charge: var * int32,\n",
      "    Muon_eta: var * float32,\n",
      "    Muon_mass: var * float32,\n",
      "    Muon_phi: var * float32,\n",
      "    Muon_pt: var * float32,\n",
      "    nMuon: uint32\n",
      "}\n",
      "[{Muon_charge: [-1, -1], Muon_eta: [1.07, -0.564], Muon_mass: [...], ...},\n",
      " {Muon_charge: [1, -1], Muon_eta: [-0.428, 0.349], Muon_mass: [...], ...},\n",
      " {Muon_charge: [1], Muon_eta: [2.21], Muon_mass: [0.106], Muon_phi: ..., ...},\n",
      " {Muon_charge: [1, 1, 1, 1], Muon_eta: [-1.59, ...], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, -1, 1, 1], Muon_eta: [-2.17, ...], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, 1, -1], Muon_eta: [-1.37, ...], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, 1], Muon_eta: [-0.532, -1], Muon_mass: [...], ...},\n",
      " {Muon_charge: [1, -1], Muon_eta: [-0.772, ...], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, 1], Muon_eta: [0.442, 0.702], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, 1], Muon_eta: [2.34, 2.35], Muon_mass: [...], ...},\n",
      " ...,\n",
      " {Muon_charge: [1, -1], Muon_eta: [0.252, 0.275], Muon_mass: [...], ...},\n",
      " {Muon_charge: [1, -1], Muon_eta: [1.27, 1.29], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, 1], Muon_eta: [0.964, ...], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, 1, 1], Muon_eta: [-0.728, ...], Muon_mass: [...], ...},\n",
      " {Muon_charge: [1, -1], Muon_eta: [0.154, 0.0944], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, 1], Muon_eta: [-1.92, 0.378], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, -1, -1, -1], Muon_eta: [-0.566, ...], Muon_mass: ..., ...},\n",
      " {Muon_charge: [-1, -1, -1], Muon_eta: [-1.47, ...], Muon_mass: [...], ...},\n",
      " {Muon_charge: [-1, 1], Muon_eta: [-2.15, 0.291], Muon_mass: [...], ...}]\n"
     ]
    }
   ],
   "source": [
    "array_out.show(type=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27390847",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## From Awkward to RDataFrame\n",
    "- Let's put the whole array in another RDataFrame as a <b>single</b> column <i>\"Events\"</i>\n",
    "    - The large-scale data is not copied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "dada3a61",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df1 = ak.to_rdataframe({\"Events\": array_out})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "ac69bc8a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from datasource Custom Datasource\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            2\n",
      "Columns from defines        1\n",
      "Event loops run             0\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "Events          awkward::Record_1M7vRWBynyE     Dataset\n",
      "awkward_index_  Long64_t                        Define"
     ]
    }
   ],
   "source": [
    "df1.Describe().Print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cd4593d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Awkward column type\n",
    "- The `awkward::Record_` type array view records are accessed through functions\n",
    "- The function names are the same as the original Awkward array record field names: \n",
    "    - `Events.nMuon()`, `Events.Muon_pt()`, `Events.Muon_eta()`, `Events.Muon_phi()`, `Events.Muon_mass()`\n",
    "- They return an iterable object - `ROOT::VecOps::RVec`: an `integer` index can be used to access the elements of the record:\n",
    "    - `Events.Muon_charge()[0]`\n",
    "- An `InvariantMass` C++ function can be applied to define a new `Dimuon_mass` column:\n",
    "```python\n",
    "df2 = df1.Filter('Events.nMuon() == 2')\n",
    "         .Filter('Events.Muon_charge()[0] != Events.Muon_charge()[1]')\n",
    "         .Define('Dimuon_mass',\n",
    "                 'InvariantMass(Events.Muon_pt(), Events.Muon_eta(), Events.Muon_phi(), Events.Muon_mass())')\n",
    "         .Filter('Dimuon_mass > 70')\n",
    "         .Range(1000)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "b5fb0f68",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df2 = df1.Filter('Events.nMuon() == 2')\\\n",
    "         .Filter('Events.Muon_charge()[0] != Events.Muon_charge()[1]')\\\n",
    "         .Define('Dimuon_mass',\\\n",
    "                 'InvariantMass(Events.Muon_pt(), Events.Muon_eta(), Events.Muon_phi(), Events.Muon_mass())')\\\n",
    "         .Filter('Dimuon_mass > 70')\\\n",
    "         .Range(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "a843f44f",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from datasource Custom Datasource\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            3\n",
      "Columns from defines        2\n",
      "Event loops run             0\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "Dimuon_mass     float                           Define\n",
      "Events          awkward::Record_1M7vRWBynyE     Dataset\n",
      "awkward_index_  Long64_t                        Define"
     ]
    }
   ],
   "source": [
    "df2.Describe().Print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "585f22d7",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Approximate mass of the Z boson: 91.44 GeV\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('\\nApproximate mass of the Z boson: {:.2f} GeV\\n'.format(\n",
    "        df2.Mean('Dimuon_mass').GetValue()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d675057",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## User-defined C++ lambda functions\n",
    "Let's define our own `invariant_mass` function that takes an Awkward type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "1b4ae9fb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "rdf3 = df1.Filter('Events.nMuon() == 2')\\\n",
    "    .Filter('Events.Muon_charge()[0] != Events.Muon_charge()[1]')\\\n",
    "    .Define(\"dimuon_mass\", \"\"\"\n",
    "//\n",
    "// Let the RDF handle the following selections:\n",
    "// if (Events.nMuon() == 2) { ...\n",
    "//    if (Events.Muon_charge()[0] != Events.Muon_charge()[1]) { ...\n",
    "\n",
    "return std::sqrt(2 * Events.Muon_pt()[0] * Events.Muon_pt()[1]\n",
    "    * (std::cosh(Events.Muon_eta()[0] - Events.Muon_eta()[1]) \n",
    "    - std::cos(Events.Muon_phi()[0] - Events.Muon_phi()[1])));\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "02ab92fc",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from datasource Custom Datasource\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            3\n",
      "Columns from defines        2\n",
      "Event loops run             1\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "Events          awkward::Record_1M7vRWBynyE     Dataset\n",
      "awkward_index_  Long64_t                        Define\n",
      "dimuon_mass     float                           Define"
     ]
    }
   ],
   "source": [
    "rdf3.Describe().Print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "2ea8cd84",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Approximate mass of the Z boson: 91.44 GeV\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('\\nApproximate mass of the Z boson: {:.2f} GeV\\n'.format(\n",
    "        rdf3.Filter('dimuon_mass > 70').Range(1000).Mean('dimuon_mass').GetValue()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "7b4d0d71",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "result = ak.from_rdataframe(\n",
    "    rdf3,\n",
    "    columns=('dimuon_mass',))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "d8452792",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{dimuon_mass: 27.9}, {...}, ..., {...}] type='24067843 * {dimuon_ma...'>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "2019d528",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "out2 = result[result['dimuon_mass'] > 70]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "e2680a39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{dimuon_mass: 114}, {...}, ..., {...}] type='5638140 * {dimuon_mass...'>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "5f3f251d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91.438265625"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(out2['dimuon_mass'][:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cea6cf6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## User-defined C++ functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "3d632610",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "compiler = ROOT.gInterpreter.Declare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "64a3d494",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "done = compiler(\"\"\"\n",
    "template<class T>\n",
    "using Vec = const ROOT::VecOps::RVec<T>;\n",
    "\n",
    "template <typename T>\n",
    "T invariant_mass(Vec<T>& pt, Vec<T>& eta, Vec<T>& phi, Vec<T>& mass) {\n",
    "  const std::size_t size = pt.size();\n",
    "  \n",
    "  R__ASSERT(eta.size() == size && phi.size() == size && mass.size() == size);\n",
    "\n",
    "  T x_sum = 0.; T y_sum = 0.; T z_sum = 0.; T e_sum = 0.;\n",
    "\n",
    "  for (std::size_t i = 0u; i < size; ++ i) {\n",
    "  // Convert to (e, x, y, z) coordinate system and update sums\n",
    "    const auto x = pt[i] * std::cos(phi[i]);\n",
    "    x_sum += x;\n",
    "\n",
    "    const auto y = pt[i] * std::sin(phi[i]);\n",
    "    y_sum += y;\n",
    "\n",
    "    const auto z = pt[i] * std::sinh(eta[i]);\n",
    "    z_sum += z;\n",
    "\n",
    "    e_sum += std::sqrt(x * x + y * y + z * z + mass[i] * mass[i]);\n",
    "  }\n",
    "\n",
    "  // Return invariant mass with (+, -, -, -) metric\n",
    "  return std::sqrt(\n",
    "      e_sum * e_sum - x_sum * x_sum - y_sum * y_sum - z_sum * z_sum);\n",
    "}\n",
    "\"\"\")\n",
    "assert done is True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "af534e2a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df4 = df1.Filter('Events.nMuon() == 2')\\\n",
    "         .Filter('Events.Muon_charge()[0] != Events.Muon_charge()[1]')\\\n",
    "         .Define('Dimuon_mass',\\\n",
    "                 'invariant_mass(Events.Muon_pt(), Events.Muon_eta(), Events.Muon_phi(), Events.Muon_mass())')\\\n",
    "         .Filter('Dimuon_mass > 70')\\\n",
    "         .Range(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "91a02bb0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Approximate mass of the Z boson: 91.44 GeV\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('\\nApproximate mass of the Z boson: {:.2f} GeV\\n'.format(\n",
    "        df4.Mean('Dimuon_mass').GetValue()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11395213",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## From Awkward to RDataFrame (2)\n",
    "- The array records can be placed in an RDataFrame as native `ROOT::VecOps::RVec`s:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "77db5805",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df3 = ak.to_rdataframe({\"ak_Muon_charge\": array_out[\"Muon_charge\"], \n",
    "                       \"ak_Muon_eta\": array_out[\"Muon_eta\"], \n",
    "                       \"ak_Muon_mass\": array_out[\"Muon_mass\"],\n",
    "                       \"ak_Muon_phi\": array_out[\"Muon_phi\"],\n",
    "                       \"ak_Muon_pt\": array_out[\"Muon_pt\"],\n",
    "                       \"ak_nMuon\": array_out[\"nMuon\"],\n",
    "                      })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "72095b90",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from datasource Custom Datasource\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            7\n",
      "Columns from defines        1\n",
      "Event loops run             0\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "ak_Muon_charge  ROOT::VecOps::RVec<int32_t>     Dataset\n",
      "ak_Muon_eta     ROOT::VecOps::RVec<float>       Dataset\n",
      "ak_Muon_mass    ROOT::VecOps::RVec<float>       Dataset\n",
      "ak_Muon_phi     ROOT::VecOps::RVec<float>       Dataset\n",
      "ak_Muon_pt      ROOT::VecOps::RVec<float>       Dataset\n",
      "ak_nMuon        uint32_t                        Dataset\n",
      "awkward_index_  Long64_t                        Define"
     ]
    }
   ],
   "source": [
    "df3.Describe().Print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "963fa3c7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df4 = df3.Filter('ak_nMuon == 2')\\\n",
    "       .Filter('ak_Muon_charge[0] != ak_Muon_charge[1]')\\\n",
    "       .Define('ak_Dimuon_mass', 'invariant_mass(ak_Muon_pt, ak_Muon_eta, ak_Muon_phi, ak_Muon_mass)')\\\n",
    "       .Filter('ak_Dimuon_mass > 70')\\\n",
    "       .Range(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e908d30a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe from datasource Custom Datasource\n",
      "\n",
      "Property                Value\n",
      "--------                -----\n",
      "Columns in total            8\n",
      "Columns from defines        2\n",
      "Event loops run             0\n",
      "Processing slots            1\n",
      "\n",
      "Column          Type                            Origin\n",
      "------          ----                            ------\n",
      "ak_Dimuon_mass  float                           Define\n",
      "ak_Muon_charge  ROOT::VecOps::RVec<int32_t>     Dataset\n",
      "ak_Muon_eta     ROOT::VecOps::RVec<float>       Dataset\n",
      "ak_Muon_mass    ROOT::VecOps::RVec<float>       Dataset\n",
      "ak_Muon_phi     ROOT::VecOps::RVec<float>       Dataset\n",
      "ak_Muon_pt      ROOT::VecOps::RVec<float>       Dataset\n",
      "ak_nMuon        uint32_t                        Dataset\n",
      "awkward_index_  Long64_t                        Define"
     ]
    }
   ],
   "source": [
    "df4.Describe().Print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "ebc21004",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Approximate mass of the Z boson: 91.44 GeV\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('\\nApproximate mass of the Z boson: {:.2f} GeV\\n'.format(\n",
    "        df4.Mean('ak_Dimuon_mass').GetValue()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f5e1295",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summary\n",
    "- Awkward Arrays and RDataFrame provide two very different ways of performing large scale\n",
    "    - By adding the ability to convert between them, users get the best of both\n",
    "- The Awkward-RDF bridge provides users with more flexibility in mixing different packages and languages in their analyses\n",
    "    - It is a part of Awkward version 2\n",
    "- Awkward version 2 will be fully released at the beginning of December 2022\n",
    "    - The time line [here](https://github.com/scikit-hep/awkward/wiki)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43c859a8",
   "metadata": {},
   "source": [
    "### Feedback is greatly appreciated! Please, report issues [here](https://github.com/scikit-hep/awkward/issues). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}