docs/query_language.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "387fdda3",
   "metadata": {
    "papermill": {
     "duration": 0.002282,
     "end_time": "2024-04-05T18:05:54.466912",
     "exception": false,
     "start_time": "2024-04-05T18:05:54.464630",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Query Language: Thicket Tutorial\n",
    "\n",
    "Thicket is a python-based toolkit for Exploratory Data Analysis (EDA) of parallel performance data that enables performance optimization and understanding of applications’ performance on supercomputers. It bridges the performance tool gap between being able to consider only a single instance of a simulation run (e.g., single platform, single measurement tool, or single scale) and finding actionable insights in multi-dimensional, multi-scale, multi-architecture, and multi-tool performance datasets.\n",
    "\n",
    "**NOTE: An interactive version of this notebook is available in the Binder environment.**\n",
    "\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/llnl/thicket-tutorial/develop)\n",
    "\n",
    "***\n",
    "\n",
    "## 1. Import Necessary Packages\n",
    "\n",
    "To explore the structure and various capabilities of thicket components, we begin by importing necessary packages. These include python extensions and thicket's statistical functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0689fbba",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:54.470902Z",
     "iopub.status.busy": "2024-04-05T18:05:54.470774Z",
     "iopub.status.idle": "2024-04-05T18:05:55.030935Z",
     "shell.execute_reply": "2024-04-05T18:05:55.030558Z"
    },
    "papermill": {
     "duration": 0.562916,
     "end_time": "2024-04-05T18:05:55.031605",
     "exception": false,
     "start_time": "2024-04-05T18:05:54.468689",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "var Roundtrip_Obj = {};\n",
       "var refresh_cycle = false;\n",
       "var clicked_cell = null;\n",
       "var cached_cells = Jupyter.notebook.get_cell_elements();\n",
       "\n",
       "/**\n",
       " * @name unindentPyCode\n",
       " * @description Removes leading indentations from a python code string.\n",
       " * \n",
       " * @param {string} code Python code in string form\n",
       " * @returns Passed code string but with no leading indentations\n",
       " */\n",
       "function unindentPyCode(code){\n",
       "    let uicode = code.split('\\n');\n",
       "    let indent = 0;\n",
       "\n",
       "    uicode.forEach((l,i, arr)=>{\n",
       "        if(i == 0){\n",
       "            indent = l.search(/\\S/);\n",
       "        }\n",
       "        arr[i] = l.slice(indent);\n",
       "    })\n",
       "    uicode = uicode.join('\\n');\n",
       "    return uicode;\n",
       "}\n",
       "\n",
       "/**\n",
       " * @name buildPythonAssignment\n",
       " * @description Builds up a python code string which assigns javascript data back into jypyter notebook namespace\n",
       " * \n",
       " * @param {string} val This is data assigned back to the python code\n",
       " * @param {string} py_var This is the variable into which val is assigned\n",
       " * @param {string} converter This is a definition of a python function which translates data back to the desired format\n",
       " * @returns The python code to be run in the jupyter shell\n",
       " */\n",
       "function buildPythonAssignment(val, py_var, converter){\n",
       "    // console.log(val, py_var, converter);\n",
       "    var holder = `'${val}'`;\n",
       "    var code = `${unindentPyCode(converter.code)}`\n",
       "    code += `\\ntmp = ${holder}`;\n",
       "    code += `\\n${py_var} = ${converter.name}(tmp)`\n",
       "\n",
       "    return code\n",
       "}\n",
       "\n",
       "/**\n",
       " * @name manageNewCell\n",
       " * \n",
       " * @description  Increments all two way bound cell ids by the number of new cells which proceed them. \n",
       " *  Ex. Adding one cell at position 2 will increment a bound cell at position 3 from 3->4. \n",
       " * \n",
       " * @param {array} newCells A list of our current cells in the notebook to be compared against cached cells\n",
       " * @param {} obj The current roundtrip object containing all data bindings\n",
       " */\n",
       "function manageNewCell(newCells, obj){\n",
       "    let newIds = [];\n",
       "\n",
       "    Object.keys(newCells).forEach(function(i){\n",
       "        if(!Object.values(cached_cells).includes(newCells[i]) && !isNaN(i)){\n",
       "            newIds.push(i);\n",
       "        }\n",
       "    });\n",
       "\n",
       "    //increment all bindings past each new id\n",
       "    for(let js_var in obj){\n",
       "        for(let id of newIds){\n",
       "            for(let key in obj[js_var][\"two_way\"]){\n",
       "                obj[js_var][\"two_way\"][key].forEach((two_way_id, i) => {\n",
       "                    if(two_way_id > id){\n",
       "                        obj[js_var][\"two_way\"][key][i] += 1;\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "        }   \n",
       "    }\n",
       "\n",
       "    cached_cells = newCells;\n",
       "}\n",
       "\n",
       "function manageDeletedCell(newCells, obj){\n",
       "    let deletedId = null;\n",
       "    \n",
       "    for(i of Object.keys(cachedCells)){\n",
       "        if (cached_cells[i] !== newCells[i]){\n",
       "            deletedId = i;\n",
       "            break;\n",
       "        }\n",
       "    }\n",
       "\n",
       "}\n",
       "\n",
       "\n",
       "function bindClickDetectToCells(){\n",
       "    let cells = Jupyter.notebook.get_cell_elements();\n",
       "\n",
       "    for(let i in Object.keys(cells)){\n",
       "        let cell = cells[i];\n",
       "\n",
       "        if(cell !== undefined){\n",
       "            cell.addEventListener('mousedown', () => {\n",
       "                    clicked_cell = i;\n",
       "                }, true)\n",
       "        }\n",
       "    }\n",
       "}\n",
       "\n",
       "bindClickDetectToCells();\n",
       "\n",
       "/**\n",
       " * @name RT_Handler\n",
       " * @description A wrapper for our roundtrip object. It is called as a proxy for the\n",
       " *      roundtrip object defined above. This enables us to define custom call backs for\n",
       " *      gets and sets on the roundtrip object. The custom set handles necessary data conversion,\n",
       " *      the registering of two-way bound variables and automatic updating of watched cells. The get\n",
       " *      allows users to interact with the underlying object without worrying about the proxy.\n",
       " */\n",
       "var RT_Handler = {\n",
       "    set(obj, prop, value){\n",
       "        //Do cell housekeeping\n",
       "\n",
       "\n",
       "        //Initial pass of value into roundtrip object\n",
       "        // from python code; there may be multiple different\n",
       "        // visualizations of the same type we need to catch\n",
       "        if (typeof value === 'object' && value.hasOwnProperty('origin') && value.origin == 'INIT'){\n",
       "            \n",
       "            /**\n",
       "             * In this code block we need to check if there is already a \n",
       "             * an array of id's which are two way bound already defined and \n",
       "             * add to it or remove from it\n",
       "             */\n",
       "            let ida = Jupyter.notebook.get_selected_index()-1;\n",
       "            value.id = ida;\n",
       "            let new_val = value;\n",
       "\n",
       "            // Block updating bindings while jupyter is running\n",
       "            if(refresh_cycle){\n",
       "                new_val = obj[prop];\n",
       "                new_val.data = value.data;\n",
       "                return Reflect.set(obj, prop, new_val);\n",
       "            }\n",
       "\n",
       "            /**\n",
       "             * The broad case where we are updating bindings \n",
       "             * on existing data\n",
       "             */\n",
       "            if(obj[prop] != undefined){\n",
       "                new_val = obj[prop];\n",
       "                new_val.data = value.data;\n",
       "                new_val.converter = value.converter;\n",
       "\n",
       "                // If there is no two way array, create one\n",
       "                // Else push on our new id\n",
       "                if(value.two_way === true){\n",
       "                    if(!Object.keys(new_val.two_way).includes(value['python_var'])){\n",
       "                        new_val.two_way[value['python_var']] = [];\n",
       "                    }\n",
       "\n",
       "                    let pybinding = new_val.two_way[value['python_var']];\n",
       "\n",
       "                    if(!pybinding.includes(value.id)){\n",
       "                        pybinding.push(value.id);\n",
       "                    }\n",
       "\n",
       "                }\n",
       "\n",
       "                //Deregister a cell id from being two-way bound now\n",
       "                else if(value.two_way === false && Object.keys(new_val.two_way).includes(value['python_var'])){\n",
       "                    let pybinding = new_val.two_way[value['python_var']];\n",
       "                    const index = pybinding.indexOf(value.id);\n",
       "                    \n",
       "                    if (index > -1) {\n",
       "                        pybinding.splice(index, 1);\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "\n",
       "            //Initalize a new two-way object if\n",
       "            // one did not exist\n",
       "            else{\n",
       "                if(new_val.two_way == true){\n",
       "                    new_val.two_way = {};\n",
       "                    new_val.two_way[value['python_var']] = [value.id];\n",
       "                }\n",
       "                else{\n",
       "                    new_val.two_way = {};\n",
       "                }\n",
       "                delete new_val.id;\n",
       "                delete new_val.from_py;\n",
       "                delete new_val.python_var;\n",
       "            }\n",
       "\n",
       "            return Reflect.set(obj, prop, new_val);\n",
       "        }\n",
       "        //Assignment from javascript code\n",
       "        else {\n",
       "            // TODO: make the py/js data identification object a\n",
       "            // formal class\n",
       "            if(obj[prop] === undefined){\n",
       "                obj[prop] = {\n",
       "                    two_way: {},\n",
       "                    origin: \"JS\",\n",
       "                    data: null,\n",
       "                    python_var: \"\",\n",
       "                    converter: null,\n",
       "                    type: typeof(value)\n",
       "                }\n",
       "            }\n",
       "\n",
       "            var execable_cells = [];\n",
       "            let origin = 'STANDARD';\n",
       "            let python_var = '';\n",
       "\n",
       "            if (typeof value === 'object' && \n",
       "                value.hasOwnProperty('origin') && \n",
       "                value.origin == 'PYASSIGN'){\n",
       "\n",
       "                origin = value.origin;\n",
       "                python_var = value.python_var;\n",
       "                value = value.data;\n",
       "            }\n",
       "\n",
       "            //TODO: Replace with imported, webpacked D3\n",
       "            require(['https://d3js.org/d3.v4.min.js'], function(d3) {\n",
       "\n",
       "                // When 2 way bound this calls automatically when something changes\n",
       "                if (obj[prop] !== undefined && Object.keys(obj[prop][\"two_way\"]).length > 0){\n",
       "\n",
       "                    let current_cell = Number(clicked_cell);\n",
       "                    let py_var = '';\n",
       "\n",
       "                    //ust set the data without updating if our current cell is not two way bound\n",
       "                    if(origin == 'STANDARD'){\n",
       "                        let found = false;\n",
       "                        for(let key in obj[prop][\"two_way\"]){\n",
       "                            if (obj[prop][\"two_way\"][key].includes(current_cell)){\n",
       "                                found = true;\n",
       "                                py_var = key;\n",
       "                            }\n",
       "                        }\n",
       "\n",
       "                        if(!found){\n",
       "                            return Reflect.set(obj[prop], \"data\", value);\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "\n",
       "                    if(origin == 'PYASSIGN'){\n",
       "                        py_var = python_var;\n",
       "                    }\n",
       "\n",
       "\n",
       "                    /**\n",
       "                     * We now have a list of registered cells we can execute.\n",
       "                     * So we look through our javascript variables to see if they\n",
       "                     * are bound to the same py variable as our current assignment\n",
       "                     * TODO: Make this list update when cells are moved up or down\n",
       "                     */\n",
       "\n",
       "                    for(let js_var in obj){\n",
       "                        let boundpyvars =  Object.keys(obj[js_var][\"two_way\"]);\n",
       "\n",
       "                        if(boundpyvars.includes(py_var)){\n",
       "                            let clls = obj[js_var][\"two_way\"][py_var].filter(x => x != current_cell );\n",
       "                            execable_cells = execable_cells.concat(clls);\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                    if(origin == 'STANDARD'){\n",
       "                        // TODO:THROW AN ERROR IF CONVERTER == NONE\n",
       "                        const code = buildPythonAssignment(value, py_var, obj[prop][\"converter\"]);\n",
       "                        \n",
       "                        //TODO: Turn this into a function that manages error reporting and printing\n",
       "                        Jupyter.notebook.kernel.execute(code, { \n",
       "                                                            shell:{\n",
       "                                                                reply: function(r){\n",
       "                                                                    //consider putting this in a reserved jupyter variable\n",
       "                                                                    if(r.content.status == 'error'){\n",
       "                                                                        console.error(`${r.content.ename} in JS->Python coversion:\\n ${r.content.evalue}`)\n",
       "                                                                    }\n",
       "                                                                }\n",
       "                                                            }\n",
       "                                                        });\n",
       "                    }\n",
       "\n",
       "                    refresh_cycle = true;\n",
       "                    Jupyter.notebook.execute_cells(execable_cells);\n",
       "\n",
       "                    /**\n",
       "                     * Test every half second to see if some of the\n",
       "                     * jupyter cells are still running. Avoids a race condition\n",
       "                     * where incorrect ids were stored in our roundtrip object.\n",
       "                     */\n",
       "                    const test_running = function(){\n",
       "                        let runtest = d3.selectAll(\".running\");\n",
       "                        if(runtest.empty()){\n",
       "                            refresh_cycle = false;\n",
       "                            return;\n",
       "                        }\n",
       "                        else{\n",
       "                            setTimeout(test_running, 500);\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                    test_running();\n",
       "                }\n",
       "\n",
       "            });\n",
       "        }   \n",
       "\n",
       "        return Reflect.set(obj[prop], \"data\", value);\n",
       "    },\n",
       "    get(obj, prop, reciever){\n",
       "        let ret = obj[prop].data\n",
       "        return ret;  \n",
       "    }\n",
       "}\n",
       "\n",
       "window.Roundtrip = new Proxy(Roundtrip_Obj, RT_Handler);\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.container { width:80% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import display\n",
    "from IPython.display import HTML\n",
    "import hatchet as ht\n",
    "\n",
    "import thicket as tt\n",
    "\n",
    "display(HTML(\"<style>.container { width:80% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "841835d6",
   "metadata": {
    "papermill": {
     "duration": 0.001886,
     "end_time": "2024-04-05T18:05:55.035680",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.033794",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 2. Read in Performance Profiles\n",
    "\n",
    "For this notebook, we select profiles generated on Lawrence Livermore National Lab (LLNL) machine, lassen. We create a thicket object generated with the same block size of 128. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7643e225",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.039815Z",
     "iopub.status.busy": "2024-04-05T18:05:55.039655Z",
     "iopub.status.idle": "2024-04-05T18:05:55.146991Z",
     "shell.execute_reply": "2024-04-05T18:05:55.146528Z"
    },
    "papermill": {
     "duration": 0.110348,
     "end_time": "2024-04-05T18:05:55.147802",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.037454",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.9/site-packages/thicket/ensemble.py:319: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  perfdata[col].replace({fill_value: None}, inplace=True)\n"
     ]
    }
   ],
   "source": [
    "problem_sizes = [\n",
    "    \"1048576\", \n",
    "    \"2097152\", \n",
    "    \"4194304\", \n",
    "    \"8388608\"\n",
    "]\n",
    "lassen1 = [f\"../data/lassen/clang10.0.1_nvcc10.2.89_{x}/Base_CUDA-block_128.cali\" for x in problem_sizes]\n",
    "lassen2 = [f\"../data/lassen/clang10.0.1_nvcc10.2.89_1048576/Base_CUDA-block_256.cali\"]\n",
    "\n",
    "# generate thicket(s)\n",
    "th_lassen = tt.Thicket.from_caliperreader(lassen1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc16d40e",
   "metadata": {
    "papermill": {
     "duration": 0.002183,
     "end_time": "2024-04-05T18:05:55.152389",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.150206",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 3. More Information on a Function\n",
    "***\n",
    "You can use the help() method within Python to see the information for a given object. You can do this by typing help(object). \n",
    "This will allow you to see the arguments for the function, and what will be returned. An example is below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6f7e0bb9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.157443Z",
     "iopub.status.busy": "2024-04-05T18:05:55.157337Z",
     "iopub.status.idle": "2024-04-05T18:05:55.159744Z",
     "shell.execute_reply": "2024-04-05T18:05:55.159438Z"
    },
    "papermill": {
     "duration": 0.005802,
     "end_time": "2024-04-05T18:05:55.160591",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.154789",
     "status": "completed"
    },
    "scrolled": false,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function median in module thicket.stats.median:\n",
      "\n",
      "median(thicket, columns=None)\n",
      "    Calculate the median for each node in the performance data table.\n",
      "    \n",
      "    Designed to take in a thicket, and append one or more columns to the\n",
      "    aggregated statistics table for the median calculation for each node.\n",
      "    \n",
      "    Arguments:\n",
      "        thicket (thicket): Thicket object\n",
      "        columns (list): List of hardware/timing metrics to perform median calculation\n",
      "            on. Note, if using a columnar joined thicket a list of tuples must be passed\n",
      "            in with the format (column index, column name).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(tt.median)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83a23b65",
   "metadata": {
    "papermill": {
     "duration": 0.002013,
     "end_time": "2024-04-05T18:05:55.164866",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.162853",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 4. Append Statistical Calculation(s)\n",
    "***\n",
    "\n",
    "We can calculate statistical aggregations per-node in the performance data and append the values to the aggregated statistics table. In the example below, we calculate the per-node median time across 4 profiles and append the median to the statistics table. The new column is called `Total time_median`. \n",
    "\n",
    "Why is this important for this notebook?\n",
    "\n",
    "When the nodes in the performance data table change, the aggregated statistics table will change depending on the metric. Therefore, the aggregated statistics table is cleared after a query has been applied. In the examples further down, we use an appended column (specifically the median of total time) as the metric to print the call trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "140ad4a4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.169173Z",
     "iopub.status.busy": "2024-04-05T18:05:55.169072Z",
     "iopub.status.idle": "2024-04-05T18:05:55.176956Z",
     "shell.execute_reply": "2024-04-05T18:05:55.176627Z"
    },
    "papermill": {
     "duration": 0.010978,
     "end_time": "2024-04-05T18:05:55.177723",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.166745",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.9/site-packages/thicket/stats/median.py:32: FutureWarning: The provided callable <function median at 0xffff4fe9adc0> is currently using DataFrameGroupBy.median. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string \"median\" instead.\n",
      "  df = thicket.dataframe[columns].reset_index().groupby(\"node\").agg(np.median)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>Total time_median</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>node</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>{'name': 'RAJAPerf', 'type': 'function'}</th>\n",
       "      <td>RAJAPerf</td>\n",
       "      <td>5.073698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Algorithm', 'type': 'function'}</th>\n",
       "      <td>Algorithm</td>\n",
       "      <td>0.015522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Algorithm_MEMCPY', 'type': 'function'}</th>\n",
       "      <td>Algorithm_MEMCPY</td>\n",
       "      <td>0.006589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Algorithm_MEMSET', 'type': 'function'}</th>\n",
       "      <td>Algorithm_MEMSET</td>\n",
       "      <td>0.004223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Algorithm_REDUCE_SUM', 'type': 'function'}</th>\n",
       "      <td>Algorithm_REDUCE_SUM</td>\n",
       "      <td>0.004685</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Stream_ADD', 'type': 'function'}</th>\n",
       "      <td>Stream_ADD</td>\n",
       "      <td>0.093114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Stream_COPY', 'type': 'function'}</th>\n",
       "      <td>Stream_COPY</td>\n",
       "      <td>0.117469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Stream_DOT', 'type': 'function'}</th>\n",
       "      <td>Stream_DOT</td>\n",
       "      <td>0.182254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Stream_MUL', 'type': 'function'}</th>\n",
       "      <td>Stream_MUL</td>\n",
       "      <td>0.117501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>{'name': 'Stream_TRIAD', 'type': 'function'}</th>\n",
       "      <td>Stream_TRIAD</td>\n",
       "      <td>0.093277</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    name  \\\n",
       "node                                                                       \n",
       "{'name': 'RAJAPerf', 'type': 'function'}                        RAJAPerf   \n",
       "{'name': 'Algorithm', 'type': 'function'}                      Algorithm   \n",
       "{'name': 'Algorithm_MEMCPY', 'type': 'function'}        Algorithm_MEMCPY   \n",
       "{'name': 'Algorithm_MEMSET', 'type': 'function'}        Algorithm_MEMSET   \n",
       "{'name': 'Algorithm_REDUCE_SUM', 'type': 'funct...  Algorithm_REDUCE_SUM   \n",
       "...                                                                  ...   \n",
       "{'name': 'Stream_ADD', 'type': 'function'}                    Stream_ADD   \n",
       "{'name': 'Stream_COPY', 'type': 'function'}                  Stream_COPY   \n",
       "{'name': 'Stream_DOT', 'type': 'function'}                    Stream_DOT   \n",
       "{'name': 'Stream_MUL', 'type': 'function'}                    Stream_MUL   \n",
       "{'name': 'Stream_TRIAD', 'type': 'function'}                Stream_TRIAD   \n",
       "\n",
       "                                                    Total time_median  \n",
       "node                                                                   \n",
       "{'name': 'RAJAPerf', 'type': 'function'}                     5.073698  \n",
       "{'name': 'Algorithm', 'type': 'function'}                    0.015522  \n",
       "{'name': 'Algorithm_MEMCPY', 'type': 'function'}             0.006589  \n",
       "{'name': 'Algorithm_MEMSET', 'type': 'function'}             0.004223  \n",
       "{'name': 'Algorithm_REDUCE_SUM', 'type': 'funct...           0.004685  \n",
       "...                                                               ...  \n",
       "{'name': 'Stream_ADD', 'type': 'function'}                   0.093114  \n",
       "{'name': 'Stream_COPY', 'type': 'function'}                  0.117469  \n",
       "{'name': 'Stream_DOT', 'type': 'function'}                   0.182254  \n",
       "{'name': 'Stream_MUL', 'type': 'function'}                   0.117501  \n",
       "{'name': 'Stream_TRIAD', 'type': 'function'}                 0.093277  \n",
       "\n",
       "[64 rows x 2 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics = [\"Total time\"]\n",
    "tt.median(th_lassen, columns=metrics)\n",
    "th_lassen.statsframe.dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d603a02a",
   "metadata": {
    "papermill": {
     "duration": 0.002117,
     "end_time": "2024-04-05T18:05:55.181950",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.179833",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 5. Thicket Query Language \n",
    "\n",
    "**Use the Query Language**\n",
    "\n",
    "Thicket's query language provides users the capability to select or `query` specific nodes based on the call tree component in thicket. The nodes in the performance data and statistics table are updated as well to reflect which nodes are remaining in the call tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a1212ef4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.186637Z",
     "iopub.status.busy": "2024-04-05T18:05:55.186543Z",
     "iopub.status.idle": "2024-04-05T18:05:55.192822Z",
     "shell.execute_reply": "2024-04-05T18:05:55.192478Z"
    },
    "papermill": {
     "duration": 0.009515,
     "end_time": "2024-04-05T18:05:55.193567",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.184052",
     "status": "completed"
    },
    "scrolled": false,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial call tree:\n",
      "    __          __       __         __ \n",
      "   / /_  ____ _/ /______/ /_  ___  / /_\n",
      "  / __ \\/ __ `/ __/ ___/ __ \\/ _ \\/ __/\n",
      " / / / / /_/ / /_/ /__/ / / /  __/ /_  \n",
      "/_/ /_/\\__,_/\\__/\\___/_/ /_/\\___/\\__/  v2024.1.0\n",
      "\n",
      "\u001b[38;5;196m5.074\u001b[0m RAJAPerf\u001b[0m\n",
      "├─ \u001b[38;5;22m0.016\u001b[0m Algorithm\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.007\u001b[0m Algorithm_MEMCPY\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.004\u001b[0m Algorithm_MEMSET\u001b[0m\n",
      "│  └─ \u001b[38;5;22m0.005\u001b[0m Algorithm_REDUCE_SUM\u001b[0m\n",
      "├─ \u001b[38;5;22m0.436\u001b[0m Apps\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.020\u001b[0m Apps_DEL_DOT_VEC_2D\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.112\u001b[0m Apps_ENERGY\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.011\u001b[0m Apps_FIR\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.035\u001b[0m Apps_HALOEXCHANGE\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.007\u001b[0m Apps_HALOEXCHANGE_FUSED\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.035\u001b[0m Apps_LTIMES\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.035\u001b[0m Apps_LTIMES_NOVIEW\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.021\u001b[0m Apps_NODAL_ACCUMULATION_3D\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.134\u001b[0m Apps_PRESSURE\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.016\u001b[0m Apps_VOL3D\u001b[0m\n",
      "│  └─ \u001b[38;5;22m0.010\u001b[0m Apps_ZONAL_ACCUMULATION_3D\u001b[0m\n",
      "├─ \u001b[38;5;34m0.936\u001b[0m Basic\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.025\u001b[0m Basic_COPY8\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.047\u001b[0m Basic_DAXPY\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.047\u001b[0m Basic_DAXPY_ATOMIC\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.036\u001b[0m Basic_IF_QUAD\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.080\u001b[0m Basic_INIT3\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.100\u001b[0m Basic_INIT_VIEW1D\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.095\u001b[0m Basic_INIT_VIEW1D_OFFSET\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.056\u001b[0m Basic_MULADDSUB\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.045\u001b[0m Basic_NESTED_INIT\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.342\u001b[0m Basic_PI_ATOMIC\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.004\u001b[0m Basic_PI_REDUCE\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.004\u001b[0m Basic_REDUCE3_INT\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.051\u001b[0m Basic_REDUCE_STRUCT\u001b[0m\n",
      "│  └─ \u001b[38;5;22m0.004\u001b[0m Basic_TRAP_INT\u001b[0m\n",
      "├─ \u001b[38;5;34m1.095\u001b[0m Lcals\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.178\u001b[0m Lcals_DIFF_PREDICT\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.064\u001b[0m Lcals_EOS\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.131\u001b[0m Lcals_FIRST_DIFF\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.010\u001b[0m Lcals_FIRST_MIN\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.132\u001b[0m Lcals_FIRST_SUM\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.151\u001b[0m Lcals_GEN_LIN_RECUR\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.094\u001b[0m Lcals_HYDRO_1D\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.065\u001b[0m Lcals_HYDRO_2D\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.137\u001b[0m Lcals_INT_PREDICT\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.008\u001b[0m Lcals_PLANCKIAN\u001b[0m\n",
      "│  └─ \u001b[38;5;22m0.125\u001b[0m Lcals_TRIDIAG_ELIM\u001b[0m\n",
      "├─ \u001b[38;5;46m1.987\u001b[0m Polybench\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.023\u001b[0m Polybench_2MM\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.032\u001b[0m Polybench_3MM\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.068\u001b[0m Polybench_ADI\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.046\u001b[0m Polybench_ATAX\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.101\u001b[0m Polybench_FDTD_2D\u001b[0m\n",
      "│  ├─ \u001b[38;5;34m1.038\u001b[0m Polybench_FLOYD_WARSHALL\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.027\u001b[0m Polybench_GEMM\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.013\u001b[0m Polybench_GEMVER\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.047\u001b[0m Polybench_GESUMMV\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.059\u001b[0m Polybench_HEAT_3D\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.211\u001b[0m Polybench_JACOBI_1D\u001b[0m\n",
      "│  ├─ \u001b[38;5;22m0.282\u001b[0m Polybench_JACOBI_2D\u001b[0m\n",
      "│  └─ \u001b[38;5;22m0.039\u001b[0m Polybench_MVT\u001b[0m\n",
      "└─ \u001b[38;5;34m0.604\u001b[0m Stream\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.093\u001b[0m Stream_ADD\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.117\u001b[0m Stream_COPY\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.182\u001b[0m Stream_DOT\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.118\u001b[0m Stream_MUL\u001b[0m\n",
      "   └─ \u001b[38;5;22m0.093\u001b[0m Stream_TRIAD\u001b[0m\n",
      "\n",
      "\u001b[4mLegend\u001b[0m (Metric: Total time_median Min: 0.00 Max: 5.07)\n",
      "\u001b[38;5;196m█ \u001b[0m4.57 - 5.07\n",
      "\u001b[38;5;208m█ \u001b[0m3.55 - 4.57\n",
      "\u001b[38;5;220m█ \u001b[0m2.54 - 3.55\n",
      "\u001b[38;5;46m█ \u001b[0m1.52 - 2.54\n",
      "\u001b[38;5;34m█ \u001b[0m0.51 - 1.52\n",
      "\u001b[38;5;22m█ \u001b[0m0.00 - 0.51\n",
      "\n",
      "name\u001b[0m User code    \u001b[38;5;160m◀ \u001b[0m Only in left graph    \u001b[38;5;28m▶ \u001b[0m Only in right graph\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Initial call tree:\")\n",
    "print(th_lassen.statsframe.tree(\"Total time_median\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71c35593",
   "metadata": {
    "papermill": {
     "duration": 0.002101,
     "end_time": "2024-04-05T18:05:55.197905",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.195804",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Example Query 1: Find a Subgraph with a Specific Root\n",
    "\n",
    "This example shows how to find a subtree starting with a specific root. More specifically, the query in this example finds a subtree rooted at the node with the name \"Stream\" followed by all nodes down to the leaf nodes.\n",
    "\n",
    "NOTE: A DeprecationWarning is generated when using “old-style” queries (i.e., queries with QueryMatcher) if you have the newest version of Hatchet installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6a2f4c62",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.202502Z",
     "iopub.status.busy": "2024-04-05T18:05:55.202413Z",
     "iopub.status.idle": "2024-04-05T18:05:55.262857Z",
     "shell.execute_reply": "2024-04-05T18:05:55.262475Z"
    },
    "papermill": {
     "duration": 0.063784,
     "end_time": "2024-04-05T18:05:55.263678",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.199894",
     "status": "completed"
    },
    "scrolled": false,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    __          __       __         __ \n",
      "   / /_  ____ _/ /______/ /_  ___  / /_\n",
      "  / __ \\/ __ `/ __/ ___/ __ \\/ _ \\/ __/\n",
      " / / / / /_/ / /_/ /__/ / / /  __/ /_  \n",
      "/_/ /_/\\__,_/\\__/\\___/_/ /_/\\___/\\__/  v2024.1.0\n",
      "\n",
      "\u001b[38;5;196m0.604\u001b[0m Stream\u001b[0m\n",
      "├─ \u001b[38;5;22m0.093\u001b[0m Stream_ADD\u001b[0m\n",
      "├─ \u001b[38;5;22m0.117\u001b[0m Stream_COPY\u001b[0m\n",
      "├─ \u001b[38;5;34m0.182\u001b[0m Stream_DOT\u001b[0m\n",
      "├─ \u001b[38;5;22m0.118\u001b[0m Stream_MUL\u001b[0m\n",
      "└─ \u001b[38;5;22m0.093\u001b[0m Stream_TRIAD\u001b[0m\n",
      "\n",
      "\u001b[4mLegend\u001b[0m (Metric: Total time_median Min: 0.09 Max: 0.60)\n",
      "\u001b[38;5;196m█ \u001b[0m0.55 - 0.60\n",
      "\u001b[38;5;208m█ \u001b[0m0.45 - 0.55\n",
      "\u001b[38;5;220m█ \u001b[0m0.35 - 0.45\n",
      "\u001b[38;5;46m█ \u001b[0m0.25 - 0.35\n",
      "\u001b[38;5;34m█ \u001b[0m0.14 - 0.25\n",
      "\u001b[38;5;22m█ \u001b[0m0.09 - 0.14\n",
      "\n",
      "name\u001b[0m User code    \u001b[38;5;160m◀ \u001b[0m Only in left graph    \u001b[38;5;28m▶ \u001b[0m Only in right graph\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-6-322ae67271ad>:2: DeprecationWarning: Old-style queries are deprecated as of Hatchet 2023.1.0 and will be removed in the             future. Please use new-style queries instead. For QueryMatcher, the equivalent             new-style queries are hatchet.query.Query for base-syntax queries and             hatchet.query.ObjectQuery for the object-dialect.\n",
      "  ht.QueryMatcher()\n",
      "/opt/conda/lib/python3.9/site-packages/thicket/stats/median.py:32: FutureWarning: The provided callable <function median at 0xffff4fe9adc0> is currently using DataFrameGroupBy.median. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string \"median\" instead.\n",
      "  df = thicket.dataframe[columns].reset_index().groupby(\"node\").agg(np.median)\n"
     ]
    }
   ],
   "source": [
    "query_ex1 = (\n",
    "    ht.QueryMatcher()\n",
    "    .match (\n",
    "        \".\", \n",
    "        lambda row: row[\"name\"].apply(\n",
    "            lambda x: re.match(\n",
    "                \"Stream\", x\n",
    "            )\n",
    "            is not None\n",
    "        ).all()\n",
    "    )\n",
    "    .rel(\"*\")\n",
    ")\n",
    "\n",
    "# applying the first query on the lassen thicket\n",
    "th_ex1 = th_lassen.query(query_ex1)\n",
    "tt.median(th_ex1, columns=[\"Total time\"])\n",
    "print(th_ex1.statsframe.tree(\"Total time_median\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39c03f96",
   "metadata": {
    "papermill": {
     "duration": 0.002323,
     "end_time": "2024-04-05T18:05:55.268636",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.266313",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Example Query 2: Find All Paths Ending with a Specific Node\n",
    "\n",
    "This example shows how to find all paths of a GraphFrame ending with a specific node. More specifically, the queries in this example can be used to find paths ending with a node named \"Stream\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "037f5ad7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.273762Z",
     "iopub.status.busy": "2024-04-05T18:05:55.273659Z",
     "iopub.status.idle": "2024-04-05T18:05:55.346867Z",
     "shell.execute_reply": "2024-04-05T18:05:55.346447Z"
    },
    "papermill": {
     "duration": 0.076683,
     "end_time": "2024-04-05T18:05:55.347614",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.270931",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-7-d669d2fda245>:2: DeprecationWarning: Old-style queries are deprecated as of Hatchet 2023.1.0 and will be removed in the             future. Please use new-style queries instead. For QueryMatcher, the equivalent             new-style queries are hatchet.query.Query for base-syntax queries and             hatchet.query.ObjectQuery for the object-dialect.\n",
      "  ht.QueryMatcher()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    __          __       __         __ \n",
      "   / /_  ____ _/ /______/ /_  ___  / /_\n",
      "  / __ \\/ __ `/ __/ ___/ __ \\/ _ \\/ __/\n",
      " / / / / /_/ / /_/ /__/ / / /  __/ /_  \n",
      "/_/ /_/\\__,_/\\__/\\___/_/ /_/\\___/\\__/  v2024.1.0\n",
      "\n",
      "\u001b[38;5;196m5.074\u001b[0m RAJAPerf\u001b[0m\n",
      "└─ \u001b[38;5;34m0.604\u001b[0m Stream\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.093\u001b[0m Stream_ADD\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.117\u001b[0m Stream_COPY\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.182\u001b[0m Stream_DOT\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.118\u001b[0m Stream_MUL\u001b[0m\n",
      "   └─ \u001b[38;5;22m0.093\u001b[0m Stream_TRIAD\u001b[0m\n",
      "\n",
      "\u001b[4mLegend\u001b[0m (Metric: Total time_median Min: 0.09 Max: 5.07)\n",
      "\u001b[38;5;196m█ \u001b[0m4.58 - 5.07\n",
      "\u001b[38;5;208m█ \u001b[0m3.58 - 4.58\n",
      "\u001b[38;5;220m█ \u001b[0m2.58 - 3.58\n",
      "\u001b[38;5;46m█ \u001b[0m1.59 - 2.58\n",
      "\u001b[38;5;34m█ \u001b[0m0.59 - 1.59\n",
      "\u001b[38;5;22m█ \u001b[0m0.09 - 0.59\n",
      "\n",
      "name\u001b[0m User code    \u001b[38;5;160m◀ \u001b[0m Only in left graph    \u001b[38;5;28m▶ \u001b[0m Only in right graph\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.9/site-packages/thicket/stats/median.py:32: FutureWarning: The provided callable <function median at 0xffff4fe9adc0> is currently using DataFrameGroupBy.median. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string \"median\" instead.\n",
      "  df = thicket.dataframe[columns].reset_index().groupby(\"node\").agg(np.median)\n"
     ]
    }
   ],
   "source": [
    "query_ex2 = (\n",
    "    ht.QueryMatcher()\n",
    "    .match(\"*\")\n",
    "    .rel(\n",
    "        \".\",\n",
    "        lambda row: row[\"name\"].apply(\n",
    "            lambda x: re.match(\n",
    "                \"Stream\", x\n",
    "            )\n",
    "            is not None\n",
    "        ).all()\n",
    "    )\n",
    ")\n",
    "\n",
    "# applying the second query on the lassen thicket\n",
    "th_ex2 = th_lassen.query(query_ex2)\n",
    "tt.median(th_ex2, columns=[\"Total time\"])\n",
    "print(th_ex2.statsframe.tree(\"Total time_median\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c70ab25",
   "metadata": {
    "papermill": {
     "duration": 0.002436,
     "end_time": "2024-04-05T18:05:55.352750",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.350314",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Example Query 3: Find All Paths with Specific Starting and Ending Nodes\n",
    "\n",
    "This example shows how to find all call paths starting with and ending with specific nodes. More specifically, the query in this example finds paths starting with a node named \"Stream\" and ending with a node named \"Stream_MUL\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "28ae2678",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.358104Z",
     "iopub.status.busy": "2024-04-05T18:05:55.358011Z",
     "iopub.status.idle": "2024-04-05T18:05:55.431212Z",
     "shell.execute_reply": "2024-04-05T18:05:55.430670Z"
    },
    "papermill": {
     "duration": 0.076661,
     "end_time": "2024-04-05T18:05:55.431853",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.355192",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-8-325f62c07381>:2: DeprecationWarning: Old-style queries are deprecated as of Hatchet 2023.1.0 and will be removed in the             future. Please use new-style queries instead. For QueryMatcher, the equivalent             new-style queries are hatchet.query.Query for base-syntax queries and             hatchet.query.ObjectQuery for the object-dialect.\n",
      "  ht.QueryMatcher()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    __          __       __         __ \n",
      "   / /_  ____ _/ /______/ /_  ___  / /_\n",
      "  / __ \\/ __ `/ __/ ___/ __ \\/ _ \\/ __/\n",
      " / / / / /_/ / /_/ /__/ / / /  __/ /_  \n",
      "/_/ /_/\\__,_/\\__/\\___/_/ /_/\\___/\\__/  v2024.1.0\n",
      "\n",
      "\u001b[38;5;196m0.604\u001b[0m Stream\u001b[0m\n",
      "└─ \u001b[38;5;22m0.118\u001b[0m Stream_MUL\u001b[0m\n",
      "\n",
      "\u001b[4mLegend\u001b[0m (Metric: Total time_median Min: 0.12 Max: 0.60)\n",
      "\u001b[38;5;196m█ \u001b[0m0.56 - 0.60\n",
      "\u001b[38;5;208m█ \u001b[0m0.46 - 0.56\n",
      "\u001b[38;5;220m█ \u001b[0m0.36 - 0.46\n",
      "\u001b[38;5;46m█ \u001b[0m0.26 - 0.36\n",
      "\u001b[38;5;34m█ \u001b[0m0.17 - 0.26\n",
      "\u001b[38;5;22m█ \u001b[0m0.12 - 0.17\n",
      "\n",
      "name\u001b[0m User code    \u001b[38;5;160m◀ \u001b[0m Only in left graph    \u001b[38;5;28m▶ \u001b[0m Only in right graph\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.9/site-packages/thicket/stats/median.py:32: FutureWarning: The provided callable <function median at 0xffff4fe9adc0> is currently using DataFrameGroupBy.median. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string \"median\" instead.\n",
      "  df = thicket.dataframe[columns].reset_index().groupby(\"node\").agg(np.median)\n"
     ]
    }
   ],
   "source": [
    "query_ex3 = (\n",
    "    ht.QueryMatcher()\n",
    "    .match(\n",
    "        \".\",\n",
    "        lambda row: row[\"name\"].apply(\n",
    "            lambda x: re.match(\n",
    "                \"Stream\", x\n",
    "            )\n",
    "            is not None\n",
    "        ).all()\n",
    "    )\n",
    "    .rel(\"*\")\n",
    "    .rel(\n",
    "        \".\",\n",
    "        lambda row: row[\"name\"].apply(\n",
    "            lambda x: re.match(\n",
    "                \"Stream_MUL\", x\n",
    "            )\n",
    "            is not None\n",
    "        ).all()\n",
    "    )\n",
    ")\n",
    "\n",
    "# applying the third query on the lassen thicket\n",
    "th_ex3 = th_lassen.query(query_ex3)\n",
    "tt.median(th_ex3, columns=[\"Total time\"])\n",
    "print(th_ex3.statsframe.tree(\"Total time_median\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa358ba0",
   "metadata": {
    "papermill": {
     "duration": 0.002456,
     "end_time": "2024-04-05T18:05:55.437202",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.434746",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Example Query 4: Find All Nodes for a Particular Software Library\n",
    "\n",
    "This example shows how to find all call paths representing a specific software library. This example is simply a variant of finding a subtree with a given root (i.e., from :ref:`this section <subgraph_root_ex>`). The example query below can be adapted to find the nodes for a subset of the MPI library, for example. In our example, we look for subtrees rooted at PolyBench_2MM, Basic_DAXPY, and Apps_ENERGY."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "99a509de",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.442508Z",
     "iopub.status.busy": "2024-04-05T18:05:55.442417Z",
     "iopub.status.idle": "2024-04-05T18:05:55.525840Z",
     "shell.execute_reply": "2024-04-05T18:05:55.525437Z"
    },
    "papermill": {
     "duration": 0.087018,
     "end_time": "2024-04-05T18:05:55.526591",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.439573",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-9-db118bf78c48>:8: DeprecationWarning: Old-style queries are deprecated as of Hatchet 2023.1.0 and will be removed in the             future. Please use new-style queries instead. For QueryMatcher, the equivalent             new-style queries are hatchet.query.Query for base-syntax queries and             hatchet.query.ObjectQuery for the object-dialect.\n",
      "  ht.QueryMatcher()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    __          __       __         __ \n",
      "   / /_  ____ _/ /______/ /_  ___  / /_\n",
      "  / __ \\/ __ `/ __/ ___/ __ \\/ _ \\/ __/\n",
      " / / / / /_/ / /_/ /__/ / / /  __/ /_  \n",
      "/_/ /_/\\__,_/\\__/\\___/_/ /_/\\___/\\__/  v2024.1.0\n",
      "\n",
      "\u001b[38;5;196m0.112\u001b[0m Apps_ENERGY\u001b[0m\n",
      "\u001b[38;5;34m0.047\u001b[0m Basic_DAXPY\u001b[0m\n",
      "\u001b[38;5;22m0.023\u001b[0m Polybench_2MM\u001b[0m\n",
      "\n",
      "\u001b[4mLegend\u001b[0m (Metric: Total time_median Min: 0.02 Max: 0.11)\n",
      "\u001b[38;5;196m█ \u001b[0m0.10 - 0.11\n",
      "\u001b[38;5;208m█ \u001b[0m0.09 - 0.10\n",
      "\u001b[38;5;220m█ \u001b[0m0.07 - 0.09\n",
      "\u001b[38;5;46m█ \u001b[0m0.05 - 0.07\n",
      "\u001b[38;5;34m█ \u001b[0m0.03 - 0.05\n",
      "\u001b[38;5;22m█ \u001b[0m0.02 - 0.03\n",
      "\n",
      "name\u001b[0m User code    \u001b[38;5;160m◀ \u001b[0m Only in left graph    \u001b[38;5;28m▶ \u001b[0m Only in right graph\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.9/site-packages/thicket/stats/median.py:32: FutureWarning: The provided callable <function median at 0xffff4fe9adc0> is currently using DataFrameGroupBy.median. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string \"median\" instead.\n",
      "  df = thicket.dataframe[columns].reset_index().groupby(\"node\").agg(np.median)\n"
     ]
    }
   ],
   "source": [
    "api_entrypoints = [\n",
    "    \"Polybench_2MM\",\n",
    "    \"Basic_DAXPY\",\n",
    "    \"Apps_ENERGY\",\n",
    "]\n",
    "\n",
    "query_ex4 = (\n",
    "    ht.QueryMatcher()\n",
    "    .match(\n",
    "        \".\",\n",
    "        lambda row: row[\"name\"].apply(\n",
    "            lambda x: x in api_entrypoints\n",
    "        ).all()\n",
    "    )\n",
    "    .rel(\"*\")\n",
    ")\n",
    "\n",
    "# applying the fourth query on the lassen thicket\n",
    "th_ex4 = th_lassen.query(query_ex4)\n",
    "tt.median(th_ex4, columns=[\"Total time\"])\n",
    "print(th_ex4.statsframe.tree(\"Total time_median\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "596fd8bd",
   "metadata": {
    "papermill": {
     "duration": 0.002654,
     "end_time": "2024-04-05T18:05:55.532132",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.529478",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Example Query 5: Find All Paths through a Specific Node\n",
    "\n",
    "This example shows how to find all call paths that pass through a specific node. More specifically, the query below finds all paths that pass through a node named \"Stream\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "92f36b94",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-05T18:05:55.537612Z",
     "iopub.status.busy": "2024-04-05T18:05:55.537508Z",
     "iopub.status.idle": "2024-04-05T18:05:55.637099Z",
     "shell.execute_reply": "2024-04-05T18:05:55.636495Z"
    },
    "papermill": {
     "duration": 0.103388,
     "end_time": "2024-04-05T18:05:55.638011",
     "exception": false,
     "start_time": "2024-04-05T18:05:55.534623",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-10-f64ab722a05a>:2: DeprecationWarning: Old-style queries are deprecated as of Hatchet 2023.1.0 and will be removed in the             future. Please use new-style queries instead. For QueryMatcher, the equivalent             new-style queries are hatchet.query.Query for base-syntax queries and             hatchet.query.ObjectQuery for the object-dialect.\n",
      "  ht.QueryMatcher()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    __          __       __         __ \n",
      "   / /_  ____ _/ /______/ /_  ___  / /_\n",
      "  / __ \\/ __ `/ __/ ___/ __ \\/ _ \\/ __/\n",
      " / / / / /_/ / /_/ /__/ / / /  __/ /_  \n",
      "/_/ /_/\\__,_/\\__/\\___/_/ /_/\\___/\\__/  v2024.1.0\n",
      "\n",
      "\u001b[38;5;196m5.074\u001b[0m RAJAPerf\u001b[0m\n",
      "└─ \u001b[38;5;34m0.604\u001b[0m Stream\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.093\u001b[0m Stream_ADD\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.117\u001b[0m Stream_COPY\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.182\u001b[0m Stream_DOT\u001b[0m\n",
      "   ├─ \u001b[38;5;22m0.118\u001b[0m Stream_MUL\u001b[0m\n",
      "   └─ \u001b[38;5;22m0.093\u001b[0m Stream_TRIAD\u001b[0m\n",
      "\n",
      "\u001b[4mLegend\u001b[0m (Metric: Total time_median Min: 0.09 Max: 5.07)\n",
      "\u001b[38;5;196m█ \u001b[0m4.58 - 5.07\n",
      "\u001b[38;5;208m█ \u001b[0m3.58 - 4.58\n",
      "\u001b[38;5;220m█ \u001b[0m2.58 - 3.58\n",
      "\u001b[38;5;46m█ \u001b[0m1.59 - 2.58\n",
      "\u001b[38;5;34m█ \u001b[0m0.59 - 1.59\n",
      "\u001b[38;5;22m█ \u001b[0m0.09 - 0.59\n",
      "\n",
      "name\u001b[0m User code    \u001b[38;5;160m◀ \u001b[0m Only in left graph    \u001b[38;5;28m▶ \u001b[0m Only in right graph\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.9/site-packages/thicket/stats/median.py:32: FutureWarning: The provided callable <function median at 0xffff4fe9adc0> is currently using DataFrameGroupBy.median. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string \"median\" instead.\n",
      "  df = thicket.dataframe[columns].reset_index().groupby(\"node\").agg(np.median)\n"
     ]
    }
   ],
   "source": [
    "query_ex5 = (\n",
    "    ht.QueryMatcher()\n",
    "    .match(\"*\")\n",
    "    .rel(\n",
    "        \".\",\n",
    "        lambda row: row[\"name\"].apply(\n",
    "            lambda x: re.match(\n",
    "                \"Stream\", x\n",
    "            )\n",
    "            is not None\n",
    "        ).all()\n",
    "    )\n",
    "    .rel(\"*\")\n",
    ")\n",
    "\n",
    "# applying the fifth query on the lassen thicket\n",
    "th_ex5 = th_lassen.query(query_ex5)\n",
    "tt.median(th_ex5, columns=[\"Total time\"])\n",
    "print(th_ex5.statsframe.tree(\"Total time_median\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 2.273756,
   "end_time": "2024-04-05T18:05:55.965752",
   "environment_variables": {},
   "exception": null,
   "input_path": "05_thicket_query_language.ipynb",
   "output_path": "05_thicket_query_language.ipynb",
   "parameters": {},
   "start_time": "2024-04-05T18:05:53.691996",
   "version": "2.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}