demos/csv_hierarchy.ipynb

{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# CSV Hierarchies\n",
    "\n",
    "In this short demo notebook, we see how to describe the experiment hierarchy in a csv file. \n",
    "\n",
    "First we define the basics of a simple brightway based calculation, without any hierarchy"
   ],
   "id": "67acc06124cc828f"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-09T10:39:16.085208Z",
     "start_time": "2024-04-09T10:39:14.187877Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import pandas as pd\n",
    "\n",
    "from enbios import Experiment\n",
    "\n",
    "BRIGHTWAY_ECOINVENT_391_PROJECT_NAME = \"ecoinvent_391\"\n",
    "experiment_config = {\n",
    "    \"adapters\": [\n",
    "        {\n",
    "            \"adapter_name\": \"brightway-adapter\",\n",
    "            \"config\": {\"bw_project\": BRIGHTWAY_ECOINVENT_391_PROJECT_NAME},\n",
    "            \"methods\": {\n",
    "                \"GWP1000\": (\n",
    "                    \"ReCiPe 2016 v1.03, midpoint (H)\",\n",
    "                    \"climate change\",\n",
    "                    \"global warming potential (GWP1000)\",\n",
    "                )\n",
    "            },\n",
    "        }\n",
    "    ]\n",
    "}"
   ],
   "id": "initial_id",
   "outputs": [],
   "execution_count": 1
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "There are a couple of test files within enbios. ",
   "id": "dcc55ee023408ae1"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-09T10:39:16.093895Z",
     "start_time": "2024-04-09T10:39:16.086914Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import os\n",
    "from enbios.const import BASE_TEST_DATA_PATH\n",
    "\n",
    "csv_hierarchy_files = BASE_TEST_DATA_PATH / \"hierarchy_csvs/inputs/\"\n",
    "os.listdir(csv_hierarchy_files)\n",
    "# do not try out files ending with \"_x\", these are invalid test files"
   ],
   "id": "5c74fc396c144205",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['hierarchy1.csv',\n",
       " 'hierarchy2.csv',\n",
       " 'hierarchy3.csv',\n",
       " 'hierarchy4.csv',\n",
       " 'hierarchy1_x.csv']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 2
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "Let's look at a first very simple example: `hierarchy1.csv`",
   "id": "39ebf728911598de"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-09T10:39:16.113696Z",
     "start_time": "2024-04-09T10:39:16.094978Z"
    }
   },
   "cell_type": "code",
   "source": "pd.read_csv(csv_hierarchy_files / \"hierarchy1.csv\").fillna(\"\")",
   "id": "84be1da9ff209437",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "  level_0  level_1 module                       config.code\n",
       "0    root             sum                                  \n",
       "1          biomass     bw  4a9e1a0ac89a66b2fe1565cbf2628d9c"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>level_0</th>\n",
       "      <th>level_1</th>\n",
       "      <th>module</th>\n",
       "      <th>config.code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>root</td>\n",
       "      <td></td>\n",
       "      <td>sum</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>biomass</td>\n",
       "      <td>bw</td>\n",
       "      <td>4a9e1a0ac89a66b2fe1565cbf2628d9c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 3
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "The columns which describe the structure of the hierarchy are named __`level_{i}`__. Basically how it internally works, is that it first checks for all columns, that have this structure: `level_<number>` and then sorts them alphabetically. You might want to check the order, if your hierarchy is very deep. Each row describes one node and the contents on one respective column defines the level. \n",
    "\n",
    "The __module__ column defines the adapter or aggregator for that node. Enbios will automatically specify the right option, since it basically just depends on if the node is at the bottom of the hierarchy (structural node - adapter) or not (functional node - aggregator).\n",
    "\n",
    "all other columns should start with `config.` and are values for the configuration. Enbios will consider all those columns as keys for a structured config dictionary, where `.` is used as a separator to create a nested dictionary, in case the config requires that.\n",
    "\n",
    "E.g. \n",
    "```\n",
    "config.loc.0, config.loc.1\n",
    "eu, es\n",
    "```\n",
    "will be turned into:\n",
    "`{'config': {'loc': ['eu', 'es']}}`\n",
    "\n",
    "Let's insert print the hierarchy"
   ],
   "id": "7069cfd595c49ccd"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-09T10:39:18.710897Z",
     "start_time": "2024-04-09T10:39:17.798577Z"
    }
   },
   "cell_type": "code",
   "source": [
    "experiment_config[\"hierarchy\"] = csv_hierarchy_files / \"hierarchy1.csv\"\n",
    "experiment = Experiment(experiment_config)\n",
    "\n",
    "experiment.get_simplified_hierarchy(print_it=True);"
   ],
   "id": "893861ae846bd6bb",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"root\": {\n",
      "    \"biomass\": null\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "execution_count": 4
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "A more complex example:",
   "id": "a9dd4aa481e56f12"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-09T10:39:26.120775Z",
     "start_time": "2024-04-09T10:39:26.099012Z"
    }
   },
   "cell_type": "code",
   "source": "pd.read_csv(csv_hierarchy_files / \"hierarchy4.csv\").fillna(\"\")",
   "id": "df89e275806b94f8",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "  level_0        level_1   level_2 module                       config.code\n",
       "0    root                             sum                                  \n",
       "1           intermediate              sum                                  \n",
       "2                          biomass     bw  4a9e1a0ac89a66b2fe1565cbf2628d9c\n",
       "3                         biomass2     bw  4a9e1a0ac89a66b2fe1565cbf2628d9c\n",
       "4          intermediate2              sum                                  \n",
       "5                         biomass3     bw  4a9e1a0ac89a66b2fe1565cbf2628d9c\n",
       "6                         biomass4     bw  4a9e1a0ac89a66b2fe1565cbf2628d9c"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>level_0</th>\n",
       "      <th>level_1</th>\n",
       "      <th>level_2</th>\n",
       "      <th>module</th>\n",
       "      <th>config.code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>root</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>sum</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>intermediate</td>\n",
       "      <td></td>\n",
       "      <td>sum</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>biomass</td>\n",
       "      <td>bw</td>\n",
       "      <td>4a9e1a0ac89a66b2fe1565cbf2628d9c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>biomass2</td>\n",
       "      <td>bw</td>\n",
       "      <td>4a9e1a0ac89a66b2fe1565cbf2628d9c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>intermediate2</td>\n",
       "      <td></td>\n",
       "      <td>sum</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>biomass3</td>\n",
       "      <td>bw</td>\n",
       "      <td>4a9e1a0ac89a66b2fe1565cbf2628d9c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>biomass4</td>\n",
       "      <td>bw</td>\n",
       "      <td>4a9e1a0ac89a66b2fe1565cbf2628d9c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 6
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-09T10:39:28.135794Z",
     "start_time": "2024-04-09T10:39:27.255181Z"
    }
   },
   "cell_type": "code",
   "source": [
    "experiment_config[\"hierarchy\"] = csv_hierarchy_files / \"hierarchy4.csv\"\n",
    "experiment = Experiment(experiment_config)\n",
    "\n",
    "experiment.get_simplified_hierarchy(print_it=True);"
   ],
   "id": "2f7b6ad96169daff",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"root\": {\n",
      "    \"intermediate\": {\n",
      "      \"biomass\": null,\n",
      "      \"biomass2\": null\n",
      "    },\n",
      "    \"intermediate2\": {\n",
      "      \"biomass3\": null,\n",
      "      \"biomass4\": null\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "execution_count": 7
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "If it is crucial to use different level column names you can call the function `csv2hierarchy` directly and passing one of the two arguments:\n",
    "`level_cols` a list of strings, which include the columns which specify the names of nodes or `levels_regex` (which is the default with the regular expression : `^level_\\d+$`. The result is a dictionary which can be inserted into the experiment config as the hierarchy."
   ],
   "id": "c1269da612ffd6b1"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-09T10:39:30.242002Z",
     "start_time": "2024-04-09T10:39:30.226947Z"
    }
   },
   "cell_type": "code",
   "source": [
    "from enbios.base.tree_operations import csv2hierarchy\n",
    "\n",
    "csv2hierarchy(csv_hierarchy_files / \"hierarchy4.csv\")"
   ],
   "id": "9c6ed27120694972",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'root',\n",
       " 'children': [{'name': 'intermediate',\n",
       "   'children': [{'name': 'biomass',\n",
       "     'config': {'code': '4a9e1a0ac89a66b2fe1565cbf2628d9c'},\n",
       "     'adapter': 'bw'},\n",
       "    {'name': 'biomass2',\n",
       "     'config': {'code': '4a9e1a0ac89a66b2fe1565cbf2628d9c'},\n",
       "     'adapter': 'bw'}],\n",
       "   'aggregator': 'sum'},\n",
       "  {'name': 'intermediate2',\n",
       "   'children': [{'name': 'biomass3',\n",
       "     'config': {'code': '4a9e1a0ac89a66b2fe1565cbf2628d9c'},\n",
       "     'adapter': 'bw'},\n",
       "    {'name': 'biomass4',\n",
       "     'config': {'code': '4a9e1a0ac89a66b2fe1565cbf2628d9c'},\n",
       "     'adapter': 'bw'}],\n",
       "   'aggregator': 'sum'}],\n",
       " 'aggregator': 'sum'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 8
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": "",
   "id": "215a631e4dcbd37c"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}