doc/source/notebooks/load_save_export.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading, saving and exporting data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pymrio includes several functions for data reading and storing. This section presents the methods to use for saving and loading data already in a pymrio compatible format. For parsing raw MRIO data see the different tutorials for [working with available MRIO databases](../handling.rst)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we use the included small test MRIO system to highlight the different function. The same functions are available for any MRIO loaded into pymrio. Expect, however, significantly decreased performance due to the size of real MRIO system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymrio\n",
    "import os\n",
    "\n",
    "io = pymrio.load_test().calc_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic save and read"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save the full system, use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pymrio.core.mriosystem.IOSystem at 0x7fe06ba0e880>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "save_folder_full = \"/tmp/testmrio/full\"\n",
    "io.save_all(path=save_folder_full)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read again from that folder do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "io_read = pymrio.load_all(path=save_folder_full)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fileio activities are stored in the included meta data history field:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Description: test mrio for pymrio\n",
       "MRIO Name: testmrio\n",
       "System: pxp\n",
       "Version: v1\n",
       "File: /tmp/testmrio/full/metadata.json\n",
       "History:\n",
       "20210224 11:32:06 - FILEIO -  Added satellite account from /tmp/testmrio/full/factor_inputs\n",
       "20210224 11:32:06 - FILEIO -  Added satellite account from /tmp/testmrio/full/emissions\n",
       "20210224 11:32:06 - FILEIO -  Loaded IO system from /tmp/testmrio/full\n",
       "20210224 11:32:06 - FILEIO -  Saved testmrio to /tmp/testmrio/full\n",
       "20210224 11:32:06 - MODIFICATION -  Calculating accounts for extension emissions\n",
       "20210224 11:32:06 - MODIFICATION -  Calculating accounts for extension factor_inputs\n",
       "20210224 11:32:06 - MODIFICATION -  Leontief matrix L calculated\n",
       "20210224 11:32:06 - MODIFICATION -  Coefficient matrix A calculated\n",
       "20210224 11:32:06 - MODIFICATION -  Industry output x calculated\n",
       "20210224 11:32:06 - FILEIO -  Load test_mrio from /home/konstans/proj/pymrio/pymrio/mrio_models/test_mrio\n",
       " ... (more lines in history)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "io_read.meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storage format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Internally, pymrio stores data in csv format, with the 'economic core' data in the root and each satellite account in a subfolder. Metadata as file as a file describing the data format ('file_parameters.json') are included in each folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['emissions',\n",
       " 'factor_inputs',\n",
       " 'metadata.json',\n",
       " 'file_parameters.json',\n",
       " 'population.txt',\n",
       " 'unit.txt',\n",
       " 'L.txt',\n",
       " 'A.txt',\n",
       " 'x.txt',\n",
       " 'Y.txt',\n",
       " 'Z.txt']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "os.listdir(save_folder_full)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file format for storing the MRIO data can be switched to a binary pickle format with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['emissions',\n",
       " 'factor_inputs',\n",
       " 'metadata.json',\n",
       " 'file_parameters.json',\n",
       " 'population.pkl',\n",
       " 'unit.pkl',\n",
       " 'L.pkl',\n",
       " 'A.pkl',\n",
       " 'x.pkl',\n",
       " 'Y.pkl',\n",
       " 'Z.pkl']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "save_folder_bin = \"/tmp/testmrio/binary\"\n",
    "io.save_all(path=save_folder_bin, table_format=\"pkl\")\n",
    "os.listdir(save_folder_bin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be used to reduce the storage space required on the disk for large MRIO databases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Archiving MRIOs databases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To archive a MRIO system after saving use pymrio.archive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrio_arc = \"/tmp/testmrio/archive.zip\"\n",
    "\n",
    "# Remove a potentially existing archive from before\n",
    "try:\n",
    "    os.remove(mrio_arc)\n",
    "except FileNotFoundError:\n",
    "    pass\n",
    "\n",
    "pymrio.archive(source=save_folder_full, archive=mrio_arc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data can be read directly from such an archive by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "tt = pymrio.load_all(mrio_arc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently data can not be saved directly into a zip archive.\n",
    "It is, however, possible to remove the source files after archiving:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directories before archiving: ['tmp', 'emission_footprints.xlsx', 'emissions', 'binary', 'full']\n",
      "Directories after archiving: ['archive.zip', 'emission_footprints.xlsx', 'emissions', 'binary', 'full']\n"
     ]
    }
   ],
   "source": [
    "tmp_save = \"/tmp/testmrio/tmp\"\n",
    "\n",
    "# Remove a potentially existing archive from before\n",
    "try:\n",
    "    os.remove(mrio_arc)\n",
    "except FileNotFoundError:\n",
    "    pass\n",
    "\n",
    "io.save_all(tmp_save)\n",
    "\n",
    "print(\"Directories before archiving: {}\".format(os.listdir(\"/tmp/testmrio\")))\n",
    "pymrio.archive(source=tmp_save, archive=mrio_arc, remove_source=True)\n",
    "print(\"Directories after archiving: {}\".format(os.listdir(\"/tmp/testmrio\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Several MRIO databases can be stored in the same archive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove a potentially existing archive from before\n",
    "try:\n",
    "    os.remove(mrio_arc)\n",
    "except FileNotFoundError:\n",
    "    pass\n",
    "\n",
    "tmp_save = \"/tmp/testmrio/tmp\"\n",
    "\n",
    "io.save_all(tmp_save)\n",
    "pymrio.archive(\n",
    "    source=tmp_save, archive=mrio_arc, path_in_arc=\"version1/\", remove_source=True\n",
    ")\n",
    "io2 = io.copy()\n",
    "del io2.emissions\n",
    "io2.save_all(tmp_save)\n",
    "pymrio.archive(\n",
    "    source=tmp_save, archive=mrio_arc, path_in_arc=\"version2/\", remove_source=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When loading from an archive which includes multiple MRIO databases, specify\n",
    "one with the parameter 'path_in_arc':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extensions of the loaded io1 ['emissions', 'factor_inputs'] and of io2: ['factor_inputs']\n"
     ]
    }
   ],
   "source": [
    "io1_load = pymrio.load_all(mrio_arc, path_in_arc=\"version1/\")\n",
    "io2_load = pymrio.load_all(mrio_arc, path_in_arc=\"version2/\")\n",
    "\n",
    "print(\n",
    "    \"Extensions of the loaded io1 {ver1} and of io2: {ver2}\".format(\n",
    "        ver1=sorted(io1_load.get_extensions()), ver2=sorted(io2_load.get_extensions())\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pymrio.load function can be used directly to only a specific satellite account \n",
    "of a MRIO database from a zip archive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extension Emissions with parameters: name, F, F_Y, S, S_Y, M, D_cba, D_pba, D_imp, D_exp, unit, D_cba_reg, D_pba_reg, D_imp_reg, D_exp_reg, D_cba_cap, D_pba_cap, D_imp_cap, D_exp_cap\n"
     ]
    }
   ],
   "source": [
    "emissions = pymrio.load(mrio_arc, path_in_arc=\"version1/emissions\")\n",
    "print(emissions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The archive function is a wrapper around python.zipfile module.\n",
    "There are, however, some differences to the defaults choosen in the original:\n",
    "\n",
    "-  In contrast to [zipfile.write](https://docs.python.org/3/library/zipfile.html), \n",
    "   pymrio.archive raises an\n",
    "   error if the data (path + filename) are identical in the zip archive.\n",
    "   Background: the zip standard allows that files with the same name and path\n",
    "   are stored side by side in a zip file. This becomes an issue when unpacking\n",
    "   this files as they overwrite each other upon extraction.\n",
    "   \n",
    "-  The standard for the parameter 'compression' is set to ZIP_DEFLATED \n",
    "   This is different from the zipfile default (ZIP_STORED) which would\n",
    "   not give any compression. \n",
    "   See the [zipfile docs](https://docs.python.org/3/library/zipfile.html#zipfile-objects) \n",
    "   for further information. \n",
    "   Depending on the value given for the parameter 'compression' \n",
    "   additional modules might be necessary (e.g. zlib for ZIP_DEFLATED). \n",
    "   Futher information on this can also be found in the zipfile python docs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing or exporting a specific table or extension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each extension of the MRIO system can be stored separetly with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_folder_em = \"/tmp/testmrio/emissions\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pymrio.core.mriosystem.Extension at 0x7fe06ba19fa0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "io.emissions.save(path=save_folder_em)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can then be loaded again as separate satellite account:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "emissions = pymrio.load(save_folder_em)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pymrio.core.mriosystem.Extension at 0x7fe0698db130>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emissions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>region</th>\n",
       "      <th colspan=\"8\" halign=\"left\">reg1</th>\n",
       "      <th colspan=\"2\" halign=\"left\">reg2</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"2\" halign=\"left\">reg5</th>\n",
       "      <th colspan=\"8\" halign=\"left\">reg6</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>sector</th>\n",
       "      <th>food</th>\n",
       "      <th>mining</th>\n",
       "      <th>manufactoring</th>\n",
       "      <th>electricity</th>\n",
       "      <th>construction</th>\n",
       "      <th>trade</th>\n",
       "      <th>transport</th>\n",
       "      <th>other</th>\n",
       "      <th>food</th>\n",
       "      <th>mining</th>\n",
       "      <th>...</th>\n",
       "      <th>transport</th>\n",
       "      <th>other</th>\n",
       "      <th>food</th>\n",
       "      <th>mining</th>\n",
       "      <th>manufactoring</th>\n",
       "      <th>electricity</th>\n",
       "      <th>construction</th>\n",
       "      <th>trade</th>\n",
       "      <th>transport</th>\n",
       "      <th>other</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>stressor</th>\n",
       "      <th>compartment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>emission_type1</th>\n",
       "      <th>air</th>\n",
       "      <td>2.056183e+06</td>\n",
       "      <td>179423.535893</td>\n",
       "      <td>9.749300e+07</td>\n",
       "      <td>1.188759e+07</td>\n",
       "      <td>3.342906e+06</td>\n",
       "      <td>3.885884e+06</td>\n",
       "      <td>1.075027e+07</td>\n",
       "      <td>1.582152e+07</td>\n",
       "      <td>1.793338e+06</td>\n",
       "      <td>19145.604911</td>\n",
       "      <td>...</td>\n",
       "      <td>4.209505e+07</td>\n",
       "      <td>1.138661e+07</td>\n",
       "      <td>1.517235e+07</td>\n",
       "      <td>1.345318e+06</td>\n",
       "      <td>7.145075e+07</td>\n",
       "      <td>3.683167e+07</td>\n",
       "      <td>1.836696e+06</td>\n",
       "      <td>4.241568e+07</td>\n",
       "      <td>4.805409e+07</td>\n",
       "      <td>3.602298e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>emission_type2</th>\n",
       "      <th>water</th>\n",
       "      <td>2.423103e+05</td>\n",
       "      <td>25278.192086</td>\n",
       "      <td>1.671240e+07</td>\n",
       "      <td>1.371303e+05</td>\n",
       "      <td>3.468292e+05</td>\n",
       "      <td>7.766205e+05</td>\n",
       "      <td>4.999628e+05</td>\n",
       "      <td>8.480505e+06</td>\n",
       "      <td>2.136528e+05</td>\n",
       "      <td>3733.601474</td>\n",
       "      <td>...</td>\n",
       "      <td>4.243738e+06</td>\n",
       "      <td>7.307208e+06</td>\n",
       "      <td>4.420574e+06</td>\n",
       "      <td>5.372216e+05</td>\n",
       "      <td>1.068144e+07</td>\n",
       "      <td>5.728136e+05</td>\n",
       "      <td>9.069515e+05</td>\n",
       "      <td>5.449044e+07</td>\n",
       "      <td>8.836484e+06</td>\n",
       "      <td>4.634899e+07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 48 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "region                              reg1                               \\\n",
       "sector                              food         mining manufactoring   \n",
       "stressor       compartment                                              \n",
       "emission_type1 air          2.056183e+06  179423.535893  9.749300e+07   \n",
       "emission_type2 water        2.423103e+05   25278.192086  1.671240e+07   \n",
       "\n",
       "region                                                                \\\n",
       "sector                       electricity  construction         trade   \n",
       "stressor       compartment                                             \n",
       "emission_type1 air          1.188759e+07  3.342906e+06  3.885884e+06   \n",
       "emission_type2 water        1.371303e+05  3.468292e+05  7.766205e+05   \n",
       "\n",
       "region                                                          reg2  \\\n",
       "sector                         transport         other          food   \n",
       "stressor       compartment                                             \n",
       "emission_type1 air          1.075027e+07  1.582152e+07  1.793338e+06   \n",
       "emission_type2 water        4.999628e+05  8.480505e+06  2.136528e+05   \n",
       "\n",
       "region                                    ...          reg5                \\\n",
       "sector                            mining  ...     transport         other   \n",
       "stressor       compartment                ...                               \n",
       "emission_type1 air          19145.604911  ...  4.209505e+07  1.138661e+07   \n",
       "emission_type2 water         3733.601474  ...  4.243738e+06  7.307208e+06   \n",
       "\n",
       "region                              reg6                              \\\n",
       "sector                              food        mining manufactoring   \n",
       "stressor       compartment                                             \n",
       "emission_type1 air          1.517235e+07  1.345318e+06  7.145075e+07   \n",
       "emission_type2 water        4.420574e+06  5.372216e+05  1.068144e+07   \n",
       "\n",
       "region                                                                \\\n",
       "sector                       electricity  construction         trade   \n",
       "stressor       compartment                                             \n",
       "emission_type1 air          3.683167e+07  1.836696e+06  4.241568e+07   \n",
       "emission_type2 water        5.728136e+05  9.069515e+05  5.449044e+07   \n",
       "\n",
       "region                                                  \n",
       "sector                         transport         other  \n",
       "stressor       compartment                              \n",
       "emission_type1 air          4.805409e+07  3.602298e+07  \n",
       "emission_type2 water        8.836484e+06  4.634899e+07  \n",
       "\n",
       "[2 rows x 48 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emissions.D_cba"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As all data in pymrio is stored as [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html), the full pandas stack for exporting tables is available. For example, to export a table as excel sheet use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "io.emissions.D_cba.to_excel(\"/tmp/testmrio/emission_footprints.xlsx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For further information see the pandas [documentation on import/export](https://pandas.pydata.org/pandas-docs/stable/io.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}