tutorial/westeros/westeros_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Westeros Tutorial Part 3 - Reporting\n",
    "\n",
    "‘Reporting’ is the term used in the MESSAGEix ecosystem to refer to any calculations performed _after_ the MESSAGE mathematical optimization problem has been solved.\n",
    "\n",
    "This tutorial introduces the reporting features provided by the ``ixmp`` and ``message_ix`` packages.\n",
    "_It was developed by Paul Natsuo Kishimoto ([@khaeru](https://github.com/khaeru)) for a MESSAGE training workshop held at IIASA in October 2019._\n",
    "<!-- Add a line here if you revise the tutorial! -->\n",
    "\n",
    "**Pre-requisites**\n",
    "- You have the *MESSAGEix* framework installed and working\n",
    "- Complete tutorial Part 1 (``westeros_baseline.ipynb``)\n",
    "  - Understand the following MESSAGEix terms: ‘variable’, ‘parameter’\n",
    "- Open the [‘Reporting’ page in the MESSAGEix documentation](https://message.iiasa.ac.at/en/stable/reporting.html); bookmark it or keep it open in a tab.\n",
    "  Some text in this tutorial is drawn from that page, and it provides a concise reference for concepts explained below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "### What does ‘reporting’ include?\n",
    "\n",
    "Individual modelers often make different distinctions between—on one hand—the internals of an optimization model and—on the other—reporting, ‘post-processing’, ‘analysis’, and other tasks.\n",
    "Doing valid research using models like MESSAGE requires that we understand these differences clearly, as well as how we choose to communicate them.\n",
    "\n",
    "For example, we might say: “The MESSAGE model shows that total secondary energy (electricity) output in Westeros in the year 720 is 9 GWa.”\n",
    "\n",
    "But, if we are using the model from ``westeros_baseline.ipynb``:\n",
    "1. The raw data from the ``Scenario``, after ``.solve()`` has been called, **only** tells us the ``ACT`` variable has certain values.\n",
    "2. To get the 9 GWa figure, we must:\n",
    "   1. Compute the product of activity (``ACT``, which is dimensionless) and output efficiency (``output`` in GWa/year), then\n",
    "   2. Sum across the ``technology`` dimension, and finally\n",
    "   3. Select the single value for the ``year`` 720.\n",
    "   \n",
    "In this example, steps A, B, and C under #2 are part of ‘reporting’.\n",
    "Even a intuitive concept like “total secondary energy” is not a *direct* output of the model, but must be reported.\n",
    "\n",
    "Next, we may want to create a plot of electricity output by year.\n",
    "Some modelers consider this part of ‘reporting’; for others, ‘reporting’ is complete when the values needed for the plot are written to a file, which they can then use with their favourite plotting tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reporting features in MESSAGEix\n",
    "\n",
    "The reporting features in ``ixmp`` and ``message_ix`` are developed to support the complicated reporting and multiple workflows required by the IIASA Energy program for research projects involving large, detailed models such as the [MESSAGE-GLOBIOM global model](https://message.iiasa.ac.at/global/).\n",
    "While powerful enough for this purpose, they are also intended to be user-friendly, flexible, and customizable.\n",
    "\n",
    "The core class used for reporting is ``message_ix.Reporter``, inherits from ``ixmp.Reporter``.\n",
    "A reporting workflow has two steps:\n",
    "\n",
    "1. Describe all computations the Reporter may handle, using ``Reporter.add()`` and other helper methods.\n",
    "2. Triggering the computation of one or more quantities or reports, using ``Reporter.get()``.\n",
    "\n",
    "This two-step process allows the Reporter to delivery good performance, by excluding unneeded computations and avoiding recomputation of intermediate results that are used in multiple places."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concepts: The Graph\n",
    "\n",
    "``ixmp.Reporter`` is built around a **graph** of *nodes* and *edges*; specifically, a *directed, acyclic graph*.\n",
    "This means:\n",
    "- Every edge has a direction; *from* one node *to* another.\n",
    "- There are no recursive loops in the graph; i.e. no node is its own ancestor.\n",
    "\n",
    "In the reporting graph, every **node** represents a numerical *calculation*—or, more generally, a *computation* (including other actions like manipulating data formats).\n",
    "The node is labeled with the name of the quantity it produces, which is called a **key**.\n",
    "\n",
    "A node's computation may depend on certain inputs.\n",
    "These are represented by the **edges** of the graph.\n",
    "\n",
    "For example, the following equation:\n",
    "\n",
    "> $C = A + B$\n",
    "\n",
    "…is represented by:\n",
    "- A node named 'A' that provides the raw value of A.\n",
    "- A node named 'B' that provides the raw value of B.\n",
    "- A node named 'C' that computes a sum of its inputs.\n",
    "- An edge from 'A' to 'C'.\n",
    "- An edge from 'B' to 'C'.\n",
    "\n",
    "We use the Reporter to describe this equation (step 1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from message_ix.reporting import Reporter\n",
    "\n",
    "# Create a new Reporter object\n",
    "rep = Reporter()\n",
    "\n",
    "# Add two nodes\n",
    "# These have no outputs; they only return a literal value.\n",
    "rep.add('A', 1)\n",
    "rep.add('B', 2)\n",
    "\n",
    "# Add one node and two edges\n",
    "rep.add('C', (lambda *inputs: sum(inputs), 'A', 'B'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a detailed explanation of what we just did:\n",
    "- We use the ``add()`` method of a Reporter object to build the graph.\n",
    "(Remember: you can type ``Reporter.add?`` or ``rep.add?`` in a new cell to use Jupyter's help features; or look at the documentation page linked above.)\n",
    "- The first argument to ``add()`` is the key of the node.\n",
    "- The second argument describes the computation to be performed:\n",
    "  - For nodes ‘A’ and ‘B’, these are simply the raw or literal value to be produced by the node.\n",
    "  - For node ‘C’, it is a Python tuple with 3 items: ``(lambda *inputs: sum(inputs), 'A', 'B')``.\n",
    "  \n",
    "    Let's break that down further:\n",
    "    - The first item, ``lambda *inputs: sum(inputs)``, is an anonymous function ([read more](https://doc.python.org/3/tutorial/controlflow.html#lambda-expressions)) that computes the sum of its inputs.\n",
    "    - All the remaining items, ``'A', B'``, are keys for other nodes in the graph.\n",
    "\n",
    "Next, let's trigger the calculation of ‘C’ (step 1), which gives the expected value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rep.get('C')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use ``Reporter.describe()`` to see steps used in this calculation.\n",
    "The graph is printed out as a hierarchical list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(rep.describe('C'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This description shows how the Reporter traverses the graph in order to calculate the quantity we asked for:\n",
    "\n",
    "1. The desired value is from node ‘C’, which computes a function of some arguments.\n",
    "2. The first argument is ‘A’.\n",
    "3. ‘A’ is the name of another node.\n",
    "4. Node ‘A’ gives a literal value `int(1)`, which is stored.\n",
    "5. The Reporter returns to ‘C’ and moves on to the next argument, ‘B’.\n",
    "6. Steps 3 and 4 are repeated for ‘B’, giving `int(2)`.\n",
    "7. All of the arguments to ‘C’ have been processed.\n",
    "8. The computation function for ‘C’ is called.\n",
    "\n",
    "   As arguments, instead of the (string) keys 'A' and 'B', this function receives the computed `int` values from steps 4 and 6 respectively.\n",
    "9. The result is returned.\n",
    "\n",
    "In this example, 'A' and 'B' are, at most, 1 step away from the node we requested, and are each used once.\n",
    "\n",
    "In more realistic examples, the graph can have:\n",
    "- Long chains of calculations, each depending on the output of its ancestors, and/or\n",
    "- Multiple connection, so that 'A' is used by more than one child calculations.\n",
    "\n",
    "However, the Reporter still follows the same procedure to traverse the graph and calculate the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Store *rep* for the solutions at the bottom of the notebook\n",
    "rep1 = rep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "Add a node ‘X’ to the graph that returns the literal value 42."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After adding ‘X’, what do you think will be the result when you run the following cell?\n",
    "Why?\n",
    "\n",
    "Write down your answer before trying the code.\n",
    "\n",
    "(Answers and code blocks that solve all exercises are listed at the bottom of the tutorial—don't peek!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(rep.describe('C'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "Extend the `Reporter` to describe the following equation:\n",
    "\n",
    "> $E = A + D * \\frac{A}{A + B} = A + D * \\frac{A}{C}; D = 12$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some helper functions you can use\n",
    "def sum_calc(*inputs):\n",
    "    return sum(inputs)\n",
    "\n",
    "def product(a, b):\n",
    "    return a * b\n",
    "\n",
    "def ratio(a, b):\n",
    "    return a / b\n",
    "\n",
    "# Replace 'C' with a reference to sum_calc (instead of an anonymous function)\n",
    "rep.add('C', (sum_calc, 'A', 'B'))\n",
    "\n",
    "# Add your code here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concepts: Quantities, Keys, and data formats\n",
    "\n",
    "In the last section, $A$, $B$, and so on were *scalar* variables with a single value.\n",
    "In energy systems modeling, including with MESSAGE*ix*, we usually deal with scientific **quantities** that are sparse, multi-dimensional arrays with associated units.\n",
    "\n",
    "That is, they have:\n",
    "- 1 or more **dimensions**, with *labels* along those dimensions (e.g. specific years; the names of specific technologies);\n",
    "- *sparse* coverage or “missingness,” i.e. there is not necessarily a value for each combination of labels; and\n",
    "- associated *units*.\n",
    "\n",
    "Mathematically, we can say the following:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "A^{ij} & = \\left[a_{i,j} \\right] \\\\\n",
    "i & \\in I \\\\\n",
    "j & \\in J \\\\\n",
    "a_{i,j} & \\in \\left\\{ \\mathbb{R}, \\text{NaN} \\right\\} \\\\\n",
    "a_{i,j} & [=]\\, \\text{units of X}\n",
    "\\end{align}\n",
    "$$\n",
    "…where ‘NaN’ means “not a number,” i.e. a missing value.\n",
    "\n",
    "(Note: The ``westeros_baseline.ipynb`` distinguished fixed *parameters*, also called “input data”, from the decision *variables* that the optimization algorithm changes to find the model solution.\n",
    "When reporting, the distinction is not important; so we refer to everything as a **quantity**.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dimensionality of quantities\n",
    "\n",
    "Some of the quantities in the MESSAGE mathematical formulation have many dimensions.\n",
    "For some calculations, we may not care about some of these dimensions.\n",
    "\n",
    "For instance, $\\text{output}$ has ten dimensions: $\\text{output}^{n^Lty^Vymnclh^Ah}$.\n",
    "But we may be interested in the total output in a given year ($y$), but not concerned about different vintages of a technology ($y^V$).\n",
    "In this case, we don't really want the 10-dimensional quantity—but its **partial sum** over all values of $y^V$.\n",
    "\n",
    "**Notation.**\n",
    "Consider a quantity with three dimensions, $A^{ijk}$, and another with two, $B^{kl}$, and a scalar $C$.\n",
    "We define partial sums over every possible combination of dimensions:\n",
    "\n",
    "$$\n",
    "\\begin{array}\n",
    "AA^{ij} = \\left[ a_{i,j} \\right],\n",
    "  & a_{i,j} = \\sum_{k}{a_{i,j,k}} \\ \\forall \\ i, j\n",
    "  & \\text{similarly } A^{ik}, A^{jk} \\\\\n",
    "A^{i} = \\left[ a_i \\right],\n",
    "  & a_i = \\sum_j\\sum_{k}{a_{i,j,k}} \\ \\forall\\  i\n",
    "  & \\text{similarly } A^j, A^k \\\\\n",
    "& \\qquad A = \\sum_i\\sum_j\\sum_k{a_{i,j,k}}\n",
    "  & \\text{(a scalar)}\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "Note that $A$ and $B$ share one dimension, $j$, but the other dimensions are distinct.\n",
    "We specify that simple arithmetic operations result in a quantity whose dimensions are the union of the dimensions of the operands. In other words:\n",
    "\n",
    "$$\n",
    "\\begin{array}\n",
    "CC + A^{i} = X^{i} = \\left[ x_{i} \\right],\n",
    "  & x_{i} = C + a_{i} \\ \\forall \\ i \\\\\n",
    "A^{jk} \\times B^{kl} = Y^{jkl} = \\left[ y_{j,k,l} \\right],\n",
    "  & y_{j,k,l} = a_{j,k} + b_{k,l} \\ \\forall \\ j, k, l \\\\\n",
    "A^{j} - B^{j} = Z^{j} = \\left[ z_{j} \\right],\n",
    "  & z_{j} = a_{j} + b_{j} \\ \\forall \\ j \\\\\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "As a result of this rule:\n",
    "- The difference $Z^j$ has the same dimensionality as *both* of its operands.\n",
    "- The sum $X^i$ has the same dimensionality as *one* of its operands.\n",
    "- The product $Y^{jkl}$ has a different dimensonality from each of its operands.\n",
    "\n",
    "These operations are called **broadcasting** and **alignment**: The scalar value $C$ is *broadcast* across all values the dimension $j$ that it lacks in order to calculate $x_i$.\n",
    "$A^{jk}$ and $B^{kl}$ are *aligned* on matching values of $j$, but *broadcast* over dimensions $l$ and $j$, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keys\n",
    "\n",
    "In the last code example, `'C'` is the node label or key that we use to refer to the output of a certain calculation—even before it has been computed.\n",
    "Likewise, the Python string `'A'` is a key.\n",
    "When computed, node ‘A’ returns a Python `int(1)`—an object representing its actual *value*.\n",
    "\n",
    "In step 1 of the reporting workflow, computations are described using *only* keys.\n",
    "No *values* are created until step 2—and *only* the values needed to provide the result of `get()`.\n",
    "\n",
    "For multi-dimensional calculations, we need keys that distinguish the partial sum of $A^{ijk}$ used in the calculation of $X^i$ from the different partial sum used in the calculation of $Y^{jkl}$.\n",
    "\n",
    "For this purpose, message_ix (or ixmp) provides the `Key` class.\n",
    "\n",
    "A Key has a name, zero or more dimensions, and an optional tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from message_ix.reporting import Key\n",
    "# from ixmp.reporting import Key  # Same class\n",
    "\n",
    "# Quantity named 'a' at two different dimensionalities\n",
    "k1 = Key('a', ['i'])\n",
    "k2 = Key('a', ['j', 'k'])\n",
    "\n",
    "(k1, k2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantity values\n",
    "\n",
    "To represent the **values** of quantities from a model or produced by reporting calculations, ixmp and message_ix use the `Quantity` class.\n",
    "Quantity is derived from [`xarray.DataArray`](http://xarray.pydata.org/en/stable/data-structures.html#dataarray)—a labeled, multi-dimensional array, with attributes.\n",
    "\n",
    "The combination of Key and Quantity lets the Reporter (and you!) handle multi-dimensional data, while automatically handling alignment and broadcasting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated reporting\n",
    "\n",
    "A `message_ix.Reporter` for a specific `Scenario` is created using the `.from_scenario()` method.\n",
    "This method automatically adds many nodes to the graph based on (a) the contents of the Scenario and (b) the known mathematical formulation of MESSAGE.\n",
    "\n",
    "### Demonstration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ixmp import Platform\n",
    "from ixmp.reporting import configure\n",
    "from message_ix.testing import make_westeros\n",
    "\n",
    "scen = make_westeros(Platform(), emissions=True, solve=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a reporter from the existing Scenario\n",
    "rep = Reporter.from_scenario(scen)\n",
    "\n",
    "# Reporter uses the Python pint to handle units. '-', used in the Westeros\n",
    "# tutorial, is not a defined SI unit. We tell the Reporter to replace it with\n",
    "# '' (unitless) everywhere it appears.\n",
    "configure(units={'replace': {'-': ''}})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is in `rep`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "name": "len-rep-graph"
    }
   },
   "outputs": [],
   "source": [
    "len(rep.graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost 8000 nodes!\n",
    "\n",
    "Remember: `rep` simply *describes* these operations; none of them is executed until or unless you `get()` them.\n",
    "\n",
    "Let's look at some of the automatically populated content of the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return the full-dimensionality Key for the MESSAGE parameter 'output'\n",
    "output = rep.full_key('output')\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return the full-dimensionality Key for the MESSAGE variable 'ACT'\n",
    "ACT = rep.full_key('ACT')\n",
    "ACT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What would happen if we were to `get()` this key?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(rep.describe(ACT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see:\n",
    "- The Reporter will call a function named `data_for_quantity()`.\n",
    "\n",
    "  This (and all built-in computations) are [described in the MESSAGEix documentation](https://message.iiasa.ac.at/en/stable/reporting.html#ixmp.reporting.utils.data_for_quantity).\n",
    "- The function gets some direct arguments: `'var', 'ACT', 'lvl'`.\n",
    "\n",
    "  From the documentation, we can see this indicates the level (rather than marginal) of an ixmp `'var'`iable (rather than parameter) named `'ACT'`.\n",
    "- The next argument is ‘scenario’, another node in the graph.\n",
    "- This node returns the same Scenario object we passed to `Reporter.from_scenario()`.\n",
    "\n",
    "In short, if we run this cell, the Reporter will extract a 6-dimensional quantity from the Scenario object and return it.\n",
    "The other 7970+ nodes will not be computed.\n",
    "\n",
    "Let's try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rep.get(ACT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More automated contents\n",
    "\n",
    "As mentioned, because `Reporter.from_scenario()` knows that `scen` follows the MESSAGE mathematical formulation, it can automatically populate the graph with useful derived quantities.\n",
    "\n",
    "For example: the $\\text{ACT}$ for various technologies $t$ is a dimensionless quantity.\n",
    "The specific commodities produced by $t$, with units, are given by the product of $\\text{ACT}$ and $\\text{output}$.\n",
    "This product is given the name ‘out’ (the documentation contains [the names for all automatic quantities](https://message.iiasa.ac.at/en/latest/reporting.html#message_ix.reporting.Reporter.from_scenario)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out = rep.full_key('out')\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show what would be done\n",
    "print(rep.describe(out))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rep.get(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the mode ($m$), time ($h$) and time_dest ($h^D$) dimensions don't contain useful information.\n",
    "We also have a single-region model, so we don't need node_loc ($n^L$) or node_dest ($n^L$) either.\n",
    "We can instead ask for a partial sum.\n",
    "\n",
    "`Key.drop()` lets us derive its key from the one we already have.\n",
    "This doesn't perform any calculation; simply returns a new Key with fewer dimensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out2 = out.drop('h', 'hd', 'm', 'nd', 'nl')\n",
    "out2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This partial sum is already described in the Reporter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(rep.describe(out2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rep.get(out2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File output\n",
    "\n",
    "As noted above, the labeled, multi-dimensional Quantity is used so that values passing between reporting calculations are in a consistent, easy-to-manipulate format.\n",
    "\n",
    "For research purposes, we often want to transform data into other, particular formats or write it to file, in order to feed it into other tools such as existing analysis or plotting codes; both our own, and collaborators'. Reporter provides multiple ways to do this.\n",
    "\n",
    "For instance, we can `get()` a Quantity and write it directly to a file in a single step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rep.write(out2, 'output.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Try using an .xlsx file name in the above.\n",
    "\n",
    "We can also define a conversion to a different data format.\n",
    "This is described in the next section.\n",
    "\n",
    "## Describing additional computations\n",
    "\n",
    "The previous section showed how to find and retrieve the results of computations for keys automatically added by `Reporter.from_scenario()`.\n",
    "Reporter also provides many helper methods to describe additional computations in step 1 of the workflow.\n",
    "\n",
    "After using these methods, we can continue to describe further calculations using them as input (step 1); or we can `get()` them (step 2).\n",
    "\n",
    "### Converting to pyam representation\n",
    "\n",
    "Here, we'll use the [`Reporter.convert_pyam()`](https://message.iiasa.ac.at/en/latest/reporting.html#message_ix.reporting.Reporter.convert_pyam) method.\n",
    "\n",
    "This adds a conversion to the `IamDataFrame` class from the [`pyam`](https://pyam-iamc.readthedocs.io) package.\n",
    "(`pyam` is built around the [data file format](https://data.ene.iiasa.ac.at/database/) used by the [Integrated Assessment Modeling Consortium](http://www.globalchange.umd.edu/iamc/) (IAMC), and offers plotting and further calculation features.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def collapse_callback(df):\n",
    "    \"\"\"Callback function to populate the IAMC 'variable' column.\"\"\"\n",
    "    df['variable'] = df['l'] + ' energy|' + df['t'] + '|' + df['c']\n",
    "    return df.drop(['c', 'l', 't'], axis=1)\n",
    "\n",
    "new_key = rep.convert_pyam(\n",
    "    quantities=out.drop('h', 'hd', 'm', 'nd', 'yv'),\n",
    "    year_time_dim='ya',\n",
    "    collapse=collapse_callback)\n",
    "\n",
    "new_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that nothing was computed. (We're still in step 1 of the reporting workflow!)\n",
    "However, the method did return a new key for the node added to the graph (in fact, a list with 1 new key; we could have applied `convert_pyam` to multiple nodes at once).\n",
    "This key has the **tag** `':iamc'` added at the end.\n",
    "\n",
    "We describe the added computation, then execute it to get a `pyam.IamDataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_key = new_key[0]  # Unwrap the single item in the list\n",
    "\n",
    "print(rep.describe(new_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = rep.get(new_key)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have retrieved the `pyam` object, we can use its built-in methods to filter and plot the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(\n",
    "    model='Westeros Electrified',\n",
    "    scenario='baseline',\n",
    "    region='Westeros') \\\n",
    "  .line_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom computations\n",
    "\n",
    "Thus far we've described reporting calculations using simple, atomic computations, including those automatically added by `Reporter.from_scenario()`.\n",
    "\n",
    "However—just as in the first, introductory example—computations are merely Python functions.\n",
    "This means they can be *any* function, no matter how complex—and thus makes it easy to insert your existing analysis codes into the graph.\n",
    "\n",
    "To demonstrate this, we add several nodes, each using a custom function.\n",
    "- `as_tidy_data()` operates on the internal Quantity value to coerce it into a `pandas.DataFrame` in a specific format.\n",
    "- `my_plot()` uses a different Python plotting package named [`plotnine`](https://plotnine.readthedocs.io) that implements a “grammar of graphics,” similar to R's `ggplot` package. It returns a plot object without drawing it.\n",
    "- `save_plot()` saves the plot to file.\n",
    "- `draw_plot()` outputs the drawn plot directly.\n",
    "\n",
    "Finally, we define a computation `'do both'` that simply computes two different nodes and returns their outputs in a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotnine as p9\n",
    "\n",
    "def as_tidy_data(qty):\n",
    "    \"\"\"Convert *qty* to a tidy data frame, as expected by plotnine.\"\"\"\n",
    "    return qty.to_frame().reset_index().rename(columns={0: 'value'})\n",
    "\n",
    "def my_plot(data):\n",
    "    \"\"\"Computation that returns a plotnine plot object.\"\"\"\n",
    "    # 'Aes'thetic mappings between column names and parts of the plot\n",
    "    aes = p9.aes(x='ya', y='value', color=\"t + ' ' + c\", shape='l')\n",
    "    \n",
    "    # Set up the plot but don't draw it\n",
    "    plot = (\n",
    "        p9.ggplot(data, aes)       # Create the plot\n",
    "        + p9.geom_line()           # Add a line\n",
    "        + p9.geom_point()          # Add points\n",
    "        + p9.labs(x='Year', y='Energy output', color='Tech & commodity',\n",
    "                  shape='Level'))  # Label axes & legend\n",
    "    \n",
    "    print('Only computed once.')\n",
    "    return plot\n",
    "\n",
    "    \n",
    "def save_plot(obj):\n",
    "    obj.save('westeros_report.pdf')\n",
    "    return 'Saved to westeros_report.pdf'\n",
    "    \n",
    "def draw_plot(obj):\n",
    "    obj.draw()\n",
    "    return 'Drawn in notebook'\n",
    "\n",
    "# Add nodes to the graph\n",
    "rep.add('tidy', (as_tidy_data, out2.drop('yv')))\n",
    "rep.add('plot', (my_plot, 'tidy'))\n",
    "rep.add('save', (save_plot, 'plot'))\n",
    "rep.add('draw', (draw_plot, 'plot'))\n",
    "rep.add('do both', ['save', 'draw'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(rep.describe('do both'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that Reporter avoids calling `my_plot()` repeatedly.\n",
    "Instead, it stores the resulting object just once.\n",
    "When the ‘save’ and ‘draw’ nodes are requested, the same object is passed to each of `save_plot()` and `draw_plot()` in turn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rep.get('do both')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "### Additional features\n",
    "\n",
    "The message_ix reporting code offers other features not covered by this tutorial.\n",
    "See the [documentation](https://message.iiasa.ac.at/en/latest/reporting.html) to learn how to:\n",
    "- Add exogenous (non-model) data to be used in other calculations, with `Reporter.add_file()`.\n",
    "- Use a function to add many nodes at once, with `Reporter.apply()`.\n",
    "- Generate a visual representation of the graph, with `Reporter.visualize()`.\n",
    "\n",
    "### Reporting is ‘experimental’\n",
    "\n",
    "The reporting code in `ixmp` and `message_ix` are in an experimental state, as indicated by the warning at the top of the documentation.\n",
    "\n",
    "This means that features, method signatures, and operation might change at any time, without notice.\n",
    "\n",
    "However, it *also* means we would greatly appreciate:\n",
    "- Reports of your experience using the reporting features in your work, and\n",
    "- Pull requests to extend the feature set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solutions to exercises\n",
    "\n",
    "### Exercise 1\n",
    "\n",
    "The result does not change, because ‘X’ is not needed to calculate ‘C’.\n",
    "\n",
    "### Exercise 2\n",
    "\n",
    "One solution involves adding some intermediate nodes—call them ‘foo1’ and ‘foo2’:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Restore the saved value rep1\n",
    "rep = rep1\n",
    "\n",
    "rep.add('D', 12)\n",
    "rep.add('foo1', (ratio, 'A', 'C'))\n",
    "rep.add('foo2', (product, 'D', 'foo1'))\n",
    "rep.add('E', (sum_calc, 'A', 'foo2'))\n",
    "rep.get('E')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another solution is to define a new anonymous function that computes E in a single step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rep.add('D', 12)\n",
    "rep.add('E', (lambda a, c, d: a + d * (a / c), 'A', 'C', 'D'))\n",
    "rep.get('E')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}