docs/examples/vision_gaia_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vision and Gaia Conversion\n",
    "\n",
    "This example illustrates conversion from Vision or Gaia excel export to power-grid-model input data. \n",
    "They function in a similar way since both are [Tabular Converters](../converters/tabular_converter.md).\n",
    "We can then calculate power-flow with it or convert to a different formats like PGM JSON."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vision conversion\n",
    "\n",
    "### 1. Load the Vision data\n",
    "\n",
    "To export the Vision file in Excel format, please follow the instructions given in the [Vision Manual](https://phasetophase.nl/pdf/VisionEN.pdf).\n",
    "\n",
    "Define source and destination paths:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_file = \"data/vision/example.xlsx\"\n",
    "destination_file = \"data/vision/sym_output.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate the converter, optionally with a source file path.\n",
    "Then use `load_input_data()` to load the data and convert it to power-grid-model data.\n",
    "The additional information that is not used in the powerflow calculation but may be useful to link the results to the source data is stored in `extra_info`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture cap --no-stderr\n",
    "from power_grid_model_io.converters import VisionExcelConverter\n",
    "\n",
    "converter = VisionExcelConverter(source_file=source_file)\n",
    "input_data, extra_info = converter.load_input_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's investigate the data we have converted, for one of the components: `nodes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0, 400.), (1, 400.), (2, 400.), (3, 400.)],\n",
       "      dtype={'names': ['id', 'u_rated'], 'formats': ['<i4', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>u_rated</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>400.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  u_rated\n",
       "0   0    400.0\n",
       "1   1    400.0\n",
       "2   2    400.0\n",
       "3   3    400.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{0: 'First Node', 1: 'Second Node', 2: 'Third Node', 3: 'Fourth Node'}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# The node data is stored as a numpy structured array in input_data[\"node\"]\n",
    "display(input_data[\"node\"])\n",
    "\n",
    "# We can use pandas to display the data in a convenient tabular format\n",
    "display(pd.DataFrame(input_data[\"node\"]))\n",
    "\n",
    "# Notice that the node names were not stored in the numpy array, as we don't need them for the calculations\n",
    "display({i: extra_info[i][\"Name\"] for i in input_data[\"node\"][\"id\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Validate the data\n",
    "Before we run a power flow calculation, it is wise validate the data. The most basic method is to use `assert_valid_input_data()`, which will raise a `ValueError` when the data is invalid. For more details on data validation, please consult the [validation Example](https://github.com/alliander-opensource/power-grid-model/blob/main/docs/examples/Validation%20Examples.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model import CalculationType\n",
    "from power_grid_model.validation import assert_valid_input_data\n",
    "\n",
    "assert_valid_input_data(input_data, calculation_type=CalculationType.power_flow, symmetric=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Run the calculation\n",
    "\n",
    "Run powerflow calculation with the `input_data` and show the results for `nodes`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999991</td>\n",
       "      <td>399.996506</td>\n",
       "      <td>-0.000040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.985602</td>\n",
       "      <td>394.240830</td>\n",
       "      <td>-0.003033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.971222</td>\n",
       "      <td>388.488784</td>\n",
       "      <td>-0.006115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.966073</td>\n",
       "      <td>386.429212</td>\n",
       "      <td>-0.006102</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu           u   u_angle\n",
       "0   0          1  0.999991  399.996506 -0.000040\n",
       "1   1          1  0.985602  394.240830 -0.003033\n",
       "2   2          1  0.971222  388.488784 -0.006115\n",
       "3   3          1  0.966073  386.429212 -0.006102"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from power_grid_model import PowerGridModel\n",
    "\n",
    "pgm = PowerGridModel(input_data=input_data)\n",
    "output_data = pgm.calculate_power_flow()\n",
    "\n",
    "display(pd.DataFrame(output_data[\"node\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross referencing objects\n",
    "The converter has generated unique numerical IDs for all the components in the VisionExcel file, in fact for some special components like _Transformer loads_, multiple PGM components have been created, each with their own numerical ID. To find out which component belongs to which id, some helper functions have been defined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PGM object #0: {'table': 'Nodes', 'key': {'Number': 111}}\n",
      "PGM object #4: {'table': 'Cables', 'key': {'Number': 444}}\n",
      "PGM object #7: {'table': 'Sources', 'key': {'Node.Number': 111, 'Subnumber': 1}}\n",
      "PGM object #9: {'table': 'Loads', 'key': {'Node.Number': 444, 'Subnumber': 1}}\n",
      "Node with Number=111: 0\n",
      "Cables with Number=444: 4\n",
      "Source with Node.Number=111 and Subnumber=1: 7\n",
      "Loads with Node.Number=444 and Subnumber=1: 9\n"
     ]
    }
   ],
   "source": [
    "print(\"PGM object #0:\", converter.lookup_id(0))\n",
    "print(\"PGM object #4:\", converter.lookup_id(4))\n",
    "print(\"PGM object #7:\", converter.lookup_id(7))\n",
    "print(\"PGM object #9:\", converter.lookup_id(9))\n",
    "\n",
    "print(\n",
    "    \"Node with Number=111:\",\n",
    "    converter.get_node_id(number=111)\n",
    ")\n",
    "\n",
    "print(\n",
    "    \"Cables with Number=444:\",\n",
    "    converter.get_branch_id(table=\"Cables\", number=444)\n",
    ")\n",
    "\n",
    "print(\n",
    "    \"Source with Node.Number=111 and Subnumber=1:\",\n",
    "    converter.get_appliance_id(table=\"Sources\", node_number=111, sub_number=1)\n",
    ")\n",
    "\n",
    "print(\n",
    "    \"Loads with Node.Number=444 and Subnumber=1:\",\n",
    "    converter.get_appliance_id(table=\"Loads\", node_number=444, sub_number=1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the data as a JSON file\n",
    "The data can be stored in a json file using the PgmJsonConverter. The file will be saved in the `destination_file` path supplied in the constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model_io.converters import PgmJsonConverter\n",
    "\n",
    "json_converter = PgmJsonConverter(destination_file=destination_file)\n",
    "json_converter.save(data=output_data, extra_info=extra_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For debugging purposes, let's check the output JSON. Notice that the node names are added to the nodes data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<pre style='max-height: 160px; white-space: pre'>{\n",
       "  \"line\":\n",
       "    [\n",
       "      {\"id\": 4, \"energized\": 1, \"loading\": 0.4494033468128292, \"p_from\": 92766.12651731551, \"q_from\": 10913.43660532122, \"i_from\": 134.82094421436187, \"s_from\": 93405.87415984583, \"p_to\": -91463.07558984967, \"q_to\": -10483.005663479944, \"i_to\": 134.82100404384875, \"s_to\": 92061.86835001291, \"id_reference\": {\"table\": \"Cables\", \"key\": {\"Number\": 444}}},\n",
       "      {\"id\": 5, \"energized\": 1, \"loading\": 0.44940353822035317, \"p_from\": 91463.07558984793, \"q_from\": 10483.005663480168, \"i_from\": 134.82100404384624, \"s_from\": 92061.86835001121, \"p_to\": -90160.02352913805, \"q_to\": -10052.564084910864, \"i_to\": 134.82106146610596, \"s_to\": 90718.70748338496, \"id_reference\": {\"table\": \"Cables\", \"key\": {\"Number\": 555}}},\n",
       "      {\"id\": 6, \"energized\": 1, \"loading\": 0.15748825414612666, \"p_from\": 30160.023529136957, \"q_from\": 10052.564084910147, \"i_from\": 47.24631623927688, \"s_from\": 31791.21048276269, \"p_to\": -29999.99999999949, \"q_to\": -9999.999999999573, \"i_to\": 47.246476243837996, \"s_to\": 31622.776601683174, \"id_reference\": {\"table\": \"Cables\", \"key\": {\"Number\": 666}}}\n",
       "    ],\n",
       "  \"node\":\n",
       "    [\n",
       "      {\"id\": 0, \"energized\": 1, \"u_pu\": 0.9999912644198532, \"u\": 399.9965057679413, \"u_angle\": -3.96611370590559e-05, \"id_reference\": {\"table\": \"Nodes\", \"key\": {\"Number\": 111}}, \"Name\": \"First Node\"},\n",
       "      {\"id\": 1, \"energized\": 1, \"u_pu\": 0.9856020738558822, \"u\": 394.24082954235286, \"u_angle\": -0.003033170356205173, \"id_reference\": {\"table\": \"Nodes\", \"key\": {\"Number\": 222}}, \"Name\": \"Second Node\"},\n",
       "      {\"id\": 2, \"energized\": 1, \"u_pu\": 0.9712219592491987, \"u\": 388.4887836996795, \"u_angle\": -0.006115298015992432, \"id_reference\": {\"table\": \"Nodes\", \"key\": {\"Number\": 333}}, \"Name\": \"Third Node\"},\n",
       "      {\"id\": 3, \"energized\": 1, \"u_pu\": 0.966073029937675, \"u\": 386.42921197506996, \"u_angle\": -0.006102268973727995, \"id_reference\": {\"table\": \"Nodes\", \"key\": {\"Number\": 444}}, \"Name\": \"Fourth Node\"}\n",
       "    ],\n",
       "  \"source\":\n",
       "    [\n",
       "      {\"id\": 7, \"energized\": 1, \"p\": 92766.12651730014, \"q\": 10913.436605231671, \"i\": 134.82094421432475, \"s\": 93405.8741598201, \"pf\": 0.9931508842642452, \"id_reference\": {\"table\": \"Sources\", \"key\": {\"Node.Number\": 111, \"Subnumber\": 1}}, \"Name\": \"Netvoeding\"}\n",
       "    ],\n",
       "  \"sym_load\":\n",
       "    [\n",
       "      {\"id\": 8, \"energized\": 1, \"p\": 60000.0, \"q\": -0.0, \"i\": 89.16863910839885, \"s\": 60000.0, \"pf\": 1.0, \"id_reference\": {\"table\": \"Loads\", \"key\": {\"Node.Number\": 333, \"Subnumber\": 1}}},\n",
       "      {\"id\": 9, \"energized\": 1, \"p\": 30000.0, \"q\": 10000.0, \"i\": 47.246476243838906, \"s\": 31622.776601683792, \"pf\": 0.9486832980505139, \"id_reference\": {\"table\": \"Loads\", \"key\": {\"Node.Number\": 444, \"Subnumber\": 1}}}\n",
       "    ]\n",
       "}\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from IPython.display import display, Markdown\n",
    "\n",
    "with Path(destination_file).open() as json_file:\n",
    "    display(Markdown(f\"<pre style='max-height: 160px; white-space: pre'>{json_file.read()}</div>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture cap --no-stderr\n",
    "\n",
    "from power_grid_model import PowerGridModel, CalculationType\n",
    "from power_grid_model.validation import assert_valid_input_data\n",
    "from power_grid_model_io.converters import VisionExcelConverter, PgmJsonConverter\n",
    "\n",
    "source_file = \"data/vision/example.xlsx\"\n",
    "destination_file = \"data/vision/sym_output.json\"\n",
    "\n",
    "converter = VisionExcelConverter(source_file=source_file)\n",
    "input_data, extra_info = converter.load_input_data()\n",
    "assert_valid_input_data(input_data, calculation_type=CalculationType.power_flow, symmetric=True)\n",
    "pgm = PowerGridModel(input_data=input_data)\n",
    "output_data = pgm.calculate_power_flow()\n",
    "json_converter = PgmJsonConverter(destination_file=destination_file)\n",
    "json_converter.save(data=output_data, extra_info=extra_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gaia Conversion\n",
    "\n",
    "```{note}\n",
    "Gaia Conversion is under development currently and will not work as intended\n",
    "```\n",
    "\n",
    "Please refer to the Gaia manual from the software for instructions on Excel exports.\n",
    "Gaia Conversion is similar to Vision conversion, so only a brief code example is given:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model_io.converters.gaia_excel_converter import GaiaExcelConverter\n",
    "\n",
    "source_file = \"data/gaia/example.xlsx\"\n",
    "types_file = \"data/gaia/types.xlsx\"\n",
    "\n",
    "gaia_converter = GaiaExcelConverter(source_file=source_file, types_file=types_file)\n",
    "input_data, extra_info = gaia_converter.load_input_data()\n",
    "assert_valid_input_data(input_data, calculation_type=CalculationType.power_flow, symmetric=True)\n",
    "model = PowerGridModel(input_data=input_data)\n",
    "output_data = model.calculate_power_flow()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "6191baaf994abc6c3e7d918f70d2cb3a487f327a3f3fca1519684915289f78b6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}