docs/examples/vision_gaia_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vision and Gaia Conversion\n",
    "\n",
    "This example illustrates conversion from Vision or Gaia excel export to power-grid-model input data. \n",
    "They function in a similar way since both are [Tabular Converters](../converters/tabular_converter.md).\n",
    "We can then calculate power-flow with it or convert to a different formats like PGM JSON."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vision conversion\n",
    "![Vision example](data/vision/example.png)\n",
    "\n",
    "### 1. Load the Vision data\n",
    "\n",
    "To export the Vision file in Excel format, please follow the instructions given in the [Vision Manual](https://phasetophase.nl/pdf/VisionEN.pdf).\n",
    "\n",
    "Define source and destination paths:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_file = \"data/vision/example.xlsx\"\n",
    "destination_file = \"data/vision/sym_output.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate the converter, optionally with a source file path.\n",
    "Then use `load_input_data()` to load the data and convert it to power-grid-model data.\n",
    "The additional information that is not used in the powerflow calculation but may be useful to link the results to the source data is stored in `extra_info`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture cap --no-stderr\n",
    "from power_grid_model_io.converters import VisionExcelConverter\n",
    "\n",
    "converter = VisionExcelConverter(source_file=source_file)\n",
    "input_data, extra_info = converter.load_input_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's investigate the data we have converted, for one of the components: `nodes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(0, 400.), (1, 400.), (2, 400.), (3, 400.)],\n",
       "      dtype={'names': ['id', 'u_rated'], 'formats': ['<i4', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>u_rated</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>400.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  u_rated\n",
       "0   0    400.0\n",
       "1   1    400.0\n",
       "2   2    400.0\n",
       "3   3    400.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{0: 'First Node', 1: 'Fourth Node', 2: 'Second Node', 3: 'Third Node'}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# The node data is stored as a numpy structured array in input_data[\"node\"]\n",
    "display(input_data[\"node\"])\n",
    "\n",
    "# We can use pandas to display the data in a convenient tabular format\n",
    "display(pd.DataFrame(input_data[\"node\"]))\n",
    "\n",
    "# Notice that the node names were not stored in the numpy array, as we don't need them for the calculations\n",
    "display({i: extra_info[i][\"Name\"] for i in input_data[\"node\"][\"id\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Validate the data\n",
    "Before we run a power flow calculation, it is wise validate the data. The most basic method is to use `assert_valid_input_data()`, which will raise a `ValueError` when the data is invalid. For more details on data validation, please consult the [validation Example](https://github.com/alliander-opensource/power-grid-model/blob/main/docs/examples/Validation%20Examples.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model import CalculationType\n",
    "from power_grid_model.validation import assert_valid_input_data\n",
    "\n",
    "assert_valid_input_data(input_data, calculation_type=CalculationType.power_flow, symmetric=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Run the calculation\n",
    "\n",
    "Run powerflow calculation with the `input_data` and show the results for `nodes`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999991</td>\n",
       "      <td>399.996569</td>\n",
       "      <td>-0.000039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.983232</td>\n",
       "      <td>393.292609</td>\n",
       "      <td>-0.003435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.987662</td>\n",
       "      <td>395.064831</td>\n",
       "      <td>-0.003235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.979773</td>\n",
       "      <td>391.909191</td>\n",
       "      <td>-0.006297</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu           u   u_angle\n",
       "0   0          1  0.999991  399.996569 -0.000039\n",
       "1   1          1  0.983232  393.292609 -0.003435\n",
       "2   2          1  0.987662  395.064831 -0.003235\n",
       "3   3          1  0.979773  391.909191 -0.006297"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from power_grid_model import PowerGridModel\n",
    "\n",
    "pgm = PowerGridModel(input_data=input_data)\n",
    "output_data = pgm.calculate_power_flow()\n",
    "\n",
    "display(pd.DataFrame(output_data[\"node\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross referencing objects\n",
    "The converter has generated unique numerical IDs for all the components in the VisionExcel file, in fact for some special components like _Transformer loads_, multiple PGM components have been created, each with their own numerical ID. To find out which component belongs to which id, some helper functions have been defined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PGM object #0: {'table': 'Nodes', 'key': {'Number': 1}}\n",
      "PGM object #4: {'table': 'Cables', 'key': {'Number': 6}}\n",
      "PGM object #7: {'table': 'Sources', 'key': {'Node.Number': 1, 'Subnumber': 1}}\n",
      "PGM object #9: {'table': 'Loads', 'key': {'Node.Number': 4, 'Subnumber': 1}}\n",
      "Node with Number=1: 0\n",
      "Cables with Number=6: 4\n",
      "Source with Node.Number=1 and Subnumber=1: 7\n",
      "Loads with Node.Number=4 and Subnumber=1: 9\n"
     ]
    }
   ],
   "source": [
    "print(\"PGM object #0:\", converter.lookup_id(0))\n",
    "print(\"PGM object #4:\", converter.lookup_id(4))\n",
    "print(\"PGM object #7:\", converter.lookup_id(7))\n",
    "print(\"PGM object #9:\", converter.lookup_id(9))\n",
    "\n",
    "print(\n",
    "    \"Node with Number=1:\",\n",
    "    converter.get_node_id(number=1)\n",
    ")\n",
    "\n",
    "print(\n",
    "    \"Cables with Number=6:\",\n",
    "    converter.get_branch_id(table=\"Cables\", number=6)\n",
    ")\n",
    "\n",
    "print(\n",
    "    \"Source with Node.Number=1 and Subnumber=1:\",\n",
    "    converter.get_appliance_id(table=\"Sources\", node_number=1, sub_number=1)\n",
    ")\n",
    "\n",
    "print(\n",
    "    \"Loads with Node.Number=4 and Subnumber=1:\",\n",
    "    converter.get_appliance_id(table=\"Loads\", node_number=4, sub_number=1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the data as a JSON file\n",
    "The data can be stored in a json file using the PgmJsonConverter. The file will be saved in the `destination_file` path supplied in the constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model_io.converters import PgmJsonConverter\n",
    "\n",
    "json_converter = PgmJsonConverter(destination_file=destination_file)\n",
    "json_converter.save(data=output_data, extra_info=extra_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For debugging purposes, let's check the output JSON. Notice that the node names are added to the nodes data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<pre style='max-height: 160px; white-space: pre'>{\n",
       "  \"line\":\n",
       "    [\n",
       "      {\"id\": 4, \"energized\": 1, \"loading\": 0.15473990756817002, \"p_from\": 30133.178503728053, \"q_from\": 10050.721853721689, \"i_from\": 46.421809438989094, \"s_from\": 31765.16104977932, \"p_to\": -29999.9999999991, \"q_to\": -10000.000000000018, \"i_to\": 46.42197227045101, \"s_to\": 31622.776601682945, \"id_reference\": {\"table\": \"Cables\", \"key\": {\"Number\": 6}}},\n",
       "      {\"id\": 5, \"energized\": 1, \"loading\": 0.2946347177166794, \"p_from\": 60482.835089115535, \"q_from\": 184.81468538821002, \"i_from\": 88.39041452675093, \"s_from\": 60483.11745342729, \"p_to\": -60000.00000000256, \"q_to\": -2.981555974335137e-10, \"i_to\": 88.3904153150038, \"s_to\": 60000.00000000256, \"id_reference\": {\"table\": \"Cables\", \"key\": {\"Number\": 5}}},\n",
       "      {\"id\": 6, \"energized\": 1, \"loading\": 0.44422983507391905, \"p_from\": 91713.619010016, \"q_from\": 10656.105417735627, \"i_from\": 133.26889142389211, \"s_from\": 92330.60432266358, \"p_to\": -90616.01359284009, \"q_to\": -10235.536539109033, \"i_to\": 133.26895052217571, \"s_to\": 91192.25914353265, \"id_reference\": {\"table\": \"Cables\", \"key\": {\"Number\": 4}}}\n",
       "    ],\n",
       "  \"node\":\n",
       "    [\n",
       "      {\"id\": 0, \"energized\": 1, \"u_pu\": 0.999991421302051, \"u\": 399.9965685208204, \"u_angle\": -3.921691889422987e-05, \"id_reference\": {\"table\": \"Nodes\", \"key\": {\"Number\": 1}}, \"ID\": 101, \"Name\": \"First Node\"},\n",
       "      {\"id\": 1, \"energized\": 1, \"u_pu\": 0.983231522194849, \"u\": 393.2926088779396, \"u_angle\": -0.003434581394274798, \"id_reference\": {\"table\": \"Nodes\", \"key\": {\"Number\": 4}}, \"ID\": 104, \"Name\": \"Fourth Node\"},\n",
       "      {\"id\": 2, \"energized\": 1, \"u_pu\": 0.9876620765421245, \"u\": 395.0648306168498, \"u_angle\": -0.003235042286685305, \"id_reference\": {\"table\": \"Nodes\", \"key\": {\"Number\": 2}}, \"ID\": 102, \"Name\": \"Second Node\"},\n",
       "      {\"id\": 3, \"energized\": 1, \"u_pu\": 0.9797729773054616, \"u\": 391.9091909221847, \"u_angle\": -0.006296501518419363, \"id_reference\": {\"table\": \"Nodes\", \"key\": {\"Number\": 3}}, \"ID\": 103, \"Name\": \"Third Node\"}\n",
       "    ],\n",
       "  \"source\":\n",
       "    [\n",
       "      {\"id\": 7, \"energized\": 1, \"p\": 91713.61901002181, \"q\": 10656.105417760706, \"i\": 133.26889142390465, \"s\": 92330.60432267224, \"pf\": 0.9933176510954675, \"id_reference\": {\"table\": \"Sources\", \"key\": {\"Node.Number\": 1, \"Subnumber\": 1}}, \"Name\": \"Grid\"}\n",
       "    ],\n",
       "  \"sym_load\":\n",
       "    [\n",
       "      {\"id\": 8, \"energized\": 1, \"p\": 60000.0, \"q\": -0.0, \"i\": 88.39041531500003, \"s\": 60000.0, \"pf\": 1.0, \"id_reference\": {\"table\": \"Loads\", \"key\": {\"Node.Number\": 3, \"Subnumber\": 1}}},\n",
       "      {\"id\": 9, \"energized\": 1, \"p\": 30000.0, \"q\": 10000.0, \"i\": 46.421972270452244, \"s\": 31622.776601683792, \"pf\": 0.9486832980505139, \"id_reference\": {\"table\": \"Loads\", \"key\": {\"Node.Number\": 4, \"Subnumber\": 1}}}\n",
       "    ]\n",
       "}\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from IPython.display import display, Markdown\n",
    "\n",
    "with Path(destination_file).open() as json_file:\n",
    "    display(Markdown(f\"<pre style='max-height: 160px; white-space: pre'>{json_file.read()}</div>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture cap --no-stderr\n",
    "\n",
    "from power_grid_model import PowerGridModel, CalculationType\n",
    "from power_grid_model.validation import assert_valid_input_data\n",
    "from power_grid_model_io.converters import VisionExcelConverter, PgmJsonConverter\n",
    "\n",
    "source_file = \"data/vision/example.xlsx\"\n",
    "destination_file = \"data/vision/sym_output.json\"\n",
    "\n",
    "converter = VisionExcelConverter(source_file=source_file)\n",
    "input_data, extra_info = converter.load_input_data()\n",
    "assert_valid_input_data(input_data, calculation_type=CalculationType.power_flow, symmetric=True)\n",
    "pgm = PowerGridModel(input_data=input_data)\n",
    "output_data = pgm.calculate_power_flow()\n",
    "json_converter = PgmJsonConverter(destination_file=destination_file)\n",
    "json_converter.save(data=output_data, extra_info=extra_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gaia Conversion\n",
    "\n",
    "```{note}\n",
    "Gaia Conversion is under development currently and will not work as intended\n",
    "```\n",
    "\n",
    "Please refer to the Gaia manual from the software for instructions on Excel exports.\n",
    "Gaia Conversion is similar to Vision conversion, so only a brief code example is given:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model_io.converters.gaia_excel_converter import GaiaExcelConverter\n",
    "\n",
    "source_file = \"data/gaia/example.xlsx\"\n",
    "types_file = \"data/gaia/types.xlsx\"\n",
    "\n",
    "gaia_converter = GaiaExcelConverter(source_file=source_file, types_file=types_file)\n",
    "input_data, extra_info = gaia_converter.load_input_data()\n",
    "assert_valid_input_data(input_data, calculation_type=CalculationType.power_flow, symmetric=True)\n",
    "model = PowerGridModel(input_data=input_data)\n",
    "output_data = model.calculate_power_flow()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "6191baaf994abc6c3e7d918f70d2cb3a487f327a3f3fca1519684915289f78b6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}