Use grid model to create scenario by rouille · Pull Request #398 · Breakthrough-Energy/PowerSimData

rouille · 2021-02-16T20:38:09Z

Purpose

Allow the user to specify the grid model he wants to use in his scenario and access the corresponding profiles.

What the code is doing

It instantiates the _Builder class with an additional parameter grid_model. To use the usa_tamu grid model for the Western interconnect, the user will call the set_builder method in the Create class as follows:

from powersimdata.scenario.scenario import Scenario

scenario = Scenario('')
scenario.state.set_builder("usa_tamu", "Western")

By default set_builder will instantiate the Builder class for the full U.S. TAMU grid.

Testing

The following script:

from powersimdata.scenario.scenario import Scenario

scenario = Scenario('')

# Start building a scenario
scenario.state.set_builder("usa_tamu", "Western")

# set plan and scenario names
scenario.state.builder.set_name("test", "dummy")
# set start date, end date and interval
scenario.state.builder.set_time("2016-08-01 00:00:00",
                                "2016-08-31 23:00:00",
                                "24H")
# set demand profile version
scenario.state.builder.set_base_profile("demand", "vJan2021")
# set hydro profile version
scenario.state.builder.set_base_profile("hydro", "vJan2021")
# set solar profile version
scenario.state.builder.set_base_profile("solar", "vJan2021")
# set wind profile version
scenario.state.builder.set_base_profile("wind", "vJan2021")

# scale capacity of solar plants in WA and AZ by 5 and 2.5, respectively
scenario.state.builder.change_table.scale_plant_capacity(
    "solar", zone_name={"Washington": 5, "Arizona": 2.5})
# scale capacity of wind farms in OR and MT by 1.5 and 2, respectively
scenario.state.builder.change_table.scale_plant_capacity(
    "wind", zone_name={"Oregon": 1.5, "Montana Western": 2})
# scale capacity of branches in NV and WY by 2
scenario.state.builder.change_table.scale_branch_capacity(
    zone_name={"Nevada": 2, "Wyoming": 2})
# add AC lines in NM and CO
scenario.state.builder.change_table.add_branch(
    [{"capacity": 200, "from_bus_id": 2053002, "to_bus_id": 2053303},
     {"capacity": 150, "from_bus_id": 2060002, "to_bus_id": 2060046}])
# add DC line between CO and CA (Bay Area)
scenario.state.builder.change_table.add_dcline(
    [{"capacity": 2000, "from_bus_id": 2060771, "to_bus_id": 2021598}])
# add a solar plant in NV, a coal plant in ID and a natural gas plant in OR
scenario.state.builder.change_table.add_plant(
    [{"type": "solar", "bus_id": 2030454, "Pmax": 75},
     {"type": "coal", "bus_id": 2074334, "Pmin": 25, "Pmax": 750, "c0": 1800, "c1": 30, "c2": 0.0025},
     {"type": "ng", "bus_id": 2090018, "Pmax": 75, "c0": 900, "c1": 30, "c2": 0.0015}])
# add a new bus, and a new one-way DC line connected to this bus
scenario.state.builder.change_table.add_bus(
    [{"lat": 48, "lon": -125, "zone_id": 201, "baseKV": 138}])
scenario.state.builder.change_table.add_dcline(
    [{"from_bus_id": 2090023, "to_bus_id": 2090024, "Pmin": 0, "Pmax": 200}])

# get grid used in scenario
grid = scenario.state.get_grid()
print(grid.plant)
# get change table used to alter the base grid.
ct = scenario.state.get_ct()
print(ct)

# review information
scenario.state.print_scenario_info()

prints the expected outputs:

Transferring ScenarioList.csv from server
100%|########################################################################################################################| 408k/408k [00:00<00:00, 3.57Mb/s]
Reading bus.csv
Reading plant.csv
Reading gencost.csv
Reading branch.csv
Reading dcline.csv
Reading sub.csv
Reading bus2sub.csv
Reading zone.csv
--> Summary
# Existing study
test | Julia | Anchor | CEM | Terrapower
# Available profiles
demand: vJan2021
hydro: vJan2021
solar: vJan2021
wind: vJan2021
        bus_id      Pg     Qg   Qmax  Qmin      Vg   mBase  status  ...  GenFuelCost  GenIOB  GenIOC  GenIOD  zone_id           zone_name        lat         lon
10390  2010683    0.00   0.00  61.16  -7.8  1.0464  207.74       0  ...          0.0     0.0     0.0       0      201          Washington  46.645100 -119.908000
10391  2010684   85.29  56.74  61.16  -7.8  1.0201  201.13       1  ...          0.0     0.0     0.0       0      201          Washington  46.645100 -119.908000
10392  2010685  138.65  56.74  61.16  -7.8  1.0201  217.36       1  ...          0.0     0.0     0.0       0      201          Washington  46.645100 -119.908000
10393  2010686    0.00   0.00  61.16  -7.8  1.0464  214.15       0  ...          0.0     0.0     0.0       0      201          Washington  46.645100 -119.908000
10394  2010687  120.63  56.74  61.16  -7.8  1.0188  188.08       1  ...          0.0     0.0     0.0       0      201          Washington  46.645100 -119.908000
...        ...     ...    ...    ...   ...     ...     ...     ...  ...          ...     ...     ...     ...      ...                 ...        ...         ...
14018  2060594    0.00   0.00   0.00   0.0  1.0000   75.00       1  ...          0.0     0.0     0.0       0      212            Colorado  37.864800 -104.109000
14019  2053000    0.00   0.00   0.00   0.0  1.0000   75.00       1  ...          0.0     0.0     0.0       0      211  New Mexico Western  35.575300 -104.898000
14020  2030454    0.00   0.00   0.00   0.0  0.0000    0.00       1  ...          0.0     0.0     0.0       0      208              Nevada  36.015500 -114.738000
14021  2074334    0.00   0.00   0.00   0.0  0.0000    0.00       1  ...          0.0     0.0     0.0       0      214               Idaho  42.880000 -115.037000
14022  2090018    0.00   0.00   0.00   0.0  0.0000    0.00       1  ...          0.0     0.0     0.0       0      202              Oregon  44.675332 -124.434329

[2619 rows x 35 columns]
{'solar': {'zone_id': {201: 5, 209: 2.5}}, 'wind': {'zone_id': {202: 1.5, 215: 2}}, 'branch': {'zone_id': {208: 2, 213: 2}}, 'new_branch': [{'from_bus_id': 2053002, 'to_bus_id': 2053303, 'Pmax': 200, 'Pmin': -200}, {'from_bus_id': 2060002, 'to_bus_id': 2060046, 'Pmax': 150, 'Pmin': -150}], 'new_dcline': [{'from_bus_id': 2060771, 'to_bus_id': 2021598, 'Pmax': 2000, 'Pmin': -2000}, {'from_bus_id': 2090023, 'to_bus_id': 2090024, 'Pmin': 0, 'Pmax': 200}], 'new_plant': [{'type': 'solar', 'bus_id': 2030454, 'Pmax': 75, 'Pmin': 0, 'plant_id_neighbor': 11977}, {'type': 'coal', 'bus_id': 2074334, 'Pmin': 25, 'Pmax': 750, 'c0': 1800, 'c1': 30, 'c2': 0.0025}, {'type': 'ng', 'bus_id': 2090018, 'Pmax': 75, 'c0': 900, 'c1': 30, 'c2': 0.0015, 'Pmin': 0}], 'new_bus': [{'lat': 48, 'lon': -125, 'zone_id': 201, 'baseKV': 138, 'Pd': 0}]}
--------------------
SCENARIO INFORMATION
--------------------
plan: test
name: dummy
state: create
interconnect: Western
base_demand: vJan2021
base_hydro: vJan2021
base_solar: vJan2021
base_wind: vJan2021
change_table: Yes
start_date: 2016-08-01 00:00:00
end_date: 2016-08-31 23:00:00
interval: 24H
engine: REISE.jl
grid_model: usa_tamu

Note that I did not create a scenario since the ScenarioList.csv is not yet formatted to handle the extra grid_model information. This will be the next step.

Where to look

write ModelImmutables class in powersimdata.network.model: abstract the grid model
rename powersimdata.network.usa_tamu.usa_tamu_model as powersimdata.network.usa_tamu.model and change import statements.
empty __init__ file
refactor are_to_loadzone function in powersimdata.network.usa_tamu.model: remove dependency on the a Grid object
refactor code/tests using area_to_loadzone
add interconnect_to_name function in powersimdata.network.usa_tamu.model: convert a list of interconnect(s) to a string that will be written in the interconnect column of the ScenarioList.csv, e.g. ["Western", "texas"] --> "Texas_Western"
move usa_tamu related tests in powersimdata.input.tests.test_grid into a new powersimdata.network.usa_tamu.tests.test_model
change calling signature of get_profile_version method of InputData class
modify standard definition of dir in sub-modules powersimdata.network.usa_tamu.constants following PEP 562 to allow easy look up in ModelImmutables class
refactor the copy_base_profile of the BackupDisk class in powersimdata.scenario.move module
delete the _create_link_to_base_profile and _prepare_transformed_profile methods of the SimulationInput class in powersimdata.scenario.executemodule. Refactor the _prepare_profile one. The reason is that we cannot create link to base profiles because the profile being normalized, we need to scale the values by the nameplate capacity of the plants at the very least. We could still create a symbolic link to the demand profiles but these being small I decided to keep the code simple and make no distinction among profiles.
refactor the Create and _Builder classes in powersimdata.scenario.create module. The module has been greatly simplified.
delete unused powersimdata.scenario.helpers module

Time estimate

It will be long

BainanXia · 2021-02-17T17:56:09Z

I've read through this line by line and would like to checkout this branch to use it for a while as integration tests before approving. Thanks for handling the refactor of area_to_loadzone function and fixing the corresponding tests. It's everywhere. All tests passed!

Update: based on the discussion with @rouille over the phone, to clarify, we can't use this branch to run scenarios until we manually fix the ScenarioList.csv on the server by adding a "grid model" column at the end. The reason I would like to check it out for now is to use updated area_to_loadzone function in PostREISE refactor as well as the ModelImmutable class to reduce the refactor work in the future. As for the integration tests, I suggest to run test scenarios in a fresh dockerization version of the framework with an empty scenario list.

danielolsen · 2021-02-17T21:34:15Z

+def test_interconnect_type():
+    interconnect = "Western"
+    with pytest.raises(TypeError):
+        check_interconnect(interconnect)


Is this showing an unnecessarily strict behavior for check_interconnect? We can now instantiate a Grid using Grid("Western"), are there other places where we do this check before doing something that really needs a list rather than a string?

Looking where the Check_interconnect is used, I would say we can implement the:

if isinstance(interconnect, str): interconnect = [interconnect]

inside the function and instantiate a ModelImmutables class in the constructor of the Grid class to check the interconnect. What do you think?

I like that code pattern that gives us a bit more flexibility on the inputs when the intent is unambiguous.

I will refactor it then

This is addressed in the last two commits.

Also, you can do that now:

(PowerSimData) [~/CEM/PowerSimData] (ben/create) brdo$ python Python 3.8.3 (default, Jul 8 2020, 14:27:55) [Clang 11.0.3 (clang-1103.0.32.62)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from powersimdata.input.grid import Grid >>> g = Grid(["Western"]) Reading bus.csv Reading plant.csv Reading gencost.csv Reading branch.csv Reading dcline.csv Reading sub.csv Reading bus2sub.csv Reading zone.csv >>> g.model_immutables.zones.keys() dict_keys(['abv', 'abv2interconnect', 'abv2loadzone', 'abv2state', 'id2abv', 'id2loadzone', 'id2timezone', 'interconnect', 'interconnect2abv', 'interconnect2id', 'interconnect2loadzone', 'interconnect2timezone', 'interconnect_combinations', 'loadzone', 'loadzone2id', 'loadzone2interconnect', 'loadzone2state', 'mappings', 'state', 'state2abv', 'state2loadzone', 'timezone2id'])

It does not work right now when we try to instantiate the Grid class from a file since we don't have access to the grid_model value in the ScenarioList file:

>>> from powersimdata.scenario.scenario import Scenario >>> s = Scenario(1712) Failed to download ScenarioList.csv from server Falling back to local cache... Failed to download ExecuteList.csv from server Falling back to local cache... SCENARIO: test | new_bus --> State analyze --> Loading grid Failed to download ScenarioList.csv from server Falling back to local cache... Traceback (most recent call last): File "/Users/brdo/CEM/PowerSimData/powersimdata/input/grid.py", line 33, in __init__ self.model_immutables = ModelImmutables(source) File "/Users/brdo/CEM/PowerSimData/powersimdata/network/model.py", line 11, in __init__ self._check_model(model) File "/Users/brdo/CEM/PowerSimData/powersimdata/network/model.py", line 31, in _check_model raise ValueError("model must be one of %s" % " | ".join(possible)) ValueError: model must be one of usa_tamu During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3080, in get_loc return self._engine.get_loc(casted_key) File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 4554, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 4562, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'grid_model' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/brdo/CEM/PowerSimData/powersimdata/scenario/scenario.py", line 40, in __init__ self.state = Analyze(self) File "/Users/brdo/CEM/PowerSimData/powersimdata/scenario/analyze.py", line 39, in __init__ self._set_ct_and_grid() File "/Users/brdo/CEM/PowerSimData/powersimdata/scenario/analyze.py", line 50, in _set_ct_and_grid self.grid = Grid( File "/Users/brdo/CEM/PowerSimData/powersimdata/input/grid.py", line 36, in __init__ _get_grid_model_from_scenario_list(source) File "/Users/brdo/CEM/PowerSimData/powersimdata/input/grid.py", line 165, in _get_grid_model_from_scenario_list return slm.get_scenario_table().loc[scenario_number, "grid_model"] File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/indexing.py", line 888, in __getitem__ return self._getitem_tuple(key) File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/indexing.py", line 1059, in _getitem_tuple return self._getitem_lowerdim(tup) File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/indexing.py", line 830, in _getitem_lowerdim return getattr(section, self.name)[new_key] File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/indexing.py", line 894, in __getitem__ return self._getitem_axis(maybe_callable, axis=axis) File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/indexing.py", line 1123, in _getitem_axis return self._get_label(key, axis=axis) File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/indexing.py", line 1072, in _get_label return self.obj.xs(label, axis=axis) File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/generic.py", line 3736, in xs loc = index.get_loc(key) File "/Users/brdo/.local/share/virtualenvs/PowerSimData-MpUK62nT/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3082, in get_loc raise KeyError(key) from err KeyError: 'grid_model'

jenhagg · 2021-02-23T01:06:58Z

+    """
+    scenario_number = int(os.path.basename(source).split("_")[0])
+    slm = ScenarioListManager(Context.get_data_access())
+    return slm.get_scenario_table().loc[scenario_number, "grid_model"]


You can also use slm.get_scenario(scenario_number)["grid_model"] (no real difference, just FYI)

>>> from powersimdata.scenario.scenario import Scenario >>> s = Scenario(1712) Failed to download ScenarioList.csv from server Falling back to local cache... Failed to download ExecuteList.csv from server Falling back to local cache... SCENARIO: test | new_bus --> State analyze --> Loading grid Failed to download ScenarioList.csv from server Falling back to local cache... Traceback (most recent call last): File "/Users/brdo/CEM/PowerSimData/powersimdata/input/grid.py", line 33, in __init__ self.model_immutables = ModelImmutables(source) File "/Users/brdo/CEM/PowerSimData/powersimdata/network/model.py", line 11, in __init__ self._check_model(model) File "/Users/brdo/CEM/PowerSimData/powersimdata/network/model.py", line 31, in _check_model raise ValueError("model must be one of %s" % " | ".join(possible)) ValueError: model must be one of usa_tamu During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/brdo/CEM/PowerSimData/powersimdata/scenario/scenario.py", line 40, in __init__ self.state = Analyze(self) File "/Users/brdo/CEM/PowerSimData/powersimdata/scenario/analyze.py", line 39, in __init__ self._set_ct_and_grid() File "/Users/brdo/CEM/PowerSimData/powersimdata/scenario/analyze.py", line 50, in _set_ct_and_grid self.grid = Grid( File "/Users/brdo/CEM/PowerSimData/powersimdata/input/grid.py", line 36, in __init__ _get_grid_model_from_scenario_list(source) File "/Users/brdo/CEM/PowerSimData/powersimdata/input/grid.py", line 165, in _get_grid_model_from_scenario_list return slm.get_scenario(scenario_number)["grid_model"] KeyError: 'grid_model'

Somehow the Traceback is smaller with this implementation. Not sure why ....

…lass

rouille added grid feature request Request for a new feature. (Only lives in Backlog) labels Feb 16, 2021

rouille added this to the HTAACD milestone Feb 16, 2021

rouille requested review from BainanXia, ahurli, danielolsen and jenhagg February 16, 2021 20:38

rouille self-assigned this Feb 16, 2021

rouille force-pushed the ben/create branch from bb8b994 to ae38970 Compare February 16, 2021 20:41