feat: add REISEjlMATReader to interpret input.mat from REISE.jl

[danielolsen]

Purpose

To be able to interpret input.mat files generated by REISE.jl, so that we can recreate a Grid based on files saved on the server. This relies on REISE.jl saving converting all the extra data we need, see Breakthrough-Energy/REISE.jl#37.

What is the code doing

In grid.py: we import the new REISEjlMATReader and instantiate it when we pass an engine of 'REISE.jl'.

In mat_reader.py:

• we move most of the grid reading/parsing code from REISEMATReader._build_network() to MATReader._read_network(), because it can be reused.
• we move the the 'reindexing buses' code out of add_information_to_model function, and into a new function reindex_model, so that add_information_to_model only has 'adding columns to dataframes' code.
• these moves make the _build_network() methods in both REISEMATReader and REISEjlMATReader extremely short: they both call _read_network (from MATReader), REISEMATReader calls reindex_model, and then they both call add_information_to_model.

Documentation that everything works as intended

Unit tests all pass.

For an integration test, we will start with Scenario 419 (Western with DC lines). First, we need to run REISE.jl with the save_extra_grid_data branch checked out to save the extra info to to generate the input.mat file. Then, we need to locally run the conversion from a v7.3 matfile to a v7 matfile (see Breakthrough-Energy/REISE.jl#37 which has code for both of these conversion). We save the new file as input2.mat.

First, we want to make sure that we have not changed anything from the old REISEMATReader object, so with develop on powersimdata checked out:

>>> import pickle
>>> from powersimdata.scenario.scenario import Scenario
>>> scenario = Scenario('419')
SCENARIO: test | WesternBase24

--> State
analyze
>>> grid = scenario.state.get_grid()
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> with open('419_Grid.pkl', 'wb') as f:
...     pickle.dump(grid, f)
...
>>> # We will also do this for Scenario 160, while we're at it
...
>>> scenario = Scenario('160')
SCENARIO: base | WACA_Anchor_AllMatchCA_2030_SpurUpgrade_OB_1_Storage1325

--> State
analyze
>>> grid = scenario.state.get_grid()
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> with open('160_Grid.pkl', 'wb') as f:
...     pickle.dump(grid, f)
...
>>> quit()

Now let's check out the reisejlmatreader branch for powersimdata, re-generate the Grid from the input.mat file, and confirm that they are equal:

>>> import pickle
>>> from powersimdata.scenario.scenario import Scenario
>>> from powersimdata.input.grid import Grid
>>> scenario = Scenario('419')
SCENARIO: test | WesternBase24

--> State
analyze
>>> new_grid = scenario.state.get_grid()
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> with open('419_Grid.pkl', 'rb') as f:
...     old_grid = pickle.load(f)
...
>>> old_grid.fields
{'bus': <powersimdata.input.grid_fields.Bus object at 0x0000027C446D86C8>, 'branch': <powersimdata.input.grid_fields.Branch object at 0x0000027C433BCF88>, 'dcline': <powersimdata.input.grid_fields.DCLine object at 0x0000027C44C58D08>, 'gencost': <powersimdata.input.grid_fields.GenCost object at 0x0000027C44C58D48>, 'plant': <powersimdata.input.grid_fields.Plant object at 0x0000027C44C58D88>, 'sub': <powersimdata.input.grid_fields.Sub object at 0x0000027C44EACFC8>, 'storage': <powersimdata.input.grid_fields.Storage object at 0x0000027C445FDFC8>}
>>> old_grid.fields.keys() == new_grid.fields.keys()
True
>>> (old_grid.bus == new_grid.bus).all().all()
True
>>> (old_grid.branch == new_grid.branch).all().all()
True
>>> (old_grid.dcline == new_grid.dcline).all().all()
True
>>> (old_grid.gencost['before'] == new_grid.gencost['before']).all().all()
True
>>> (old_grid.gencost['after'] == new_grid.gencost['after']).all().all()
True
>>> (old_grid.plant == new_grid.plant).all().all()
True
>>> (old_grid.sub == new_grid.sub).all().all()
True
>>> old_grid.storage == new_grid.storage
True

Great! So our refactor has not changed anything in the Grid objects generated by the REISEMATReader objects. But what about the Grid objects generated by the new REISEjlMATReader objects?

>>> reisejl_grid = Grid(interconnect=['Western'], source=r'C:\Users\dolsen\repos\iv\scenario_419\julia_output_test_extra_grid_info\input2.mat', engine='REISE.jl')
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> reisejl_grid.fields.keys() == new_grid.fields.keys()
True
>>> (reisejl_grid.bus == new_grid.bus).all().all()
True
>>> (reisejl_grid.branch == new_grid.branch).all().all()
True
>>> (reisejl_grid.dcline == new_grid.dcline).all().all()
True
>>> (reisejl_grid.gencost['before'] == new_grid.gencost['before']).all().all()
True
>>> (reisejl_grid.gencost['after'] == new_grid.gencost['after']).all().all()
True
>>> (reisejl_grid.plant == new_grid.plant).all().all()
True
>>> (reisejl_grid.sub == new_grid.sub).all().all()
True
>>> reisejl_grid.storage == new_grid.storage
True

Great! The Grids created by reading the input.mat files are identical.

Repeating for Scenario 160, which has storage:

>>> from powersimdata.scenario.scenario import Scenario
>>> from powersimdata.input.grid import Grid
>>> scenario = Scenario('160')
SCENARIO: base | WACA_Anchor_AllMatchCA_2030_SpurUpgrade_OB_1_Storage1325

--> State
analyze
>>> import pickle
>>> new_grid = scenario.state.get_grid()
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> with open('160_Grid.pkl', 'rb') as f:
...     old_grid = pickle.load(f)
...
>>> (old_grid.bus == new_grid.bus).all().all()
True
>>> (old_grid.branch == new_grid.branch).all().all()
True
>>> (old_grid.dcline == new_grid.dcline).all().all()
True
>>> (old_grid.gencost['before'] == new_grid.gencost['before']).all().all()
True
>>> (old_grid.gencost['after'] == new_grid.gencost['after']).all().all()
True
>>> (old_grid.plant == new_grid.plant).all().all()
True
>>> (old_grid.sub == new_grid.sub).all().all()
True
>>> old_grid.storage == new_grid.storage
True
>>> old_grid.fields.keys() == new_grid.fields.keys()
True

So we can confirm that the refactor of REISEMATReader didn't mess with anything to do with storage either. Confirming that the new REISEjlMATReader works is a bit more involved, because the case.mat file for Scenario 160 on the server was created before we started saving all our extra info in there. @rouille did the reprocessing in the 160_grid.mat file on the server, so let's use that file to create the case.mat file that we would expect to see created today. In MATLAB:

>> grid = load('C:\Users\dolsen\repos\iv\scenario_160\160_grid.mat');
>> ref_mpc = grid.mdi.mpc;
>> % transposing 1xN arrays to Nx1, since MAT.jl doesn't have a 'squeeze_me' option
>> ref_mpc.genid = ref_mpc.genid';
>> ref_mpc.genfuel = ref_mpc.genfuel';
>> ref_mpc.branchid = ref_mpc.branchid';
>> ref_mpc.iess = ref_mpc.iess';
>> % resetting gencost to its original values, before linearization
>> ref_mpc.gencost = ref_mpc.gencost_orig;
>> % trimming extra lines off of gen and genfuel, that were added for storage
>> ref_mpc.gen = ref_mpc.gen(1:size(ref_mpc.genid,1), :);
>> ref_mpc.genfuel = ref_mpc.genfuel(1:size(ref_mpc.genid,1), :);
>> % undoing the ext2int reindexing of buses
>> for i = 1:size(ref_mpc.bus, 1)
ref_mpc.bus(i,1) = ref_mpc.busid(ref_mpc.bus(i,1),1);
end
>> for i = 1:size(ref_mpc.gen, 1)
ref_mpc.gen(i,1) = ref_mpc.busid(ref_mpc.gen(i,1),1);
end
>> for i = 1:size(ref_mpc.branch, 1)
ref_mpc.branch(i,1) = ref_mpc.busid(ref_mpc.branch(i,1),1);
ref_mpc.branch(i,2) = ref_mpc.busid(ref_mpc.branch(i,2),1);
end
>> mpc = ref_mpc;
>> save('C:\Users\dolsen\repos\iv\scenario_160\case.mat', 'mpc');

Now we can use this case.mat file to run REISE.jl, which will produce an input.mat file, which can be converted from v7.3 to v7. Finally, we can load this file in and confirm that the created Grid objects are the same:

>>> from powersimdata.scenario.scenario import Scenario
>>> from powersimdata.input.grid import Grid
>>> scenario = Scenario('160')
SCENARIO: base | WACA_Anchor_AllMatchCA_2030_SpurUpgrade_OB_1_Storage1325

--> State
analyze
>>> new_grid = scenario.state.get_grid()
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> reisejl_grid = Grid(interconnect=['Western'], source=r'C:\Users\dolsen\repos\iv\scenario_160\julia_output\input2.mat', engine='REISE.jl')
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> reisejl_grid.fields.keys() == new_grid.fields.keys()
True
>>> (reisejl_grid.bus == new_grid.bus).all().all()
True
>>> (reisejl_grid.branch == new_grid.branch).all().all()
True
>>> (reisejl_grid.dcline == new_grid.dcline).all().all()
True
>>> (reisejl_grid.gencost['before'] == new_grid.gencost['before']).all().all()
True
>>> (reisejl_grid.gencost['after'] == new_grid.gencost['after']).all().all()
True
>>> (reisejl_grid.plant == new_grid.plant).all().all()
True
>>> (reisejl_grid.sub == new_grid.sub).all().all()
True
>>> reisejl_grid.storage == new_grid.storage
True

Again, REISE.jl has saved all required data, and REISEjlMATReader knows how to interpret it, so that it matches exactly with what was created and read by REISE!

Time to review

One hour. The code itself is fairly simple, but the documentation steps are fairly involved.

[rouille]

Looks good. I like the different tests you carried out to make sure that outputs were kept the same for the refactored MATReader and new REISEMATReader and that REISEjlMATREader was consistent with REISEMATReader on the same scenario.

[danielolsen]

I wanted to be extra sure I didn't accidentally break anything. It may be worthwhile to adapt some of this test code into an __eq__ magic method for the Grid object, so that next time we can just test old_grid == new_grid, which currently evaluates False even though we have the same fields and all fields contain the same data.

[rouille]

What do you think of the naming of the classes? It is outside the scope of this PR but I feel like we can be more explicit. For instance:
MATReader --> ScenarioGrid to emphasize that this a a scenario specific grid.
REISEMATReader --> FromREISE
REISEjlMATReader --> FromREISEjl
Just a thought

[danielolsen]

@rouille I am agnostic to the naming, as long as the code works ;) I think it would be fine to address in this PR, if we decide on new names before we are ready to merge the functional code.

Note for reviewers: since the fix in #151, I have not yet repeated the integration tests performed in the original. The new Grid equality tests should be useful here.

[rouille]

What do you think @BainanXia of the current naming. Is it worth changing the name of the objects?

[BainanXia]

@rouille I like the new names you proposed. One of the reason is that the naming should focus on the high-level functionality of the code instead of including more details of the properties, which will be easier to scale in the future. For example, if we have other engines (or same engine) dealing with other type of input files (not .mat anymore), we can still call it ScenarioGrid, FromREISE, etc.

[rouille]

@danielolsen, do you mind changing the names. I guess we then also need to rename mat_reader.py as scenario_grid.py. Sorry for the extra work.

[danielolsen]

No problem, filenames have been changed and all tests still pass.

[rouille]

Awesome

[danielolsen]

Let's check again that we didn't corrupt the Grid objects as instantiated by REISEMATReader/FromREISE are not corrupted. With develop checked out:

PS C:\Users\dolsen\repos\iv\PowerSimData> python
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> from powersimdata.scenario.scenario import Scenario
>>> scenario = Scenario('419')
SCENARIO: test | WesternBase24

--> State
analyze
>>> grid = scenario.state.get_grid()
--> Loading grid
419_grid.mat not found in C:\Users\dolsen\ScenarioData\ on local machine
Transferring 419_grid.mat from server
100%|#####################################| 2.46M/2.46M [00:00<00:00, 2.21Mb/s]
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> with open('419_Grid.pkl', 'wb') as f:
...     pickle.dump(grid, f)
...
>>> scenario = Scenario('160')
SCENARIO: base | WACA_Anchor_AllMatchCA_2030_SpurUpgrade_OB_1_Storage1325

--> State
analyze
100%|#####################################| 9.38k/9.38k [00:00<00:00, 72.3kb/s]
>>> grid = scenario.state.get_grid()
--> Loading grid
160_grid.mat not found in C:\Users\dolsen\ScenarioData\ on local machine
Transferring 160_grid.mat from server
100%|#####################################| 3.20M/3.20M [00:00<00:00, 1.71Mb/s]
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> with open('160_Grid.pkl', 'wb') as f:
...     pickle.dump(grid, f)
...
>>> quit()

With reisejlmatreader checked out:

PS C:\Users\dolsen\repos\iv\PowerSimData> python
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> from powersimdata.scenario.scenario import Scenario
>>> from powersimdata.input.grid import Grid
>>> scenario = Scenario('419')
SCENARIO: test | WesternBase24

--> State
analyze
>>> new_grid = scenario.state.get_grid()
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> with open('419_Grid.pkl', 'rb') as f:
...     old_grid = pickle.load(f)
...
>>> new_grid == old_grid
True
>>> scenario = Scenario('160')
SCENARIO: base | WACA_Anchor_AllMatchCA_2030_SpurUpgrade_OB_1_Storage1325

--> State
analyze
>>> new_grid = scenario.state.get_grid()
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> with open('160_Grid.pkl', 'rb') as f:
...     old_grid = pickle.load(f)
...
>>> old_grid == new_grid
True
>>> from powersimdata.input.grid import Grid
>>> western_grid = Grid(['Western'])
Reading bus.csv
Reading plant.csv
Reading gencost.csv
Reading branch.csv
Reading dcline.csv
Reading sub.csv
Reading bus2sub.csv
Reading zone.csv
>>> # Just to be sure we can still get some False returns
>>> old_grid == western_grid
False
quit()

We can also confirm that the scenarios match between FromREISE and FromREISEjl:

>>> from powersimdata.scenario.scenario import Scenario
>>> from powersimdata.input.grid import Grid
>>> reise_grid = Scenario('419').state.get_grid()
SCENARIO: test | WesternBase24

--> State
analyze
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> reisejl_grid = Grid(interconnect=['Western'], source=r'C:\Users\dolsen\repos\iv\scenario_419\julia_output_test_extra_grid_info\input2.mat', engine='REISE.jl')
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> reisejl_grid == reise_grid
True
>>> # Now let's try Scenario 160
...
>>> reise_grid= Scenario('160').state.get_grid()
SCENARIO: base | WACA_Anchor_AllMatchCA_2030_SpurUpgrade_OB_1_Storage1325

--> State
analyze
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> reisejl_grid = Grid(interconnect=['Western'], source=r'C:\Users\dolsen\repos\iv\scenario_160\julia_output\input2.mat', engine='REISE.jl')
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> new_grid == reisejl_grid
False

Wait... False? Digging in, we find that the difference are the gencost (to be expected, since one is linearized) and the gen table of Storage, which differs for bus_id, ramp_10, and ramp_30. Ramp 10 and Ramp 30 makes sense, because they get reset by REISE's modifications to the 'generators' in the grid. bus_id is different because we have not reindexed this column for Storage:

>>> reise_grid.storage['gen'].bus_id
0     2334.0
1     2506.0
2     2709.0
3     2742.0
4     2632.0
5     4471.0
6     3959.0
7     4400.0
8     4121.0
9     4164.0
10    5672.0
11    4905.0
Name: bus_id, dtype: float64

I added re-indexing of the storage rows if they are present, and told grid equality to ignore ramp_10 and ramp_30 for storage['gen'], and now everything works the way it should:

>>> from powersimdata.scenario.scenario import Scenario
>>> from powersimdata.input.grid import Grid
>>> reise_grid = Scenario('160').state.get_grid()
SCENARIO: base | WACA_Anchor_AllMatchCA_2030_SpurUpgrade_OB_1_Storage1325

--> State
analyze
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> reisejl_grid = Grid(interconnect=['Western'], source=r'C:\Users\dolsen\repos\iv\scenario_160\julia_output\input2.mat', engine='REISE.jl')
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading sub
Loading bus2sub
>>> reise_grid == reisejl_grid
True
>>> reise_grid = Scenario('419').state.get_grid()
SCENARIO: test | WesternBase24

--> State
analyze
--> Loading grid
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> reisejl_grid = Grid(interconnect=['Western'], source=r'C:\Users\dolsen\repos\iv\scenario_419\julia_output_test_extra_grid_info\input2.mat', engine='REISE.jl')
Loading bus
Loading plant
Loading heat_rate_curve
Loading gencost_before
Loading gencost_after
Loading branch
Loading dcline
Loading sub
Loading bus2sub
>>> reise_grid == reisejl_grid
True

[rouille]

Thanks for the extra tests and debugging my code!

[BainanXia]

This is solid! @danielolsen did the test that I was about to do. Thanks.