Develop into Master by ahurli · Pull Request #242 · Breakthrough-Energy/PostREISE

ahurli · 2021-03-16T18:01:12Z

Merging Develop into Master, which will be tagged with a new release. See below for the release notes.

Closed Issues:

Implement Pipelines Phase 1 (Implement Pipelines Phase 1 #196)
LMP map hover-over tips and web compatibility (LMP map hover-over tips and web compatibility #190)
turbines map - demo and code (turbines map - demo and code #179)
Transmission "traffic map" web compatibility (Transmission "traffic map" web compatibility #178)
Renewable capacity added (aka renewables at retirements) map function (Renewable capacity added (aka renewables at retirements) map function #177)
US states goals map - update code and create demo notebook (US states goals map - update code and create demo notebook #175)
Curtailment time series by area (Curtailment time series by area #172)
Create functions that operate on indices of time series (Create functions that operate on indices of time series #164)
Create functions that check input parameters type and value (Create functions that check input parameters type and value #157)
Add emissions analysis for more pollutants than just CO2 (Add emissions analysis for more pollutants than just CO2 #156)
Factor out functions that check input parameters (Factor out functions that check input parameters #152)
Refactor map projection (Refactor map projection #150)
Rename modules and notebooks for consistency (Rename modules and notebooks for consistency #148)
Refactor PostREISE modules to use generator constants (Refactor PostREISE modules to use generator constants #146)
Enable USA plotting in AnalyzePG class (Enable USA plotting in AnalyzePG class #144)
Update import statements (Update import statements #142)
Refactor website tests [ongoing] (Refactor website tests [ongoing] #139)
Bokeh web map compatibility (Bokeh web map compatibility #131)
bokeh map changes to work with bokeh_upload_demo.ipynb (bokeh map changes to work with bokeh_upload_demo.ipynb #128)
Update init files (Update __init__ files #124)
Add linting to PostREISE (Add linting to PostREISE #123)
os package dependencies for matplotlib (os package dependencies for matplotlib #120)
LMP map improve (LMP map improve #117)
Move plots and features in the spot check integration script into codebase (Move plots and features in the spot check integration script into codebase #116)
UI for filter control bar (UI for filter control bar #112)
Eastern mapping visualization improvements (Eastern mapping visualization improvements #109)
barplot overlay on US state map visualization tool (barplot overlay on US state map visualization tool #106)
color code by us state (color code by us state #103)
improve utilization mapping (improve utilization mapping #101)
Update carbon emission maps (Update carbon emission maps #99)
[deleted] ([deleted] #93)
Reorganize analyze (Reorganize analyze #87)
AnalyzePG stack plots are fragile due to 'negative curtailment' (AnalyzePG stack plots are fragile due to 'negative curtailment' #85)
Calculate Averaged CONGU and CONGL (Calculate Averaged CONGU and CONGL #82)
Add analysis of capacity value of VRE generators (Add analysis of capacity value of VRE generators #80)
Delete Output MAT-file Following Extraction (Delete Output MAT-file Following Extraction #78)
Save MAT-file to be read by MATReader object (Save MAT-file to be read by MATReader object #75)
[deleted] ([deleted] #72)
Use Pmin and Pmax (Use Pmin and Pmax #67)
Add most recent spot check code to PostREISE (Add most recent spot check code to PostREISE #65)
User test internal MVP dashboard (User test internal MVP dashboard #61)
Finish internal MVP Dashboard (Finish internal MVP Dashboard #60)
Add Storage to Stacked Time Series (Add Storage to Stacked Time Series #38)
Extract SolveTime and Optimization time from matlab output (Extract SolveTime and Optimization time from matlab output #28)
Extract power flow profile for DC lines (Extract power flow profile for DC lines #27)
Plot Texas and Texas+Western scenarios (Plot Texas and Texas+Western scenarios #25)
fix: double-extracting congestion lower, not extracting upper (fix: double-extracting congestion lower, not extracting upper #23)
Linux specific call to server. (Linux specific call to server. #19)
Investigate branches that have 0 powerflow in Western Base Case (Investigate branches that have 0 powerflow in Western Base Case #15)
WECC Congestion Analysis - General Function Development (WECC Congestion Analysis - General Function Development #13)
Transfer Data (Transfer Data #10)
check why hydro is curtailed, according to Ben's generation and demand plots (check why hydro is curtailed, according to Ben's generation and demand plots #7)
Check why hydro power is curtailed according to Ben's generation and demand plots (Check why hydro power is curtailed according to Ben's generation and demand plots #6)
Automatic Transmission-line Congestion Analysis (Automatic Transmission-line Congestion Analysis #5)
Don't exit python but rather return or throw an exception. (Don't exit python but rather return or throw an exception. #4)
Create proper timer in extract (Create proper timer in extract #1)

Merged pull requests (features):

Implement shortfall bar chart (Implement shortfall bar chart #240) (@BainanXia)
Improve flexibility of state line plotting function (Improve flexibility of state line plotting function #234) (@danielolsen)
feat: abstract grid model (feat: abstract grid model #233) (@rouille)
Implement pie chart generation vs capacity (Implement pie chart generation vs capacity #231) (@BainanXia)
Implement bar charts for generation vs capacity (Implement bar charts for generation vs capacity #215) (@BainanXia)
feat: add option to create bokeh figures for the website (feat: add option to create bokeh figures for the website #203) (@merrielle)
Add transmission upgrade comparison plots (Add transmission upgrade comparison plots #202) (@danielolsen)
Factor out generation and curtailment plots (Factor out generation and curtailment plots #201) (@BainanXia)
Improve flexibility of map plots (Improve flexibility of map plots #199) (@danielolsen)
LMP Map: hover tips and web compatibility for mapping visualization (LMP Map: hover tips and web compatibility for mapping visualization #191) (@victoriahunt)
Transmission utilization, risk and binding incidents maps improvement (Transmission utilization, risk and binding incidents maps improvement #189) (@victoriahunt)
Renewable capacity map (Renewable capacity map #188) (@victoriahunt)
State goals web map (State goals web map #176) (@victoriahunt)
Carbon map web legend (Carbon map web legend #174) (@victoriahunt)
Aggregate curtailment time series by area (Aggregate curtailment time series by area #173) (@rouille)
Interconnections map b2b (Interconnections map b2b #170) (@victoriahunt)
Add calculation of NOx and SO2 emissions (Add calculation of NOx and SO2 emissions #168) (@danielolsen)
Add 2016 historical capacity and generation data (Add 2016 historical capacity and generation data #167) (@rouille)
Create functions that operate on indices of time series (Create functions that operate on indices of time series #165) (@rouille)
Write check and helper functions (Write check and helper functions #158) (@rouille)
Factor out functions that check input parameters (Factor out functions that check input parameters #153) (@rouille)
Enable USA plotting in the AnalyzePG class (Enable USA plotting in the AnalyzePG class #145) (@rouille)
Pr template (Pr template #141) (@kamilahjae)
Add hover tool tips to Interconnections Map (Add hover tool tips to Interconnections Map #140) (@victoriahunt)
feat: add interconnection map and improve lmp map performance (feat: add interconnection map and improve lmp map performance #136) (@merrielle)
Map improvement (Map improvement #130) (@victoriahunt)
Integrate part of spot check - simulated and historical data prep and plot by state (Integrate part of spot check - simulated and historical data prep and plot by state #129) (@jon-hagg)
feat: analyze binding constraints (ramp, pmin, pmax) (feat: analyze binding constraints (ramp, pmin, pmax) #121) (@danielolsen)
Map average locational marginal price (Map average locational marginal price #119) (@victoriahunt)
When calculating MW-miles of transmission upgrades, exclude some branches (When calculating MW-miles of transmission upgrades, exclude some branches #118) (@danielolsen)
Calculate capacity value (Calculate capacity value #114) (@danielolsen)
Plot energy & carbon stacks (Plot energy & carbon stacks #113) (@danielolsen)
feat: bar plots on US states map per us state for emissions (feat: bar plots on US states map per us state for emissions #108) (@victoriahunt)
feat: add requirements_freeze.txt to lock down package dependencies (feat: add requirements_freeze.txt to lock down package dependencies #107) (@dmuldrew)
Color by us state (Color by us state #104) (@victoriahunt)
Add state borders (Add state borders #98) (@victoriahunt)
Calculate averaged congestion and save file on server (Calculate averaged congestion and save file on server #92) (@rouille)
Calculate averaged congestion and save output file on server (Calculate averaged congestion and save output file on server #89) (@rouille)
feat: sum generation by type and zone (feat: sum generation by type and zone #86) (@danielolsen)
feat: add info attribute to MockScenario (feat: add info attribute to MockScenario #83) (@BainanXia)
Delete Output MAT-files (Delete Output MAT-files #79) (@rouille)
Save MAT-file to be read by MATReader object (Save MAT-file to be read by MATReader object #76) (@rouille)
feat: add name attribute to MockAnalyze (feat: add name attribute to MockAnalyze #74) (@BainanXia)
feat: more functionality for MockAnalyze objects (feat: more functionality for MockAnalyze objects #71) (@danielolsen)
feat: add plot tornado function and state mappers from spot check (feat: add plot tornado function and state mappers from spot check #66) (@merrielle)
feat: analyze curtailment (feat: analyze curtailment #57) (@danielolsen)
feat: plant parsing helpers (feat: plant parsing helpers #56) (@danielolsen)
feat: support for 'biomass' and 'other' generators, slight refactor (feat: support for 'biomass' and 'other' generators, slight refactor #55) (@danielolsen)
feat: add historical data and NREL data to codebase to aid with plotting. Add a function to facilitate the creation of custom data. (feat: add historical data and NREL data to codebase to aid with plotting. Add a function to facilitate the creation of custom data. #54) (@merrielle)
feat: add ability to select custom dates when plotting bar, pie, and shortfall graphs (feat: add ability to select custom dates when plotting bar, pie, and shortfall graphs #53) (@merrielle)
feat: add code to plot shadow price data (feat: add code to plot shadow price data #52) (@merrielle)
feat: calculate congestion surplus (feat: calculate congestion surplus #50) (@danielolsen)
Calculate MW-miles of upgraded lines (Calculate MW-miles of upgraded lines #48) (@rouille)
Carbon emission calculation (Carbon emission calculation #47) (@rouille)
Add wecc congest (Add wecc congest #45) (@rouille)
Integrate spot check code for plotting multiple scenarios (Integrate spot check code for plotting multiple scenarios #44) (@merrielle)
Carbon plotting (Carbon plotting #43) (@victoriahunt)
feat: parse change_table entries into interstate/intrastate (feat: parse change_table entries into interstate/intrastate #42) (@danielolsen)
feat: calculate MW-miles for transmission-expanded change tables (feat: calculate MW-miles for transmission-expanded change tables #41) (@danielolsen)
Changes to AnalyzePG (Changes to AnalyzePG #40) (@rouille)
Carbon analysis with tests and generic mock objects (Carbon analysis with tests and generic mock objects #39) (@ghost)
feat: add extraction of storage variable results (feat: add extraction of storage variable results #37) (@ghost)
feat: Add HOME_DIR (feat: Add HOME_DIR #35) (@rouille)
Extract log (Extract log #34) (@rouille)
Revert "feat: Add HOME_DIR" (Revert "feat: Add HOME_DIR" #33) (@rouille)
Add net demand to stacked generation plot (Add net demand to stacked generation plot #32) (@rouille)
feat: Add HOME_DIR (feat: Add HOME_DIR #31) (@rouille)
Extract log (Extract log #30) (@rouille)
Extract power flow on HVDC line (Extract power flow on HVDC line #29) (@rouille)
Enable Texas plotting in Analyzepg class (Enable Texas plotting in Analyzepg class #26) (@rouille)
feat: wecc congestion stats (feat: wecc congestion stats #20) (@victoriahunt)
feat: parse results including geothermal and dfo gens (feat: parse results including geothermal and dfo gens #18) (@ghost)
feat: extract duals for LMP and congestion from matpower to pkl (feat: extract duals for LMP and congestion from matpower to pkl #16) (@ghost)
Transfer for scenario object (Transfer for scenario object #14) (@kasparm)
Add analysis and plotting routines (Add analysis and plotting routines #9) (@rouille)
Output data transfer (Output data transfer #8) (@kasparm)
Transmission congestion analysis (Transmission congestion analysis #3) (@mlamherr)
Speedup read (Speedup read #2) (@kasparm)

Merged pull requests (fixes, etc.):

fix: typo in curtailment summarization (fix: typo in curtailment summarization #232) (@danielolsen)
Update dependencies (Update dependencies #230) (@jon-hagg)
Better formatting for auto generated PR description (Better formatting for auto generated PR description #229) (@jon-hagg)
ci: update some packages, and add workflow to automate package updates (ci: update some packages, and add workflow to automate package updates #218) (@jon-hagg)
ci: use custom action to trigger docs build (ci: use custom action to trigger docs build #216) (@jon-hagg)
Update link in PR template (Update link in PR template #214) (@rouille)
chore: update package version number (chore: update package version number #213) (@kasparm)
docs: add link to installation and contribution guide (docs: add link to installation and contribution guide #212) (@kasparm)
ci: remove obsolete token (ci: remove obsolete token #211) (@jon-hagg)
Update org info (Update org info #210) (@jon-hagg)
ci: use built in black profile for isort (ci: use built in black profile for isort #209) (@jon-hagg)
refactor: remove redundant map_demand_to_buses (refactor: remove redundant map_demand_to_buses #208) (@danielolsen)
refactor: remove seaborn dependency for colors (refactor: remove seaborn dependency for colors #207) (@danielolsen)
Improve shadow price plotting module (Improve shadow price plotting module #206) (@rouille)
fix: regenerate lock file to include requests (fix: regenerate lock file to include requests #205) (@jon-hagg)
Fix docstrings (Fix docstrings #204) (@rouille)
No longer clip time series resampled by mean in time.py (No longer clip time series resampled by mean in time.py #198) (@BainanXia)
Add license (Add license #197) (@rouille)
Create data intake template (Create data intake template #195) (@rouille)
chore: update matplotlib and numpy (chore: update matplotlib and numpy #194) (@jon-hagg)
Add flake8 workflow and fix warnigns (Add flake8 workflow and fix warnigns #193) (@jon-hagg)
Add terms and conditions to ATTRIBUTION file (Add terms and conditions to ATTRIBUTION file #192) (@rouille)
Notebooks (Notebooks #187) (@rouille)
Remove zipp (unused package) (Remove zipp (unused package) #185) (@jon-hagg)
fix: wrong var name for mapper in map_risk_bind (fix: wrong var name for mapper in map_risk_bind #184) (@merrielle)
Regenerate lock file (Regenerate lock file #183) (@jon-hagg)
fix: typo in tox.ini (fix: typo in tox.ini #182) (@jon-hagg)
Format docstring (Format docstring #181) (@rouille)
CI enhancements (CI enhancements #171) (@jon-hagg)
List of available resources (List of available resources #169) (@rouille)
refactor: expect datetime indices in tests with mock scenarios (refactor: expect datetime indices in tests with mock scenarios #166) (@danielolsen)
refactor: remove reference to id2type (refactor: remove reference to id2type #163) (@danielolsen)
refactor: move design analyses out of PostREISE (refactor: move design analyses out of PostREISE #162) (@danielolsen)
Update pandas (Update pandas #161) (@jon-hagg)
Fix some warnings from sphinx and remove large states data objects from docs (Fix some warnings from sphinx and remove large states data objects from docs #160) (@jon-hagg)
Fix rsa error due to paramiko (Fix rsa error due to paramiko #159) (@jon-hagg)
refactor pyproj calls for v2.x compatibility (refactor pyproj calls for v2.x compatibility #154) (@danielolsen)
Formatting changes (Formatting changes #151) (@jon-hagg)
Rename modules and notebooks (Rename modules and notebooks #149) (@rouille)
Use module enclosing generators constants (Use module enclosing generators constants #147) (@rouille)
Update import statements (Update import statements #143) (@rouille)
fix: remove unused broken import (fix: remove unused broken import #138) (@jon-hagg)
Summarize curtailment fix (Summarize curtailment fix #135) (@danielolsen)
Add pytest to ci and setup pipenv (Add pytest to ci and setup pipenv #134) (@jon-hagg)
fix: allow small tolerance for negative pg/carbon data (fix: allow small tolerance for negative pg/carbon data #133) (@danielolsen)
fix: typo in module name (fix: typo in module name #127) (@jon-hagg)
Update packages to latest, with couple exceptions (Update packages to latest, with couple exceptions #126) (@jon-hagg)
Update init files (Update __init__ files #125) (@rouille)
Format everything and add github action to validate (Format everything and add github action to validate #122) (@jon-hagg)
Calculate wind_offshore curtailment (Calculate wind_offshore curtailment #115) (@danielolsen)
Eastern visuals improve (Eastern visuals improve #111) (@victoriahunt)
fix: project negative curtailments to zero (fix: project negative curtailments to zero #110) (@BainanXia)
Move modules (Move modules #105) (@rouille)
refactor/doc: make curtailment more robust, update README (refactor/doc: make curtailment more robust, update README #100) (@danielolsen)
Move modules (Move modules #97) (@rouille)
refactor: save congu and congl as sparse DataFrames (refactor: save congu and congl as sparse DataFrames #96) (@danielolsen)
refactor: avoid append to speed up extraction (refactor: avoid append to speed up extraction #90) (@danielolsen)
Reorganize analyze (Reorganize analyze #88) (@rouille)
fix: ensure that MockAnalyze returns a dict for ct (fix: ensure that MockAnalyze returns a dict for ct #84) (@danielolsen)
doc: updating README to include more of our analysis functions (doc: updating README to include more of our analysis functions #77) (@danielolsen)
test: add hydro into mock scenario (test: add hydro into mock scenario #73) (@BainanXia)
Use Pmax inplace of GenMWMax (Use Pmax inplace of GenMWMax #69) (@rouille)
test: fix shadowprice tests (test: fix shadowprice tests #68) (@merrielle)
chore: add pycharm and os files to gitignore (chore: add pycharm and os files to gitignore #59) (@merrielle)
chore: update pandas version in requirements (chore: update pandas version in requirements #51) (@merrielle)
chore: add gitignore file (chore: add gitignore file #49) (@merrielle)
fix: Set correctly the columns name in data frame (fix: Set correctly the columns name in data frame #36) (@rouille)
fix: double-extracting congestion lower, not extracting upper (fix: double-extracting congestion lower, not extracting upper #24) (@ghost)
Windows compatible (Windows compatible #22) (@rouille)

feat: bar plots on US states map per us state, for emissions, issue PostREISE #106

…107)

Negative curtailments caused by rounding issues will break the stack plot.

fix: order operations for speed Take abs value after calculating medians in median_calc chore: rerun notebook to suppress warnings

Eastern visuals improve

Calculate capacity value

Plot energy & carbon stacks

Calculate curtailment for wind_offshore generators

feat: demo notebook added

Format everything and add github action to validate

#218)

…summarize.py

…n_by_type_zone

multi refactor: implement bar charts for generation vs capacity

…acity

multi refactor: implement pie chart generation vs capacity

…elines refactor: improve flexibility of state line plotting function

Abstract grid model

…ll_chart multi refactor: implement shortfall bar chart

rouille

Thanks!

victoriahunt and others added 30 commits April 27, 2020 14:12

Merge pull request #108 from intvenlab/barplots_on_usstates

47ebebd

feat: bar plots on US states map per us state, for emissions, issue PostREISE #106

feat: add requirements_freeze.txt to lock down package dependencies (#…

284b282

…107)

fix: project negative curtailments to zero (#110)

9410b64

Negative curtailments caused by rounding issues will break the stack plot.

fix: speed up code by filtering

48a0ee4

fix: order operations for speed Take abs value after calculating medians in median_calc chore: rerun notebook to suppress warnings

Merge pull request #111 from intvenlab/eastern_visuals_improve

b162329

Eastern visuals improve

feat: add capacity value calculation

91704ac

test: test capacity value calculations

864736d

chore: rename calculate_capacity_value to reflect method (NLDC)

c2f3747

refactor: move input checking to separate function

93e2dff

feat: add 'net load peak' calculation of capacity value

fa4d9be

test: add tests for calculate_net_load_peak

09a1891

Merge pull request #114 from intvenlab/capacity_value

c8f1a9e

Calculate capacity value

feat: plot single scenario energy/carbon stacked bars

12e4063

feat: add energy/carbon stacks for two scenarios

7740ec1

refactor: instead of plot 1 or plot 2, plot n

67ab3fd

doc: point to demo notebook within module

3e9cfe6

doc: add demo notebook for energy carbon stack

458f598

Merge pull request #113 from intvenlab/plot_energy_carbon_stack

6755517

Plot energy & carbon stacks

refactor: enable curtailment lookup for wind_offshore

c98ae59

test: change curtailment tests to include wind_offshore

fdde9ed

fix: test all curtailment columns

f34573f

Merge pull request #115 from intvenlab/calculate_offshore_curtailment

6ebc800

Calculate curtailment for wind_offshore generators

Create plot_lmp.py

117b4d5

feat: demo notebook added

feat: analyze binding constraints (ramp, pmin, pmax) (#121)

b6a2023

style: run black on the codebase

ce8d2bb

chore: add ignore file for mass style changes

41f97b3

Merge pull request #122 from intvenlab/jon/black

d57d9fa

Format everything and add github action to validate

chore: update packages (#126)

cdf46b4

chore: update __init__ files

b31b895

fix: typo in module name (#127)

fa2eb66

jenhagg and others added 27 commits February 24, 2021 14:03

ci: update some packages, and add workflow to automate package updates (

cbb7670

#218)

chore: clarify functionalities of functions in capacity_value.py and …

a90223c

…summarize.py

feat: add function that summarizes capacity table for a scenario

e82f89b

feat: add optinal paramters time_range and time_zone in sum_generatio…

6e3a975

…n_by_type_zone

feat: add plot function to generate bar charts

556cbd5

test: add tests for plot_bar_generation_vs_capacity

d973ce5

feat: add demo notebook for plot_bar_generation_vs_capacity

33720dd

Merge pull request #215 from Breakthrough-Energy/bainan/multi_bar_chart

35e46ca

multi refactor: implement bar charts for generation vs capacity

ci: nicer looking PR description (#229)

d03b431

Update dependencies (#230)

1797068

feat: add plot_pie_generation_vs_capacity

60bff9e

feat: add demo notebook for plot_pie_generation_vs_capacity

379d2d8

docs: update doc strings in the plot function and demo notebook

3462250

test: write tests for input validations of plot_pie_generation_vs_cap…

b8f018d

…acity

Merge pull request #231 from Breakthrough-Energy/bainan/multi_pie_chart

c048a81

multi refactor: implement pie chart generation vs capacity

fix: typo in curtailment summarization (#232)

7a2e391

feat: allow states to be plotted on existing bokeh figure

af3c48d

feat: add more default values for state line plotting

17473b2

refactor: re-run state goals demo using new call signature

0b32518

Merge pull request #234 from Breakthrough-Energy/daniel/flexible_stat…

55316e7

…elines refactor: improve flexibility of state line plotting function

refactor: abstract grid model (#233)

c3e83e5

Abstract grid model

feat: add_plot_bar_shortfall

f5b1f1b

test: write tests for input validations of plot_bar_shortfall

978947c

feat: add demo notebook for plot_bar_shortfall

69331d3

data: add target files to support plot_bar_shortfall demo

016eed0

docs: update ATTRIBUTION file

093080c

Merge pull request #240 from Breakthrough-Energy/bainan/multi_shortfa…

4fea93c

…ll_chart multi refactor: implement shortfall bar chart

ahurli requested a review from rouille March 16, 2021 18:01

rouille approved these changes Mar 16, 2021

View reviewed changes

ahurli merged commit 3215e93 into master Mar 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Develop into Master#242

Develop into Master#242
ahurli merged 384 commits intomasterfrom
develop

ahurli commented Mar 16, 2021 •

edited

Loading

Uh oh!

rouille left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

ahurli commented Mar 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rouille left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

ahurli commented Mar 16, 2021 •

edited

Loading