Reorganize analyze

[rouille]

Purpose

Seperate generation based and transmission based analysis module.

What is the code doing

No new code. Folders have been created, modules have been moved into new folders. some modules have been renamed and import statements have been updated accordingly. Also, some functions have been moved (the haversine function in distance and the sum_generation_by_type_zone in summarize)

Where to look

The folder structure has been discussed with @danielolsen. Running the test is a good way to make sure that all the modules have been correctly relocated. To run the test, you will need to checkout the fix_haversine_import branch in PowerSimData (dependent PR)

Here are the results of the tests on my machine:

================================================================ test session starts =================================================================
platform darwin -- Python 3.6.5, pytest-4.5.0, py-1.8.0, pluggy-0.13.0
rootdir: /Users/brdo/REM/PostREISE
collected 90 items                                                                                                                                   

analyze/generation/tests/test_carbon.py ....                                                                                                   [  4%]
analyze/generation/tests/test_curtailment.py ............                                                                                      [ 17%]
analyze/generation/tests/test_summarize.py ...                                                                                                 [ 21%]
analyze/tests/test_helpers.py ....                                                                                                             [ 25%]
analyze/transmission/tests/test_congestion_surplus.py ...                                                                                      [ 28%]
analyze/transmission/tests/test_mwmiles.py ....                                                                                                [ 33%]
analyze/transmission/tests/test_statelines.py .....                                                                                            [ 38%]
plot/multi/tests/test_plot_bar.py ..........                                                                                                   [ 50%]
plot/multi/tests/test_plot_helpers.py ...............                                                                                          [ 66%]
plot/multi/tests/test_plot_pie.py ....                                                                                                         [ 71%]
plot/multi/tests/test_plot_shortfall.py .........                                                                                              [ 81%]
plot/tests/test_plot_shadowprice.py .........                                                                                                  [ 91%]
process/test/test_transfer.py ..                                                                                                               [ 93%]
tests/test_mocks.py ......                                                                                                                     [100%]

============================================================= 90 passed in 3.43 seconds ==============================================================

Time estimate

It should be super quick. I will do an interactive rebase after all eventual comments are addressed.

[danielolsen]

I've confirmed that all tests work on my machine as well. I will approve after the interactive rebase.

python -m pytest
============================= test session starts =============================
platform win32 -- Python 3.7.4, pytest-5.3.1, py-1.8.0, pluggy-0.13.1
rootdir: C:\Users\dolsen\repos\iv\postreise
collected 90 items

postreise\analyze\generation\tests\test_carbon.py ....                   [  4%]
postreise\analyze\generation\tests\test_curtailment.py ............      [ 17%]
postreise\analyze\generation\tests\test_summarize.py ...                 [ 21%]
postreise\analyze\tests\test_helpers.py ....                             [ 25%]
postreise\analyze\transmission\tests\test_congestion_surplus.py ...      [ 28%]
postreise\analyze\transmission\tests\test_mwmiles.py ....                [ 33%]
postreise\analyze\transmission\tests\test_statelines.py .....            [ 38%]
postreise\plot\multi\tests\test_plot_bar.py ..........                   [ 50%]
postreise\plot\multi\tests\test_plot_helpers.py ...............          [ 66%]
postreise\plot\multi\tests\test_plot_pie.py ....                         [ 71%]
postreise\plot\multi\tests\test_plot_shortfall.py .........              [ 81%]
postreise\plot\tests\test_plot_shadowprice.py .........                  [ 91%]
postreise\process\test\test_transfer.py ..                               [ 93%]
postreise\tests\test_mocks.py ......                                     [100%]

============================== warnings summary ===============================
C:\Python37\lib\site-packages\win32\lib\pywintypes.py:2
  C:\Python37\lib\site-packages\win32\lib\pywintypes.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp, sys, os

C:\Python37\lib\site-packages\jinja2\utils.py:485
  C:\Python37\lib\site-packages\jinja2\utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping

C:\Python37\lib\site-packages\jinja2\runtime.py:318
  C:\Python37\lib\site-packages\jinja2\runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======================= 90 passed, 3 warnings in 4.39s ========================

[rouille]

One thing I did not do is rename the generate_cong_stats function to generate_utilization_stats in the utilization module to kind of following the naming of the module. What do you guys thin?. @victoriahunt wrote this function.

[rouille]

interactive rebase done. Tests are still all passing:

================================================================ test session starts =================================================================
platform darwin -- Python 3.6.5, pytest-4.5.0, py-1.8.0, pluggy-0.13.0 -- /Users/brdo/PyEnv/REM/v1/bin/python
cachedir: .pytest_cache
rootdir: /Users/brdo/REM/PostREISE
collected 90 items                                                                                                                                   

postreise/analyze/generation/tests/test_carbon.py::TestCarbonCalculation::test_carbon_calc_always_on PASSED                                    [  1%]
postreise/analyze/generation/tests/test_carbon.py::TestCarbonCalculation::test_carbon_calc_decommit PASSED                                     [  2%]
postreise/analyze/generation/tests/test_carbon.py::TestCarbonCalculation::test_carbon_calc_simple PASSED                                       [  3%]
postreise/analyze/generation/tests/test_carbon.py::TestCarbonSummarization::test_carbon_summarization PASSED                                   [  4%]
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentTimeSeries::test_calculate_curtailment_time_series_default PASSED [  5%]
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentTimeSeries::test_calculate_curtailment_time_series_solar PASSED [  6%]
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentTimeSeries::test_calculate_curtailment_time_series_solar_wind PASSED [  7%]
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentTimeSeries::test_calculate_curtailment_time_series_wind PASSED [  8%]
postreise/analyze/generation/tests/test_curtailment.py::TestCheckResourceInScenario::test_error_geothermal_curtailment PASSED                  [ 10%]
postreise/analyze/generation/tests/test_curtailment.py::TestCheckResourceInScenario::test_error_no_solar PASSED                                [ 11%]
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentPercentage::test_calculate_curtailment_percentage_default PASSED [ 12%]
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentPercentage::test_calculate_curtailment_percentage_solar PASSED [ 13%]
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentPercentage::test_calculate_curtailment_percentage_solar_wind PASSED [ 14%]
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentPercentage::test_calculate_curtailment_percentage_wind PASSED  [ 15%]
postreise/analyze/generation/tests/test_curtailment.py::TestSummarizeCurtailmentByBus::test_summarize_curtailment_by_bus PASSED                [ 16%]
postreise/analyze/generation/tests/test_curtailment.py::TestSummarizeCurtailmentByLocation::test_summarize_curtailment_by_location PASSED      [ 17%]
postreise/analyze/generation/tests/test_summarize.py::TestSumGenerationByTypeZone::test_sum_generation PASSED                                  [ 18%]
postreise/analyze/generation/tests/test_summarize.py::TestSumGenerationByTypeZone::test_with_scenario_not_analyze PASSED                       [ 20%]
postreise/analyze/generation/tests/test_summarize.py::TestSumGenerationByTypeZone::test_with_string PASSED                                     [ 21%]
postreise/analyze/tests/test_helpers.py::TestSummarizePlantToBus::test_summarize_all_buses_false PASSED                                        [ 22%]
postreise/analyze/tests/test_helpers.py::TestSummarizePlantToBus::test_summarize_all_buses_true PASSED                                         [ 23%]
postreise/analyze/tests/test_helpers.py::TestSummarizePlantToBus::test_summarize_default PASSED                                                [ 24%]
postreise/analyze/tests/test_helpers.py::TestSummarizePlantToLocation::test_summarize_location PASSED                                          [ 25%]
postreise/analyze/transmission/tests/test_congestion_surplus.py::TestCalculateCongestionSurplus::test_calculate_congestion_surplus_single_time PASSED [ 26%]
postreise/analyze/transmission/tests/test_congestion_surplus.py::TestCalculateCongestionSurplus::test_calculate_congestion_surplus_three_times PASSED [ 27%]
postreise/analyze/transmission/tests/test_congestion_surplus.py::TestMappingHelpers::test_map_demand_to_buses PASSED                           [ 28%]
postreise/analyze/transmission/tests/test_mwmiles.py::TestCalculateMWMiles::test_calculate_mw_miles_many_scaled PASSED                         [ 30%]
postreise/analyze/transmission/tests/test_mwmiles.py::TestCalculateMWMiles::test_calculate_mw_miles_no_scale PASSED                            [ 31%]
postreise/analyze/transmission/tests/test_mwmiles.py::TestCalculateMWMiles::test_calculate_mw_miles_one_line_scaled PASSED                     [ 32%]
postreise/analyze/transmission/tests/test_mwmiles.py::TestCalculateMWMiles::test_calculate_mw_miles_one_transformer_scaled PASSED              [ 33%]
postreise/analyze/transmission/tests/test_statelines.py::TestClassifyInterstateIntrastate::test_classify_interstate_intrastate_bad_ct PASSED   [ 34%]
postreise/analyze/transmission/tests/test_statelines.py::TestClassifyInterstateIntrastate::test_classify_interstate_intrastate_empty_ct PASSED [ 35%]
postreise/analyze/transmission/tests/test_statelines.py::TestClassifyInterstateIntrastate::test_classify_interstate_intrastate_none PASSED     [ 36%]
postreise/analyze/transmission/tests/test_statelines.py::TestClassifyInterstateIntrastate::test_classify_interstate_intrastate_several PASSED  [ 37%]
postreise/analyze/transmission/tests/test_statelines.py::TestClassifyInterstateIntrastate::test_classify_interstate_intrastate_two PASSED      [ 38%]
postreise/plot/multi/tests/test_plot_bar.py::test_construct_bar_ax_data_fills_in_missing_resources_with_0 PASSED                               [ 40%]
postreise/plot/multi/tests/test_plot_bar.py::test_construct_bar_ax_data PASSED                                                                 [ 41%]
postreise/plot/multi/tests/test_plot_bar.py::test_construct_bar_ax_data_with_user_set_resource_types PASSED                                    [ 42%]
postreise/plot/multi/tests/test_plot_bar.py::test_get_bar_resource_types PASSED                                                                [ 43%]
postreise/plot/multi/tests/test_plot_bar.py::test_get_bar_resource_types_does_not_have_duplicates PASSED                                       [ 44%]
postreise/plot/multi/tests/test_plot_bar.py::test_get_bar_resource_types_includes_resource_types_only_present_in_some_scenarios PASSED         [ 45%]
postreise/plot/multi/tests/test_plot_bar.py::test_get_bar_resource_types_handles_resource_types_not_found_in_known_resource_types PASSED       [ 46%]
postreise/plot/multi/tests/test_plot_bar.py::test_get_bar_display_val_greater_than_ten PASSED                                                  [ 47%]
postreise/plot/multi/tests/test_plot_bar.py::test_get_bar_display_val_less_than_ten PASSED                                                     [ 48%]
postreise/plot/multi/tests/test_plot_bar.py::test_get_bar_display_val_zero PASSED                                                              [ 50%]
postreise/plot/multi/tests/test_plot_helpers.py::test_handle_plot_inputs PASSED                                                                [ 51%]
postreise/plot/multi/tests/test_plot_helpers.py::test_handle_plot_inputs_throws_valueError_for_incorrect_interconnect PASSED                   [ 52%]
postreise/plot/multi/tests/test_plot_helpers.py::test_handle_plot_inputs_throws_valueError_when_wrong_number_of_scenario_names PASSED          [ 53%]
postreise/plot/multi/tests/test_plot_helpers.py::test_handle_plot_inputs_throws_valueError_if_not_enough_scenario_ids_or_custom_data PASSED    [ 54%]
postreise/plot/multi/tests/test_plot_helpers.py::test_handle_shortfall_inputs_where_is_match_CA_is_false PASSED                                [ 55%]
postreise/plot/multi/tests/test_plot_helpers.py::test_handle_shortfall_inputs_where_is_match_CA_is_true PASSED                                 [ 56%]
postreise/plot/multi/tests/test_plot_helpers.py::test_handle_shortfall_inputs_with_baseline_target_and_demand_params PASSED                    [ 57%]
postreise/plot/multi/tests/test_plot_helpers.py::test_format_scenario_data PASSED                                                              [ 58%]
postreise/plot/multi/tests/test_plot_helpers.py::test_unit_conversion_increase_by_2 PASSED                                                     [ 60%]
postreise/plot/multi/tests/test_plot_helpers.py::test_unit_conversion_decrease_by_1 PASSED                                                     [ 61%]
postreise/plot/multi/tests/test_plot_helpers.py::test_make_empty_data_with_western PASSED                                                      [ 62%]
postreise/plot/multi/tests/test_plot_helpers.py::test_make_empty_data_with_texas PASSED                                                        [ 63%]
postreise/plot/multi/tests/test_plot_helpers.py::test_make_gen_cap_custom_data_given_no_data PASSED                                            [ 64%]
postreise/plot/multi/tests/test_plot_helpers.py::test_make_gen_cap_custom_data_given_cap_data PASSED                                           [ 65%]
postreise/plot/multi/tests/test_plot_helpers.py::test_make_gen_cap_custom_data_given_gen_and_cap_data PASSED                                   [ 66%]
postreise/plot/multi/tests/test_plot_pie.py::test_construct_pie_ax_data PASSED                                                                 [ 67%]
postreise/plot/multi/tests/test_plot_pie.py::test_roll_up_small_pie_wedges PASSED                                                              [ 68%]
postreise/plot/multi/tests/test_plot_pie.py::test_roll_up_small_pie_wedges_with_zero_values PASSED                                             [ 70%]
postreise/plot/multi/tests/test_plot_pie.py::test_roll_up_small_pie_wedges_with_other_category PASSED                                          [ 71%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_construct_shortfall_ax_data PASSED                                                     [ 72%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_construct_shortfall_ax_data_when_shortfall_is_zero PASSED                              [ 73%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_construct_shortfall_ax_data_when_target_is_zero PASSED                                 [ 74%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_construct_shortfall_data_for_western_with_independent_scenario PASSED                  [ 75%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_construct_shortfall_data_for_western_with_collaborative_scenarios PASSED               [ 76%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_get_total_generated_renewables PASSED                                                  [ 77%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_get_total_generated_renewables_for_CA PASSED                                           [ 78%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_get_total_generated_renewables_for_WA PASSED                                           [ 80%]
postreise/plot/multi/tests/test_plot_shortfall.py::test_get_total_generated_renewables_for_WA_with_is_match_CA PASSED                          [ 81%]
postreise/plot/tests/test_plot_shadowprice.py::test_construct_bus_data PASSED                                                                  [ 82%]
postreise/plot/tests/test_plot_shadowprice.py::test_get_lmp_split_points_with_min_lmp_below_neg_one PASSED                                     [ 83%]
postreise/plot/tests/test_plot_shadowprice.py::test_get_lmp_split_points_with_min_between_neg_one_and_one PASSED                               [ 84%]
postreise/plot/tests/test_plot_shadowprice.py::test_get_lmp_split_points_with_min_lmp_more_than_one PASSED                                     [ 85%]
postreise/plot/tests/test_plot_shadowprice.py::test_construct_branch_data PASSED                                                               [ 86%]
postreise/plot/tests/test_plot_shadowprice.py::test_get_bus_legend_bars_and_labels PASSED                                                      [ 87%]
postreise/plot/tests/test_plot_shadowprice.py::test_get_bus_legend_bars_and_labels_with_rounding PASSED                                        [ 88%]
postreise/plot/tests/test_plot_shadowprice.py::test_get_bus_legend_bars_and_labels_clamps_large_numbers_on_end PASSED                          [ 90%]
postreise/plot/tests/test_plot_shadowprice.py::test_get_bus_legend_bars_and_labels_min_bar_len_is_one PASSED                                   [ 91%]
postreise/process/test/test_transfer.py::test_setup_server_connection PASSED                                                                   [ 92%]
postreise/process/test/test_transfer.py::test_get_scenario_file_from_server PASSED                                                             [ 93%]
postreise/tests/test_mocks.py::TestMocks::test_mock_grid_failures PASSED                                                                       [ 94%]
postreise/tests/test_mocks.py::TestMocks::test_mock_grid_successes PASSED                                                                      [ 95%]
postreise/tests/test_mocks.py::TestMocks::test_mock_hydro_stored_properly PASSED                                                               [ 96%]
postreise/tests/test_mocks.py::TestMocks::test_mock_pg_stored_properly PASSED                                                                  [ 97%]
postreise/tests/test_mocks.py::TestMocks::test_mock_solar_stored_properly PASSED                                                               [ 98%]
postreise/tests/test_mocks.py::TestMocks::test_mock_wind_stored_properly PASSED                                                                [100%]

============================================================= 90 passed in 5.23 seconds ==============================================================

[danielolsen]

Tests pass for me too. Thanks for taking the initiative on this!