Reorganize root of repo by patrickbrown4 · Pull Request #36 · ReEDS-Model/ReEDS

patrickbrown4 · 2026-04-19T20:06:49Z

Summary

Addresses issues 1713 / 2017 (on the old repo) by reorganizing the root of the repo.

Feedback welcome!

Technical details

Proposed structure

The general idea is to move scripts/functions that are used during the run to subfolders of reeds/:

The GAMS scripts move from the root of the repo to reeds/core
The input_processing scripts move to reeds/input_processing
Job-submission scripts go to reeds/hpc
Solver files go to reeds/solver
ReEDS_Augur goes to reeds/resource_adequacy
- reeds2pras goes to reeds/resource_adequacy/reeds2pras
- Mentions of "Augur" are changed to "resource adequacy calculations"
- The handoff data between ReEDS and PRAS are now in {casename}/handoff
Auto-generated code (b_sets.gms, report_params.gms) goes to {casename}/autocode
Root-level scripts mostly go to helpers

OLD                                  →    NEW
---------------------------------    →    --------------------------------------------
├── input_processing                 →    ├── helpers
│   ├── __init__.py                  →    │   ├── interim_report_batch.py
│   ├── aggregate_regions.py         →    │   ├── interim_report.py
│   ├── calc_financial_inputs.py     →    │   ├── restart_runs.py
│   ├── check_inputs.py              →    │   └── runstatus.py
│   ├── climateprep.py               →    ├── reeds
│   ├── copy_files.py                →    │   ├── core
│   ├── forecast.py                  →    │   │   ├── setup
│   ├── fuelcostprep.py              →    │   │   │   ├── a_createmodel.gms
│   ├── h2_storage.py                →    │   │   │   ├── b_inputs.gms
│   ├── hourly_load.py               →    │   │   │   ├── c_model.gms
│   ├── hourly_plots.py              →    │   │   │   ├── d_mga.gms
│   ├── hourly_repperiods.py         →    │   │   │   ├── d_objective.gms
│   ├── hourly_writetimeseries.py    →    │   │   │   └── e_solveprep.gms
│   ├── hydcf.py                     →    │   │   ├── solve
│   ├── mcs_sampler.py               →    │   │   │   ├── 1_tc_phaseout.py
│   ├── outage_rates.py              →    │   │   │   ├── 2_financials.gms
│   ├── plantcostprep.py             →    │   │   │   ├── 2_temporal_params.gms
│   ├── recf.py                      →    │   │   │   ├── 3_solve_allyears.gms
│   ├── transmission.py              →    │   │   │   ├── 3_solve_oneyear.gms
│   ├── writecapdat.py               →    │   │   │   ├── 3_solve_window.gms
│   ├── writedrshift.py              →    │   │   │   ├── 4_post_solve_adjustments.gms
│   ├── WriteHintage.py              →    │   │   │   ├── 5_varfix.gms
│   └── writesupplycurves.py         →    │   │   │   ├── 6_data_dump.gms
├── reeds                            →    │   │   │   └── solve.py
│   ├── __init__.py                  →    │   │   ├── solve_pcm
│   ├── checks.py                    →    │   │   │   ├── solve_pcm.gms
│   ├── financials.py                →    │   │   │   └── unfix_op.gms
│   ├── inputs.py                    →    │   │   └── terminus
│   ├── io.py                        →    │   │       ├── dump_alldata.gms
│   ├── log.py                       →    │   │       ├── powfrac_calc.gms
│   ├── output_calc.py               →    │   │       ├── report_dump.py
│   ├── plots.py                     →    │   │       ├── report_params.csv
│   ├── prasplots.py                 →    │   │       └── report.gms
│   ├── ra.py                        →    │   ├── hpc
│   ├── reedsplots.py                →    │   │   ├── aws_setup.sh
│   ├── remote.py                    →    │   │   └── srun_template.sh
│   ├── report_utils.py              →    │   ├── input_processing
│   ├── spatial.py                   →    │   │   ├── __init__.py
│   ├── techs.py                     →    │   │   ├── aggregate_regions.py
│   ├── timeseries.py                →    │   │   ├── calc_financial_inputs.py
│   └── units.py                     →    │   │   ├── check_inputs.py
├── ReEDS_Augur                      →    │   │   ├── climateprep.py
│   ├── augur_switches.csv           →    │   │   ├── copy_files.py
│   ├── capacity_credit.py           →    │   │   ├── forecast.py
│   ├── diagnostic_plots.py          →    │   │   ├── fuelcostprep.py
│   ├── prep_data.py                 →    │   │   ├── h2_storage.py
│   └── stress_periods.py            →    │   │   ├── hourly_load.py
├── reeds2pras                       →    │   │   ├── hourly_plots.py
│   ├── src                          →    │   │   ├── hourly_repperiods.py
│   │   ├── main                     →    │   │   ├── hourly_writetimeseries.py
│   │   ├── models                   →    │   │   ├── hydcf.py
│   │   └── utils                    →    │   │   ├── mcs_sampler.py
│   ├── test                         →    │   │   ├── outage_rates.py
│   │   └── Project.toml             →    │   │   ├── plantcostprep.py
│   ├── Project.toml                 →    │   │   ├── recf.py
│   ├── R2P_Test_Summary.png         →    │   │   ├── runfiles.csv
│   └── README.md                    →    │   │   ├── transmission.py
├── Augur.py                         →    │   │   ├── writecapdat.py
├── aws_setup.sh                     →    │   │   ├── writedrshift.py
├── b_inputs.gms                     →    │   │   ├── WriteHintage.py
├── c_mga.gms                        →    │   │   └── writesupplycurves.py
├── c_supplymodel.gms                →    │   ├── resource_adequacy
├── c_supplyobjective.gms            →    │   │   ├── reeds2pras
├── cases_examples.csv               →    │   │   │   ├── src
├── cases_small.csv                  →    │   │   │   │   ├── main
├── cases_standardscenarios.csv      →    │   │   │   │   ├── models
├── cases_test.csv                   →    │   │   │   │   └── utils
├── cases.csv                        →    │   │   │   ├── test
├── cbc.opt                          →    │   │   │   │   └── Project.toml
├── CITATION.cff                     →    │   │   │   ├── Project.toml
├── CODE_OF_CONDUCT.md               →    │   │   │   ├── R2P_Test_Summary.png
├── CONTRIBUTING.md                  →    │   │   │   └── README.md
├── cplex.op2                        →    │   │   ├── __init__.py
├── cplex.opt                        →    │   │   ├── capacity_credit.py
├── createmodel.gms                  →    │   │   ├── diagnostic_plots.py
├── d_solve_iterate.py               →    │   │   ├── prep_data.py
├── d_solveallyears.gms              →    │   │   ├── ra_calcs.py
├── d_solveoneyear.gms               →    │   │   ├── ra_switches.csv
├── d_solvepcm.gms                   →    │   │   └── stress_periods.py
├── d_solveprep.gms                  →    │   ├── solver
├── d_solvewindow.gms                →    │   │   ├── cbc.opt
├── d1_financials.gms                →    │   │   ├── cplex.op2
├── d1_temporal_params.gms           →    │   │   ├── cplex.opt
├── d2_post_solve_adjustments.gms    →    │   │   └── gurobi.opt
├── d2_unfix_op.gms                  →    │   ├── __init__.py
├── d2_varfix.gms                    →    │   ├── checks.py
├── d3_data_dump.gms                 →    │   ├── financials.py
├── dump_alldata.gms                 →    │   ├── inputs.py
├── e_powfrac_calc.gms               →    │   ├── io.py
├── e_report_dump.py                 →    │   ├── log.py
├── e_report_params.csv              →    │   ├── plots.py
├── e_report.gms                     →    │   ├── prasplots.py
├── environment.yml                  →    │   ├── reedsplots.py
├── gurobi.opt                       →    │   ├── remote.py
├── interim_report_batch.py          →    │   ├── report_utils.py
├── interim_report.py                →    │   ├── results.py
├── LICENSE                          →    │   ├── spatial.py
├── Manifest.toml                    →    │   ├── techs.py
├── Project.toml                     →    │   ├── timeseries.py
├── pyproject.toml                   →    │   └── units.py
├── raw_value_streams.py             →    ├── cases_examples.csv
├── README.md                        →    ├── cases_small.csv
├── requirements_dev.txt             →    ├── cases_standardscenarios.csv
├── restart_runs.py                  →    ├── cases_test.csv
├── run_pcm.py                       →    ├── cases.csv
├── runbatch.py                      →    ├── CITATION.cff
├── runfiles.csv                     →    ├── CODE_OF_CONDUCT.md
├── runstatus.py                     →    ├── CONTRIBUTING.md
├── sources_documentation.md         →    ├── environment.yml
├── sources.csv                      →    ├── LICENSE
├── srun_template.sh                 →    ├── Manifest.toml
├── tc_phaseout.py                   →    ├── Project.toml
└── valuestreams.py                  →    ├── README.md
                                     →    └── runreeds.py

Switches added/removed/changed

GSw_SkipAugurYear -> GSw_SkipRAyear
keep_augur_files -> keep_resource_adequacy_files

Additional changes

Added back a few hidden hourlize/reeds_to_rev files (.pylintrc and .pre-commit-config.yaml) that were dropped in the repo move

Issues resolved

1713 / 2017 (on the old repo)

Validation, testing, and comparison report(s)

Zero change for the USA_defaults case: results-v20260426_mainK0_USA_defaults,v20260425_reorgK0_USA_defaults.pptx

To check

Checklist for author

Details to double-check

Included comparison reports for appropriate test cases
Documentation updated if necessary
Code formatting standardized
Reusable functions used where possible instead of copy/pasted code

General information to guide review

Zero impact on results of default case
No large data file(s) added/modified
No substantive impact on runtime for full-US reference case
No substantive impact on folder size for full-US reference case
No change to process flow (runbatch.py, d_solve_iterate.py)
No change to code organization
No change to package requirements (environment.yml or Project.toml)

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

No

…{year}.gdx -> ccdata_{year}.gdx, Augur (in general) -> resource adequacy

…a functions back to reeds.resource_adequacy.stress_periods

…comments

…cumentation.md build until we decide what to do about it

wesleyjcole

I'm a fan of this new structure (it will take my brain a while to adjust to the new names and organization). Thanks for the thought you've put in on this.

I'll kick off a Windows test.

wesleyjcole · 2026-04-22T23:21:27Z

 Project.toml,Project.toml,1,ignore,ignore,,,,,,,,,,,
-b_inputs.gms,b_inputs.gms,1,ignore,ignore,,,,,,,,,,,


These files are removed, but the new files are not added. I didn't see how the new files get copies over to the runs folder. Can you help me understand that?

Yep that now happens upfront in runreeds.py, next to where we create the other folders in the case folder:

ReEDS/runreeds.py

Lines 1238 to 1243 in 1758746

## Copy code folder

shutil.copytree(

os.path.join(reeds_path, 'reeds'),

os.path.join(casedir, 'reeds'),

ignore=shutil.ignore_patterns('test'),

)

wesleyjcole · 2026-04-22T23:23:51Z

-solve_year = 2035 #need ReEDS Augur data for the input solve year
-weather_year = 2012 # must be 2007-2013 or 2016-2023


Why drop these comments?

For the first one, just because it didn't say much (and I was looking for all mentions of augur); for the second, because I'd like to have allowed values specified and checked as far upstream as possible instead of repeated throughout the code (to make it less painful when we hopefully someday start doing yearly updates to weather data).

But the bigger reason is that I think we should point people who want to run PRAS outside of the ReEDS/PRAS loop to postprocessing/run_reeds2pras.py (which handles the data handoff from ReEDS and a lot of data integrity checking before starting ReEDS2PRAS) instead of directly to the ReEDS2PRAS Julia package. That would be a bigger rewrite to this file though.

bsergi

The new structure is great. A couple of questions and a few small bugs, but otherwise looking good.

I did a test run locally (mac) and the gamslog.txt didn't include any of the input_processing script outputs so I think something is off there.

bsergi · 2026-04-22T21:10:34Z

@@ -19,6 +21,8 @@
                    help='Proceed without double-checking')
 parser.add_argument('--more_copyfiles', '-m', type=str, default='',


does the -m argument still work for individual files as long as the user specifies the full path (e.g., "reeds/core/terminus/report.gms")?

Also I think we'd need to add the path in the copy_cplex below for this to work

Good catch. For the copy_cplex functionality, it's a little different now since we only keep the single cplex file used by the run in the {casepath} root.

Instead of making copy_cplex take a number (which wouldn't work anyway if the user was using cbc, gurobi, or something else), we change it to a binary --copy_solve_settings flag, which, if True, copies the solver file specified by the solver/GSw_gopt switches for that case (parsed by reeds.inputs.get_optfile()) from the repo to the root of the casepath?

I like that. The only nuance would be that the user would have to modify the solve file in place for each batch of different settings, but I don't think that's too demanding and the simplicity of your proposed switch is nice.

Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

wesleyjcole · 2026-04-23T14:31:23Z

The Pacific test case ran fine for me on Windows.

I also noticed that there are still several instances of the old names throughout the code (e.g., "d_solveoneyear.gms" and "augur"). If you want any help cleaning those up, let me know.

Co-authored-by: Brian Sergi <bsergi@nlr.gov>

wesleyjcole · 2026-04-24T15:55:28Z

The Pacific test case ran fine for me on Windows.
I also noticed that there are still several instances of the old names throughout the code (e.g., "d_solveoneyear.gms" and "augur"). If you want any help cleaning those up, let me know.

Was your gamslog.txt file missing the input_processing script outputs, or is that just me?

My `gamslog.txt1 file had the input_processing script outputs in it.

patrickbrown4 · 2026-04-24T15:56:52Z

Was your gamslog.txt file missing the input_processing script outputs, or is that just me?

Yeah that was a separate issue, fixed by #26; it should work normally if you check it out now.

…ases.py working across the transition

bsergi

Thanks for the updates--a few small additional notes but overall looks good to me.

bsergi · 2026-05-04T14:51:35Z

+    """
+    sw = get_switches_base(case, **kwargs)
+    GSw_gopt = int(sw.GSw_gopt)
+    if GSw_gopt == 1:


I'm confused about this if block. Isn't the entry for GSw_gopt an integer? When would the length be more than 1? If there is more capability here than I am remembering it would be good to update the description for GSw_gopt in cases.csv.

Yeah it's a weird format but that's how it's described at https://gams.com/49/docs/UG_GamsCall.html#GAMSAOoptfile:

Even though I've never personally needed more than 10, I figured it'd be safest just to match the GAMS format

Ah ok, I see that the str conversion is just trying to get the number of digits here.

…onexistent runreeds_aws.py

patrickbrown4 added 16 commits April 15, 2026 11:09

copy .gov branch

bda96d5

add inadvertently dropped climate heuristics inputs

41c4296

delete files that have been renamed

2bb1cc7

rename ReEDS_Augur -> handoff, augur_data -> reeds_data, ReEDS_Augur_…

873eb2f

…{year}.gdx -> ccdata_{year}.gdx, Augur (in general) -> resource adequacy

move b_sets.gms to reeds/core/setup

10db2bd

move reeds.prase back to reeds.inputs; move reeds.resource_adequacy.r…

b90cef2

…a functions back to reeds.resource_adequacy.stress_periods

move auto-generated code to {casename}/autocode; fix report paths

1ce6daa

fix report_params.csv path; copy solver file

ce95181

rename run.py -> runreeds.py; fix restart_runs.py (add --copy_reeds)

694b005

Merge branch 'dev' into pb/reorg

f8a0d12

revert unintended changes; reeds/inputs -> reeds/input_processing in …

03aa081

…comments

fix a few more input_processing

1fc51cb

Merge branch 'main' into pb/reorg

c275928

add dev branch to CI

07038eb

Merge branch 'dev' into pb/reorg

3e0c012

fix hourly_writetimeseries.py import; turn off sources.csv/sources_do…

2e94718

…cumentation.md build until we decide what to do about it

patrickbrown4 requested review from bsergi, kodiobika and wesleyjcole April 21, 2026 20:24

patrickbrown4 changed the base branch from dev to main April 22, 2026 20:08

Merge branch 'main' into pb/reorg

7879de3

wesleyjcole reviewed Apr 22, 2026

View reviewed changes

bsergi requested changes Apr 23, 2026

View reviewed changes

patrickbrown4 and others added 2 commits April 23, 2026 08:26

Merge branch 'main' into pb/reorg

e814e0e

Fix straggler script names

f62577c

Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

patrickbrown4 and others added 4 commits April 23, 2026 08:32

remove references to dev branch

c5dfd10

Fix straggler script path

f3594b0

Co-authored-by: Brian Sergi <bsergi@nlr.gov>

more stragglers; simplify code copy

1758746

clean up straggler filenames

7772a5c

fix more filepaths

978a86d

patrickbrown4 added 6 commits April 24, 2026 10:08

restart_runs.py: change copy_cplex -> copy_solver_settings

3cc2a23

move get_optfile() from reeds.inputs to reeds.io

1712bd3

move reeds.output_calc -> reeds.reports

e70602e

fix optfile path

1bd27a8

fix runstatus.py and optfile path

278571f

reeds.io.read_output(): add backwards-compatibility to keep compare_c…

187ff2b

…ases.py working across the transition

patrickbrown4 mentioned this pull request Apr 28, 2026

ci: simplify workflow setup and labels #56

Merged

patrickbrown4 added 2 commits April 27, 2026 20:38

import gdxpds first to avoid warning

6068324

fix casemaker.py yearset

dfb6834

This was referenced Apr 29, 2026

Add NG Multipliers as inputs #65

Draft

MGA updates & employment outputs #21

Open

Merge branch 'main' into pb/reorg

30a0e18

bsergi approved these changes May 4, 2026

View reviewed changes

wesleyjcole approved these changes May 4, 2026

View reviewed changes

Comment thread docs/source/developer_best_practices.md

patrickbrown4 added 3 commits May 4, 2026 10:05

remove references to non-existing rabatch.py/augurbatch.py

2d282ec

update developer_best_practices.md for pathlib; remove reference to n…

0856d36

…onexistent runreeds_aws.py

Merge branch 'main' into pb/reorg

52f31b8

github-actions Bot added github_actions data_changes hourlize model_changes input_processing docs release tests postprocessing switches labels May 4, 2026

		Project.toml,Project.toml,1,ignore,ignore,,,,,,,,,,,
		b_inputs.gms,b_inputs.gms,1,ignore,ignore,,,,,,,,,,,

	## Copy code folder
	shutil.copytree(
	os.path.join(reeds_path, 'reeds'),
	os.path.join(casedir, 'reeds'),
	ignore=shutil.ignore_patterns('test'),
	)

		solve_year = 2035 #need ReEDS Augur data for the input solve year
		weather_year = 2012 # must be 2007-2013 or 2016-2023

		@@ -19,6 +21,8 @@
		help='Proceed without double-checking')
		parser.add_argument('--more_copyfiles', '-m', type=str, default='',

Conversation

patrickbrown4 commented Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Technical details

Proposed structure

Switches added/removed/changed

Additional changes

Issues resolved

Validation, testing, and comparison report(s)

To check

Checklist for author

Details to double-check

General information to guide review

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

Uh oh!

wesleyjcole left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bsergi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wesleyjcole commented Apr 23, 2026

Uh oh!

wesleyjcole commented Apr 24, 2026

Uh oh!

patrickbrown4 commented Apr 24, 2026

Uh oh!

bsergi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

patrickbrown4 commented Apr 19, 2026 •

edited

Loading