Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stage initial conditions for cycled and forecast-only via stage_ic job. #2475

Open
aerorahul opened this issue Apr 9, 2024 · 2 comments
Open

Stage initial conditions for cycled and forecast-only via stage_ic job. #2475

aerorahul opened this issue Apr 9, 2024 · 2 comments
Assignees
@KateFriedman-NOAA
Labels
feature New feature or request

Comments

@aerorahul
Copy link
Contributor

aerorahul commented Apr 9, 2024 •
edited by KateFriedman-NOAA

What new functionality do you need?

Presently, the cycled initial conditions are staged by setup_expt.py.
The initial conditions for forecast-only are staged by a job stage_ic.

This feature requests development of the stage_ic job to stage cycled initial conditions in addition to the ones for forecast-only.

What are the requirements for the new functionality?

The new stage_ic job should be able to handle both cycled and forecast-only modes as well as for gfs and gefs systems.
Extra points for making the job generic to allow users to provide (files) initial conditions of their choosing from arbitrary locations and structure.

Acceptance Criteria

Add the stage_ic job to the first half-cycle of the cycled mode.
@aerorahul aerorahul added feature New feature or request triage Issues that are triage labels Apr 9, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the triage Issues that are triage label Apr 10, 2024
@aerorahul aerorahul mentioned this issue Apr 29, 2024
Closed
7 tasks
@WalterKolczynski-NOAA
Copy link
Contributor

@KateFriedman-NOAA See #2559 for changes relevant to this issue

@WalterKolczynski-NOAA
Copy link
Contributor

@KateFriedman-NOAA #2545 is also adding options to stage_ic you'll have to handle

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Rename exglobal_stage script ahead of rewrite
25a959e 
- Change the extension of the exglobal_stage ex-script
from "sh" to "py".

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Update for pythonization of stage script
9bf02ee 
- Update script extension for ex-script from "sh" to "py".
- Pull COM* variable declares up from ex-script.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Add PYTHONPATH to stage_ic.sh
8d2203d 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Remove IC staging from setup
ac9fe80 
Remove the functions and calls to set up
symlinks to ICs in ROTDIR.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Add initial changes to pythonize and yamlize staging
04d75e3 
- Add initial new yaml files for staging information
- Add new stage.py to python tasks.
- Add first draft pythonization of stage ex-script.

Much more work is still to be done.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Remove warm start check for ingesting fv3.yaml.j2
f6548e3 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Merge remote-tracking branch 'upstream/develop' into feature/issue_2475
7fc7b89 
Revert changes to workflow/setup_expt.py; will do in later task

Refs NOAA-EMC#2475

* upstream/develop:
  Sea-ice analysis insertion (NOAA-EMC#2584)
  Refactored archiving (NOAA-EMC#2491)
  Add remove RUNDIRS step in CI before creating experements (NOAA-EMC#2607)
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Split fv3 yaml into warm and cold
11df1af 
Also fix to set target and remove source

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Fix other yamls to work
606c34a 
Add target, remove source, and update file info

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Add COM variables exports in JGLOBAL_STAGE_IC
0176d2e 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Update fv3_cold yaml
4861ed1 
Update to use mkdir and copy instead of target and required

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Update stage python scripts
b7a8c82 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Remove cycle_HH and cycle_YMD from fv3_cold yaml
5898ee2 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Remove unneeded funcions from stage.py
5e8f92f 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Refactor stage yamls
1cf0063 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Add MEMDIR array to JGLOBAL_STAGE_IC
9a39787 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Remove stage_ic job dependencies from setup
cd362fd 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Set MEMDIR as empty if not set
859d31d 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Add keys_gefs in exglobal_stage_ic.py
a805fa8 
If RUN=gefs add keys_gefs to keys.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 30, 2024
@KateFriedman-NOAA
Add USE_OCN_PERTURB_FILES to gfs config.base
719701b 
Set to .false. by default; needed for staging job

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 30, 2024
@KateFriedman-NOAA
Stage job yaml updates
cf50f54 
- remove master yaml, no longer using
- update fv3_cold, ice, ocean, and wave yamls

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 30, 2024
@KateFriedman-NOAA
Updates to stage.py
b625906 
- General cleanup
- Rework determine_stage function
- Rework execute_stage function

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 30, 2024
@KateFriedman-NOAA
Yaml updates
4e94734 
- Delete da.yaml.j2; will remake in follow-up work
- Update fv3_warm.yaml.j2 to not use src/head variables

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 31, 2024
@KateFriedman-NOAA
Move COM declares inside MEMDIR loop
b7301f0 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 31, 2024
@KateFriedman-NOAA
Remove yamls not yet needed
4a92a10 
Also clean up unneeded python imports

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 31, 2024
@KateFriedman-NOAA
Remove nest option and cleanup python imports
35cb29d 
Will add nest option in future task

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 31, 2024
@KateFriedman-NOAA
Undo change to setup_expt.py
21c1d22 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 3, 2024
@KateFriedman-NOAA
Adjust MEMDIR export in JGLOBAL_STAGE_IC
a4cec79 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 3, 2024
@KateFriedman-NOAA
Add fv3_warm and fv3_nest yamls back in
87302c2 
- Add DO_NEST back into stage.py
- Add back in reworked yamls for fv3_warm and the nest

Refs NOAA-EMC#2475
This was referenced Jun 3, 2024
Closed
Draft
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 3, 2024
@KateFriedman-NOAA
Add whitespaces to address pynorms errors
9be34aa 
Add extra whitespaces in exglobal_stage_ic.py to address E231 error.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 3, 2024
@KateFriedman-NOAA
Address pynorm errors in exglobal_stage_ic.py
c2232cd 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 3, 2024
@KateFriedman-NOAA
Resolve pynorms error in stage.py
561be58 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 3, 2024
@KateFriedman-NOAA
Indent secondary yaml loops
3f08f03 
Improve readability of the stage yamls

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 5, 2024
@KateFriedman-NOAA
Rename fv3 yamls to atmosphere yamls
f793d0b 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 14, 2024
@KateFriedman-NOAA
Replace BASE_CPLIC with BASE_IC
60511e9 
Also add ICSDIR to config.base

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 14, 2024
@KateFriedman-NOAA
Update JGLOBAL_STAGE_IC for cycled mode
f9e3a9c 
Also update wave COM variable

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 14, 2024
@KateFriedman-NOAA
Add ICSDIR setting to stage configs
81d070c 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 14, 2024
@KateFriedman-NOAA
Add cycled keys to exglobal_stage_ic.py
bb94111 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 14, 2024
@KateFriedman-NOAA
Add cycled analysis yaml to stage.py
a675a3d 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 14, 2024
@KateFriedman-NOAA
Add cycled staging job to mesh
b97a482 
- set job for gdas_half cycledef only
- update fcst jobs dependencies

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 14, 2024
@KateFriedman-NOAA
Remove fill_ROTDIR function and update icsdir option
937a245 
- Delete the fill_ROTDIR functions used in half cycle setup;
functions are replaced by half cycle stage_ic jobs
- Update --icsdir setup flag to allow user provided path for all RUNs

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 14, 2024
@KateFriedman-NOAA
Merge remote-tracking branch 'origin/develop' into feature/issue_2475
87309fc 
* origin/develop:
  Add observation preparation job for aerosols DA to workflow (NOAA-EMC#2624)
  Remove ocean daily files (NOAA-EMC#2689)
  Update Jenkinsfile
  Add Hercules-EMC to the Jenkins configurable parameter list (NOAA-EMC#2685)
  Update gdas.cd and gsi_utils hashes (NOAA-EMC#2641)
  Add ability to use GEFS replay ICs (NOAA-EMC#2559)
  Replace `sleep` with `wait_for_file` (NOAA-EMC#2586)
  Add COM template for JEDI obs (NOAA-EMC#2678)
  Link both global-nest fix files and non-nest ones at the same time (NOAA-EMC#2632)
  Update ufs-weather-model  (NOAA-EMC#2663)
  Add ability to process ocean/ice products specific to GEFS (NOAA-EMC#2561)
  Update cleanup job to use COMIN/COMOUT (NOAA-EMC#2649)
  Add overwrite to creat experiment in BASH CI (NOAA-EMC#2676)
  Add handling to select CRTM cloud optical table based on cloud scheme and update calcanal_gfs.py  (NOAA-EMC#2645)

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 17, 2024
@KateFriedman-NOAA
Introduce relpaths to stage yamls
d0945da 
Makes yamls more generic and uses paths set previously

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 17, 2024
@KateFriedman-NOAA
Create analysis stage yaml
3c45d58 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 17, 2024
@KateFriedman-NOAA
Combine stage yamls into one
03b5024 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 17, 2024
@KateFriedman-NOAA
Staging job updates
f3c72b2 
- Add new keys
- Remove determine function
- Update execute function to use single stage.yaml.j2

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 17, 2024
@KateFriedman-NOAA
Staging variable updates
31bb29c 
- Add RDATE and DTG_PREFIX
- Remove USE_OCN_PERTURB_FILES
- Always declare COM_ATMOS_ANALYSIS

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 20, 2024
@KateFriedman-NOAA
Update for COMOUT, replay, and add mediator
e833c2a 
- Convert COM variables to COMOUT variables
- Add mediator IC copy
- Add DTG_PREFIX to filename constructor
- Add replay updates

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 20, 2024
@KateFriedman-NOAA
Resolve pynorms errors
df0c933 
Refs NOAA-EMC#2475
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KateFriedman-NOAA KateFriedman-NOAA

Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@aerorahul @WalterKolczynski-NOAA @KateFriedman-NOAA @HenryWinterbottom-NOAA