No such file or directory: config.experiment

[benjamin-cash]

Hi @NeilBarton-NOAA - When I use the new /ci/cases/sfs/C96mx100_S2S.yaml, I am getting the following error:

./setup_expt.py gefs forecast-only --idate 1994050100 --edate 1994050100 --pslot C192mx025_yaml_test --app S2S --icsdir /work2/02441/bcash/frontera/ufs_input/ICSDIR/HR4/C192mx025 --resdetatmos 192 --resensatmos 192 --resdetocean 0.25 --start cold --gfs_cyc 1 --nens 10 --comroot /scratch1/02441/bcash/RUNS/COMROOT --expdir /scratch1/02441/bcash/RUNS/EXPDIR --yaml /work2/02441/bcash/frontera/global-workflow/ci/cases/sfs/C96mx100_S2S.yaml

EDITED:  /scratch1/02441/bcash/RUNS/EXPDIR/C192mx025_yaml_test/config.base as per user input.
EDITED:  /scratch1/02441/bcash/RUNS/EXPDIR/C192mx025_yaml_test/config.stage_ic as per user input.
Traceback (most recent call last):
  File "/work2/02441/bcash/frontera/global-workflow/workflow/./setup_expt.py", line 378, in <module>
    main()
  File "/work2/02441/bcash/frontera/global-workflow/workflow/./setup_expt.py", line 368, in main
    update_configs(host, user_inputs)
  File "/work2/02441/bcash/frontera/global-workflow/workflow/./setup_expt.py", line 84, in update_configs
    edit_config(cfg_file, cfg_file, cfg_dict)
  File "/work2/02441/bcash/frontera/global-workflow/workflow/./setup_expt.py", line 154, in edit_config
    with open(input_config, 'rt') as fi:
         ^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/scratch1/02441/bcash/RUNS/EXPDIR/C192mx025_yaml_test/config.experiment'

I don't get this error if I revert to an earlier version of the .yaml files. I'm not sure yet what change is causing this - none of my other experiments have created a config.experiment file.

[benjamin-cash]

Looking around, I assume it is coming from the experiment block in C96mx100_S2S.yaml. Did you make an update to your branch related to this block?

[benjamin-cash]

Update - Merged the current SFS branch into my container branch and the problem persists, so I think that wasn't it.

[NeilBarton-NOAA]

Ben, I haven't tested this for C192mx025, Could you run it with C96mx100 to see if you still get the error?

[benjamin-cash]

Same error - I also tested this on Hercules, so it isn't just on Frontera. I'm working from my branch here: https://github.com/benjamin-cash/global-workflow/tree/feature/SFS_container, and I just merged your branch into that so I shouldn't be missing anything. I'm going to do a clean checkout of your branch now and see if that works for me.

[benjamin-cash]

Same result with a new checkout of the code, @NeilBarton-NOAA does this work for you straight from the repo?

[NeilBarton-NOAA]

yes, I'm using the create_experiment.py script though

${HOMEgfs}/workflow/create_experiment.py --yaml "${HOMEgfs}/ci/cases/sfs/C96mx100_S2S.yaml"

[benjamin-cash]

Interesting! That runs without issue - it turns out that you can't feed C96mx100_S2S.yaml directly to setup_expt, which is not something I had realized.

[NeilBarton-NOAA]

thanks for testing

[benjamin-cash]

Speaking of testing, flagging something as a future issue. In one of the configurations I was testing the runlength was set to 100+ days, with 6h output and multiple ensemble members. It was taking over an hour to complete rocotorun, rocotostat, etc, and it seems that processing all of the atmos_prod tasks brought rocoto to a standstill.

[NeilBarton-NOAA]

Thanks Ben. We also have this issue for GEFS and am informing the g-w team at this week's meeting.

[benjamin-cash]

As a follow up, 24 hour output and 124 days is still sluggish (5 min or so) but not nearly as bad.