BUILD_FORECAST_SCENARIO Hook Timing

[notstarboard]

Hi All! Another question for you - Dave Ullman built a customization for ASGS that allows us to close the Providence Hurricane Barrier based on a date we add to the ASGS config file. It works using ASGS's hooks to call a script that generates a time_varying_weir.in file based on the configured barrier closure time. The config file looks like the following:

TIMEVARYINGWEIR=on
WEIRCLOSEDATE=2021081218
hooksScripts[BUILD_NOWCAST_SCENARIO]="generate_tvwfile.sh"
hooksScripts[BUILD_FORECAST_SCENARIO]="generate_tvwfile.sh"

This seems to work as intended for the nowcast, but it definitely doesn't work for the forecast. generate_tvwfile.sh uses "$SCENARIODIR/time_varying_weir.in" as the name of the tvw file it generates. So, if SCENARIODIR is still the nowcast folder when the code runs via the hook, the time_varying_weir.in file will be created in the nowcast directory instead of the forecast directory. As a result the barrier does not close. The run still succeeds, but runs with the barrier open and closed also have identical output. I would have expected slight differences just due to the barrier having been closed during the nowcast piece, so it's possible that's broken too, but the results are identical.

Here's the relevant section from the log file (notice the BUILD_FORECAST_SCENARIO hook and the outfile in the nowcast directory):

[2023-May-11-T18:50:38+0000] INFO: asgs_main->manageHooks->executeHookScripts(): Executing BUILD_FORECAST_SCENARIO hook script /home/joshua_port_uri_edu/asgs-fork/asgs/generate_tvwfile.sh.
Loading properties.
Finished loading properties.
[2023-May-11-T18:50:39+0000] INFO: generate_tvwfile.sh: running write_tvwfile.sh with args: /home/joshua_port_uri_edu/asgs-fork/asgs/input/meshes/ricv1_noriv_nopump/time_varying_weir.in_template /work/pi_iginis_uri_edu/joshua_port_uri_edu/asgs/scratch/asgs2405052/16/nowcast/time_varying_weir.in 2021081218 2021081218
write_tvwfile.sh: infile=/home/joshua_port_uri_edu/asgs-fork/asgs/input/meshes/ricv1_noriv_nopump/time_varying_weir.in_template
write_tvwfile.sh: outfile=/work/pi_iginis_uri_edu/joshua_port_uri_edu/asgs/scratch/asgs2405052/16/nowcast/time_varying_weir.in
Cold start at: 2021/08/12/18
Start Weir close at: 2021/08/12/18
coldstart time = 1628791200
weir close time = 1628791200
start weir closure at 0 days 0 hours 0 minutes
end weir closure at 0 days 1 hours 0 minutes

---

My question is this: Is there another hook we should be using instead of BUILD_FORECAST_SCENARIO so that SCENARIODIR will be set as the forecast folder? Alternatively, is there another approach you would recommend (perhaps using the same hook, but changing "nowcast" to "forecast" in the SCENARIODIR whenever it runs via the BUILD_FORECAST_SCENARIO hook)? I'm pretty sure the code generates a valid tvw file based on the configuration (and if it doesn't I can sort that out). I'd just like advice on how to get the file to the right place at the right time.

I'm going to reach out to Dave in a moment to see if he ever found another solution for this, but I'd love to get your thoughts! Thanks.

[wwlwpd]

What does this script look like and what environmental variables is it affecting?

I suggest that hooks are not the place to override variables, especially ones that are not exported by the time asgs_main.sh runs.

Bash variables can't be updated by a child process in a way that persists them in the parent process. To persist environmental variables between child and parent, we either:

• use run.properties to write the latest value, read last value
• export a variable via _ASGS_EXPORTED_VARS (these are more for internal bookkeeping), then update them via script (even asgs_main.sh) using a source command to inline them (i.e., in the child output a valid bash script that updates the variable, then source it)

My suggestion is to use run.properties as the local metadata store that you can use to read/write when you need variables that might change. There's no central function to read or write using it, but there are plenty of examples. output/opendap_post2.sh might be a good one to check out. If that's not germane to your need, please clarify here.

Thanks for asking, Josh!

[notstarboard]

This script doesn't change any existing environment variables, so we should be okay there! It just uses SCENARIODIR to determine where to put the tvw file it generates, which is why I'm so focused on that variable. The code does add a couple of lines to run.properties as well that are used in generate_tvwfile.sh, which I think Dave set up via a customization to asgs_main. I am totally on board the "let's not make customizations to core ASGS scripts unless absolutely necessary" train, so in the medium to long term it would probably be good to find a new solution for this. In the short term though I'm just trying to fix up what we have!

I'm basically trying to figure out the best way of getting custom code to run after the forecast directory has been created (so the time_variable_weir.in file can be dropped there), but the forecast hasn't started yet (so the file is in place by the time it is needed). I was assuming the BUILD_FORECAST_SCENARIO hook would do that, especially because of lines 78 and 79 here:

asgs/manageHooks.sh

Line 78 in cd0b377

"INITIALIZE_FORECAST_SCENARIO"|"CAPACITY_WAIT"|"BUILD_FORECAST_SCENARIO"|"SUBMIT_FORECAST_SCENARIO")

I would have assumed that mypath == $pwd == /work/.../forecast when the BUILD_FORECAST_SCENARIO code hooks in, but that doesn't seem to be how it works in practice. My forecast scenario is named "forecast" - I'm now realizing that it might have been confusing to just talk about the "forecast" directory!

[wwlwpd]

I am totally on board the "let's not make customizations to core ASGS scripts unless absolutely necessary" train

What we dicourage is forking ASGS, then doing all kinds of crazy stuff that deviates so far from the main code that you're stuck on your own frankenstein. We can't help people who do that. It's not an ASGS thing, it's a general OSS principle that some people not used to tracking upstream code violate. Don't worry, you're not doing that here..not yet.

So I think in general we are completely open to making the hooks (or anything) more useful. So if there is a generic improvement that could be submitted as a PR (even as a draft), then it might save people a lot of headache in the future. I think you're the first group to really try to use these things, other than @jasonfleming (who added support for hooks).

Can you share your changes and the custom script somehow so I can see what you're doing?

[notstarboard]

Yeah, sure! The two tvw files are the bulk of the customization. If I recall correctly there are just a couple lines added to asgs_main to write the tvw information to the run.properties file. I included my config file just to show how I'm calling the code, but it's the same as my comment above. There may be a couple of other related tweaks - my memory's a bit hazy.

TVW.zip

[notstarboard]

And the fort15 template for the ricv1 mesh calls out the time_varying_weir.in file - here's the line:

&TVWControl USE_TVW=.TRUE. TVW_FILE='time_varying_weir.in' NOUT_TVW=3 TOUTS_TVW=15 TOUTF_TVW=30 NSPOOL_TVW=900 /
%offsetline%

It's just like the end of the template here:

https://github.com/StormSurgeLive/asgs/blob/cd0b377fe83068071a59c1552a68707ba8c82e42/input/meshes/uriv18/fort.15.template

[notstarboard]

I did a bit of testing with this and I found that the forecast job had already been submitted per the log file (and per my recollection while monitoring the test), but when you go to SCRATCHDIR/asgs######/status/hook.status.json it shows that ASGS is only on the NOWCAST_POLLING status:

"hook.status.latest" : "NOWCAST_POLLING"

[2023-May-13-T23:03:04+0000] INFO: asgs_main->manageHooks.sh->writeHookStatus(): Writing status associated with ASGS hooks to '/work/pi_iginis_uri_edu/joshua_port_uri_edu/asgs/scratch/asgs3838186/status/hook.status.json'.
[2023-May-13-T23:03:04+0000] INFO: asgs_main.sh>downloadCycloneData(): Checking remote site for new advisory...
[2023-May-13-T23:03:04+0000] INFO: asgs_main.sh>downloadCycloneData(): Options for /home/joshua_port_uri_edu/asgs-fork/asgs/get_atcf.pl are as follows : --storm 08 --year 2021 --ftpsite filesystem --fdir /home/joshua_port_uri_edu/asgs-fork/henri_test --hdir /home/joshua_port_uri_edu/asgs-fork/henri_test --rsssite filesystem --trigger rssembedded --adv 16
[2023-May-13-T23:03:04+0000] INFO: asgs_main.sh>downloadCycloneData(): Checking remote site for new advisory...
INFO: get_atcf.pl: Advisory '16' for storm HENRI was found in the index-at.xml file.
[2023-May-13-T23:04:03+0000] INFO: padcirc.forecast: asgs_main.sh>monitorJobs(): The padcirc.forecast job has started.

[notstarboard]

I am currently trying to resolve this by setting SCENARIODIR=$ADVISDIR/$ENSTORM when the BUILD_FORECAST_SCENARIO hook code is called instead of $PWD. It's possible pwd just isn't the SCENARIODIR when this hook activates, so provided the forecast folder exists at this time, this should solve the problem. If it turns out that the folder doesn't exist yet I'll have to figure out when it gets created and then somehow call the code after that.

[notstarboard]

I assumed ADVISDIR and ENSTORM would have been set at the environment level, but at least one of them was null at the time of the hook running. So, this didn't do the trick.

I know from before that PWD gives the nowcast folder, so I'm now just specifying the forecast folder path by starting with PWD and then changing "nowcast" to "forecast". That will certainly give the right folder, then, even if it's a bit janky. The only question is whether the folder already exists at the time the hook runs. And it very likely does, given the name of the hook, when it seems to run from the code I skimmed, and the fact that the very next step in the log file involves writing to the forecast directory. I'll reply back with how this test goes.

[notstarboard]

The forecast folder actually doesn't exist yet, but the error is now on the second to last line of the code, so this could be the final tweak. I'm now explicitly creating the forecast folder in this code. If ASGS checks if the folder exists and/or doesn't have error handling around trying to create the folder, this will work fine. If ASGS errors out if it's unable to create the folder because one already exists, this will fail, and if that happens I 100% will need to either play around with different hooks or call this code somewhere else.

[notstarboard]

This seems to have worked! Now if it runs into trouble it would probably be because of the files that it generates. I'm going to check out the output and see if things look good, and if so I'll close this.

[notstarboard]

This appeared to do the trick! At a minimum there are now deviations in water level between the barrier open and barrier closed cases, so clearly the TVW file had an effect. Whether the file itself is exactly what we need is something we can validate.

I still think it's a bit surprising behavior that the forecast/enstorm/stormdir (or whatever we're calling it) directory doesn't exist when the BUILD_FORECAST_SCENARIO hook runs. If that's still the case (haven't rebuilt ASGS in a bit!), the timing of that could probably stand to be pushed back a tad (or the folder creation could be moved up, I suppose). It seems it's not necessary to get this customization going, though!

[notstarboard]

My issue is resolved here, so I'm going to close. If you'd like to discuss more about the hooks and this would be a good place to do that, of course feel free to open it back up!

[jasonfleming]

If this script needs to run after the other input files have been created but before adcprep runs, then I think we need a new hook called PREPROCESS_SCENARIO.

[notstarboard]

A new hook would be a great way to resolve this! I checked in with Dave about how he got around this and he actually moved the BUILD_FORECAST_SCENARIO hook later in asgs_main, so he basically made himself a new hook.

[jasonfleming]

Thank you @notstarboard can you tell me exactly where he moved it? That would help guide our placement of a new hook. I looked at the asgs_main.sh in the zip file you posted above, but the BUILD_FORECAST_SCENARIO hook is in its original location.

[dave-gso]

@jasonfleming, josh let me know about this thread. i moved the executeHookScripts "BUILD_FORECAST_SCENARIO" statement down below the SWAN hotstarting section and the section that makes the scenario directory. if this is not clear, i can send you my version of asgs_main.sh

[jasonfleming]

Hey @dave-gso yes, please send me your modified asgs_main.sh this will help us improve asgs for your case as well as other asgs developers and operators. :-)