Feature Request: Warm Start

[rbreslavsky]

I'd like to request warm start capability as a feature for pyFun. Right now, this is possible by using shell commands to copy over previously run simulation files from other "seed" runs, but requires diligent tracking of phases and phase iteration count

Perhaps the feature could work by indicating one or more seed conditions (alpha, Mach, etc) in the run matrix, or a seed run case number. It would also be nice to be able to drop the seed run iteration history as an option (restart = 'on_nohistorykept).

[nasa-ddalle]

I've added a "WarmStart" setting in the "RunControl" setting as of commit 7ca55f61. If you set this to true, as in

"RunControl": {
    "WarmStart": true
}

Then at least CAPE will get out of your way and not set restart = 'off'. I've got a good plan to actually have pyfun go and get the appropriate .flow file automatically, but this is a start.

[nasa-ddalle]

Ok! So commit 5c74099 has an implementation for FUN3D to use warm starts that I've tested in action. There are two ways to use it. The first is to just set the WarmStart flag in the RunControl section and do nothing else, i.e.

"RunControl": {
    "WarmStart": true
}

This will just tell pyfun to set restart='on' for your first phase (in fun3d.00.nml), but you're responsible for actually putting the requisite grid file and .flow file (usually pyfun00.flow) into the case folder.

The second method actually has CAPE do all the work for you, and works a little bit like this:

"RunControl": {
    "WarmStart": true,
    "WarmStartFolder": "../../poweroff/m%(mach).2fa%(alpha).1f",
    "WarmStartProject": null
}

When you try to start a case, it's going to look for a grid file and a .flow file in the WarmStartFolder (which is expanded relative to the case folder in which you're trying to run the warm-started case). You can use the value of any run matrix key in your current run matrix and format the string using the Python "%(mach).2f" syntax. That for example will take the value of the Mach number for your case (or w/e you defined the run matrix key mach to mean) and print it as a float w/ two decimals... such as m0.90. If pyfun doesn't find the required files in that folder, it won't start the case.

The WarmStartProject flag is optional, and it's primary feature is allowing you to start the new (warm-started) case from an adapted mesh in source folder. So you could set this to "pyfun01" for example to use the result after one mesh adaptation in a previous case.

I should probably add a WarmStartIter flag so that the case won't start unless the source folder has run to an explicit step (currently it'll run as long as any pyfun00.flow file is present).

Regarding your "seed conditions" concept, one way to do this is something like this:

"RunControl": {
    "WarmStart": true,
    "WarmStartFolder": "../%(mach).2fa0.0"
}

This will try to warm-start each case from a similarly-named case but at 0.0 degrees angle of attack. The new WarmStart capability is smart enough to turn WarmStart off for any case where the WarmStartFolder is the same as the run folder. However, you will have to do some work to use different settings for m0.90a0.0 and m0.90a1.0. You could either use "hooks" for that or just have a separate JSON file for the a0.0 cases.

I think I've got a good solution here, but I'm interested to hear your thoughts!

[rbreslavsky]

I think this is a great start!

I'm also thinking that this can be used to drive a simulation to a certain number of steps, and then setting up a new folder where we can dump out plt files for every boundary_animation_freq steps so that we can average those plt files for visualizations. This would be a nice ability because FUN3D otherwise periodically generates plt files every boundary_animation_freq steps for the entire iteration history (rather than say for steps > N), which can be cumbersome.

[vcoralic]

@nasa-ddalle, could you elaborate on what you mean by:

However, you will have to do some work to use different settings for m0.90a0.0 and m0.90a1.0. You could either use "hooks" for that or just have a separate JSON file for the a0.0 cases.

The statement confuses me because in my mind the run/solver settings of the simulation that was used to generate the warm start files can inherently be different than those that consumes said warm start results. I am probably not quite understanding what you are trying to say. In any case though, this is a great simplifier to the current workflow we are having to use.

@rbreslavsky, in your last post, you are referring to mimicking what's happening on the Cart3D side with respect to solution averaging, correct? If that's the case, I think that's probably a feature independent of the warm start. I think the "right" way to implement something like this is not to continually start/stop the simulation at a given output frequency, but rather to run everything in one shot and average at the end. The guiding principle here is that compute is expensive, but storage is cheap. I am not familiar enough with Cart3D to advocate changing what Derek has done on that side of things, but for FUN3D, I think the most efficient thing to do is run up until you are ready to start averaging, say that's phase N, then go into phase N+1 with your output/averaging frequency set, run until the end, and then have a CAPE post-processing method just average the outputs into a final file. @nasa-ddalle, any thoughts? Again, we should probably separate this out into a separate issue :).

[nasa-ddalle]

@vcoralic We're on the same page, it's just a little bit of a challenging sentence for me to write out clearly...

The statement confuses me because in my mind the run/solver settings of the simulation that was used to generate the warm start files can inherently be different than those that consumes said warm start results.

In this case the m0.90a0.0 and m0.90a1.0 are going to need different settings. The fact that they have similar names is kind of a distraction. My comment really just describes two ways of achieving the objective of having those two cases have different settings. One is to basically have two separate run matrices that generate similar-looking case names (that's what I meant by two JSON files). The other approach is probably not as practical but allows you to have different settings within the same run matrix.... You can achieve this now using "hooks," which are Python functions that run at certain points during the case setup, e.g. in the tutorial at https://github.com/nasa-ddalle/pyover02-powered_nacelle. In version 1.1 having conditions-dependent settings should get a lot easier, but in any case it's probably easier to just have two run matrices for achieving the WarmStart objective in most cases.