Question on Dakota Postprocessing

[kwesi-boafo]

Dear Dakota Community,
I am coupling Dakota to another solver for UQ applications and I have an issue with passing output data back to Dakota.
Using a driver script, I successfully performed preprocessing as well as executing the solver. At the post-processing stage, I want to pass the responses back to the Dakota results.out file but without success so far.

For a sample size of 5 for instance, 'workdir.1-workdir.5' were created, containing the output files from the solver executable. Now I am trying to extract data from some of these output files (these data are in columns) so that the extracted data can be written to the results.out file in the main directory containing the script. Currently though, what I get with my script is that a results.out file is created in each 'workdir*' and instead of the column of data printed, I rather get the names of the responses printed to file. The actual values are rather displayed on the screen.

I am sure there is a way to go about this and I will appreciate suggestions. The post-processing part of my bash script is appended below:

TIME= awk '{print $1}' PLOTT.asc
TFUELT= awk '{print $2}' TFUELT.asc
TCOOLT= awk '{print $2}' TCOOLT.asc
REACTT= awk '{print $2}' REACTT.asc

echo "$TIME TIME" >>$results
echo "$TFUELT TFUELT" >>$results
echo "$TCOOLT TCOOLT" >>$results
echo "$REACTT REACTT" >>$results

Best regards,
/Emmanuel

[jadamstephens]

Emmanuel,

There are a couple of things to note.

First: If Dakota is creating the work directories (workdir.N) for you, then Dakota expects to find the results files in those work directories. They are being (correctly) written there by your workflow because, even though your driver script may be located one directory up, Dakota is executing it in each work directory. (If you echo out the 'pwd' command in the driver script, you can see this.)

Second: The four commands where you use awk are suspicious to me. There may be a shell script where this will work, but it won't work in bash, for example.

You might need something like this, instead:

TIME=$(awk '{print $1}' PLOTT.asc)

The $() tells Bash to collect the stdout from the enclosed command and turn it into a string. That string will be stored in the variable TIME. An alternative for some older shells is to enclose the command in backticks: `awk '{print $1}' PLOTT.asc`.

This assumes that PLOTT.asc contains a single row of data. If there are multiple rows of data and you want to treat all of them like Dakota responses, you'll need to make bigger changes to your driver and study. A Dakota response is a scalar (single) value, not a vector. You'll need to define additional responses in your Dakota input (TIME1, TIME2, etc) and perhaps loop over the result of that awk command to write them into the results file.

[kwesi-boafo]

Thanks Adam for your helpful hints and tips.
I have resolved the issues with the Bash script but I am facing issues with specifying the responses correctly in the Dakota input file. I tried your suggestion of increasing the number of entries for each response variable but I keep getting errors. I read a bit from the users manual and thought that the use of the Field Data option might be appropriate (since I have a number of data points for each response) however, the information provided in that section of the manual was not that detailed.

[jadamstephens]

Either ordinary responses or field responses would work. The set up in Dakota is slightly different, but the driver script would be nearly identical.

For ordinary responses, the responses section of your Dakota input needs to look something like:

responses
    response_functions 12   # assuming there are 12 total scalar values
    response_descriptors 'TIME1'
                         'TIME2'
                         'TIME3'
                         <... repeat for the other four responses >
    no_gradients
    no_hessians

In your bash script, you'll need something like:

id=0
for r in $(awk '{print $1}' PLOTT.asc)
do
  id=$((id+1))
  echo $r TIME${id} >> $results
done

And repeat for the other three responses, modifying as needed.

[kwesi-boafo]

Thanks a lot Adam! the script works but I guess switching to python would give more flexibility and functions.
Best regards

[kwesi-boafo]

Hello Adam,
I am returning to this subject after a pause, and will be grateful for your suggestions:

I revised my bash script with the loop as you suggested and observed the following:
The values of the responses are written to the results file as expected; however, only a single numeric value (1) is written to the '.dat' file. As I have shown below, the statistics produced by Dakota obviously have some issues, but I don't know how to correct them.
So, how can I modify the Dakota input and driver script to ensure that the correct response values are written back to Dakota, producing meaningful statistics? Each evaluation produces more than 3,000 values for each parameter. Excerpts of the Dakota input, driver script, and output from the console are shown below:

Dakota output on the console:

"Evaluation 1 has completed

Active response data for evaluation 1:
Active set vector = { 1 1 }
1.0000000000e+00 TFUEL
1.0000000000e+00 TCLAD

Active response data for evaluation 2:
Active set vector = { 1 1 }
1.0000000000e+00 TFUEL
1.0000000000e+00 TCLAD

<<<<< Function evaluation summary: 2 total (2 new, 0 duplicate)

Statistics based on 2 samples:

Sample moment statistics for each response function:
Mean Std Dev Skewness Kurtosis
TFUEL 1.0000000000e+00 0.0000000000e+00 0.0000000000e+00 -3.0000000000e+00
TCLAD 1.0000000000e+00 0.0000000000e+00 0.0000000000e+00 -3.0000000000e+00

95% confidence intervals for each response function:
LowerCI_Mean UpperCI_Mean LowerCI_StdDev UpperCI_StdDev
TFUEL 1.0000000000e+00 1.0000000000e+00 0.0000000000e+00 0.0000000000e+00
TCLAD 1.0000000000e+00 1.0000000000e+00 0.0000000000e+00 0.0000000000e+00

Dakota Input:
responses,
response_functions = 2
response_descriptors = 'TFUEL'
'TCLAD'
no_gradients
no_hessians

Driver Script:

writing response1 to results

id=2
r=2
for r in $(awk '{print $9}' GH2.out.ch01)
do
id=$((id+1))
r=$((r+1))
echo $r FUELT${id} >> $results

echo $r >> $results

done

writing response2 to results

it=2
k=2
for k in $(awk '{print $8}' GH2.out.ch01)
do
it=$((it+1))
k=$((k+1))
echo $k TCLAD${it} >> $results
done