Save all SOILWAT2 output within the STEPWAT2 framework

[kpalmqui]

Each year that STEPWAT2 runs, we would like to save all SOILWAT2 output variables that are available. We would like the functionality to allow the user to select the timeframes for that output (daily, weekly, monthly, yearly), perhaps utilizing the outsetup_v31.in file. This will require that all output available within SOILWAT2 is passed into STEPPE and saved in memory until it is outputted.

One way to do this would be to save all of the daily SOILWAT2 output to a single .csv file, where the first column is Year, the second is DOY, and the subsequent columns represent all of the daily variables as separate columns. There would be similar files for the weekly, monthly, and yearly output. It would be fantastic to also have the user be able to specify whether they wanted output for individual iterations (e.g. daily information for iteration 1) or whether they wanted output averaged across iterations (daily values for each year that represents the mean across iterations).

[dschlaep]

One way to do this would be to save all of the daily SOILWAT2 output to a single .csv file, where the first column is Year, the second is DOY, and the subsequent columns represent all of the daily variables as separate columns

This behavior is almost identical to what SOILWAT2 is doing when run stand-alone, i.e, write out the output to files on disk (in tab- or comma-delimited format).

It seems to be that resolving this issue is mostly about adjusting the 'STEPWAT' define flags in the SW_Output.c file of SOILWAT2?

[kpalmqui]

Thank you Daniel for the comment. Brenden is looking to altering the STEPWAT define flags to pass the SOILWAT2 variables into the STEPPE side of STEPWAT2. This will be critical for overhauling resource partitioning. The other issue is getting SOILWAT2 output saved to disk. My understanding is that in SOILWAT2 standalone, separate files are written for each SOILWAT2 output variable (24) for each time period that is specified (daily, weekly, monthly, yearly) for a total of anywhere between 24 and 96 files, depending on the time periods specified. I was sort of hoping we might have a single file with all of the daily variables and a separate file with all of the yearly variables for a total of 2 files when timeperiods = daily and yearly (rather than 48). But perhaps this is already possible when SOILWAT2 standalone and I just don't know about that functionality?

[BrendenBe1]

Currently I am working on getting the values passed back to STEPPE, which includes adjusting the STEPWAT flags and adding code to structures in STEPWAT to store the values in, and then storing those values in SW_Output.c.

It seems that the code executed only when STEPWAT is defined inhibits certain values from being calculated, such as daily, weekly, and monthly PPT values. STEPWAT only provides values for yearly PPT.

The daily, weekly, and monthly values are calculated for SOILWAT2 standalone and when RSOILWAT is defined, so I am trying to figure out how to get STEPWAT to calculate these values so I can then pass them to STEPPE.

[dschlaep]

hoping we might have a single file with all of the daily variables and a separate file with all of the yearly variables for a total of 2 files when timeperiods = daily and yearly (rather than 48). But perhaps this is already possible when SOILWAT2 standalone and I just don't know about that functionality?

You are correct that SOILWAT2 standalone produces an output file separately for each output type and time period. However, we have no code that relies on this functionality and as you say it generates quickly tons of files -- which is poor behavior.

I vote that we change the SOILWAT2 code to produce one output file for each time period. This can then be both used for the standalone and the STEPWAT2 versions.

[kpalmqui]

Sounds good Daniel- I guess it does make sense to edit the SOILWAT2 code to restructure the output rather than introduce new code in the STEPPE side to create this output.

[dschlaep]

It seems that the code executed only when STEPWAT is defined inhibits certain values from being calculated, such as daily, weekly, and monthly PPT values. STEPWAT only provides values for yearly PPT.

Isn't this hard-coded by STEPWAT2? See function sxw.c/_write_sw_outin() which rewrites the output.in file for SOILWAT to default values (except for transpiration. This function is called by SXW_Init.

[dschlaep]

I guess it does make sense to edit the SOILWAT2 code to restructure the output rather than introduce new code in the STEPPE side to create this output.

Including the option to calculate [time period]-aggregated output! Then the rSFSW2 wrapper can also simply import SOILWAT2 daily aggregated values instead of calculating them slowly in R.

[BrendenBe1]

sxw.c/_write_sw_outin() is rewriting the output.in file but it looks like its all yearly values, nothing for monthly or other time periods. Am I missing an output file or function where STEPPE gets daily and monthly values?

I will start working on changing the output files produced by SOILWAT2.

[dschlaep]

If you change what sxw.c/_write_sw_outin() is writing to output.in, e.g., add some monthly time periods, then I expect that STEPPE is receiving those monthly data (unless SOILWAT2/output.c has the STEPWAT flag such that it cannot).

…

On Jun 20, 2017, at 16:30, Brenden Bernal ***@***.***> wrote: sxw.c/_write_sw_outin() is rewriting the output.in file but it looks like its all yearly values, nothing for monthly or other time periods. Am I missing an output file or function where STEPPE gets daily and monthly values? I will start working on changing the output files produced by SOILWAT2. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEAp2_Kq_NcvCM45xylUtRQlhiy8X99Bks5sGCvNgaJpZM4N7qHF>.

[kpalmqui]

Solution:
1)Restructure SOILWAT2 output in 4 csv files (daily, weekly, monthly, and yearly) that contain all variables (see issue 21 in SOILWAT2).
2)Add code that will enable those output files to be written when SOILWAT2 is running within the STEPWAT2 framework.

[BrendenBe1]

@dschlaep I was wondering how we want to format the CSV's for the SOILWAT output? We were initially planning on having 4 files (1 for each timestep) and in those files having all the variables. I was talking to Kyle about this and she had the idea to have 8 files instead, 2 sets for each timestep, where one of the sets are for variables without layers and the other set are for soil variables that contain data for multiple layers. This seems like it could be a bit easier to implement but I wanted to get some more input before implementing anything.

[dschlaep]

I wouldn't restrict our output files to be of the 'comma-separated-value' (csv) type. We have currently the 'OUTSEP' implemented and, thus, we generage deliminated files (https://en.wikipedia.org/wiki/Delimiter-separated_values). If OUTSEP = , then the outputs are (non-Germanic) csv-files (or at least pretty close). If OUTSEP = \t, then the outputs are tab-deliminated files etc.

[dschlaep]

I agree that it may be a good idea to separate output for soil-array variables from other variables. Each SOILWAT2 run/call may have a different number of soil layers, thus each soil-array variable output has a different number of columns.

[kpalmqui]

Agreed. We should retain the functionality for different file output types (tab-deliminated, csv).
Let's proceed with separate output files for soil variables with layers and other variables. If the user wants all time periods and all SOILWAT2 output, this will result in 8 files. Let's move conservation regarding this thread to the SOILWAT2 issue 21

[BrendenBe1]

alright, then we will go with 2 files per timestep.

[BrendenBe1]

For the output files when running STEPPE would we want them to always be created or would we want them to only be created if the user specifies it? Right now there is a parameter you can add when running STEPWAT (-o) that outputs the SOILWAT values. Do we want to keep this functionality and only have the output when this -o flag is specified or do we want to make it so that files are output every time, no matter the flags?

update: we are going with only having the output when the -o flag is present. I will be changing the functionality of the -o flag since the old way it functioned is not suitable

[kpalmqui]

Update: when SOILWAT2 runs within STEPWAT2, we will give the user the option to save separate output files for each iterations(this would potentially result in 8*n, where n=number of iterations) and the option to have an a single file that has mean and sd values for each variable across iterations. All of the raw daily SOILWAT2 output for each year will be saved to memory for each iteration. Once that iteration is complete, the output will be written to disk (if the user wants it) and appended to the previous iteration's output to get running mean and sd values across iterations. The output associated with that particular iteration will then be removed from memory.

[BrendenBe1]

The functionality for saving output from SOILWAT in STEPPE is complete. There are two possible input flags (-o and -i) that allow the user to accomplish this. The -o flag saves the average value over all the iterations and stores that in the output files. The -i flag saves the SOILWAT output for every iteration and saves each iteration in its own file with the naming convention "sw2_timeperiod_iteration" (ex: sw2_daily_1). These flags can be used alone or they can be used together depending on the users needs.