Monthly diversion records are different between HydroBase and web services

[smalers]

A comparison of HydroBase and REST web services for an automated test shows a difference in output. I have opened a dialogue with DWR staff to figure it out. The image below illustrates the issue. Note that the local database does not provide observation flag for monthly values whereas rest web services does provide observation flag.

[smalers]

Web services do not return records for months without diversion records and therefore TSTool uses a null value. To resolve this issue, I implemented carry-forward logic on monthly diversion time series for web services. The following is an example of the result, with database on the left and web service on the right. Note that the flag is returned by web services but not database query and "c" is used to indicate TSTool carry forward.

See 1965-10. The problem is that the logic only fills forward zero values, as per previous instructions from DWR. Since the last monthly value is non-zero, the logic does not fill a zero. See the daily time series below (HydroBase on left, web services on the right), which show that the zero value is recorded on 1965-09-13.

Questions are therefore:

1. Is it valid to use the carry forward logic on monthly time series as a start? This could be a first pass and then deal with the following. I'm assuming it is OK because the monthly values or volume and not average flow so there is not an issue of needing a certain number of days to compute the month.
2. The fill carry forward logic is applied to time series individually. It is easy to understand for diversion total. However, it is also applied for water class time series. Is it OK to ignore all other time series at the structure when determining that a zero value has been reached? For example, what if a structure has 10 classes. They all need a zero value at the end of the year, correct?
3. When the situation like the above arises, what is the best way to handle:
   1. When a monthly diversion record time series is requested, also query the matching daily time series when needed to fill such values? This is a performance hit. This is the default approach to take unless another option is available.
   2. Can the first day carried and last day carried be used equivalent to the above? This may be faster than reading the daily time series. I don't see a way to get this from diversion record services. The problem is this will only impact the end of the year and intervening months may also go to zero, so probably not an option.
   3. Other option?
4. A similar issue occurs at the beginning of year but the algorithm does not require an initial zero record value in the year as per DWR instruction.

[smalers]

I implemented logic so that if a monthly time series has a missing value, the daily diversion records for the previous month are examined to see if the last recorded daily value is a zero. If so, then the missing monthly value is filled with zero. The result is as follows. I'm going to implement some tests and call it good unless there is feedback from the State that this is incorrect.

[smalers]

TSTool 14.5.1 contains updates that resolve this issue. Keep this issue open until feedback from the State is received, but it should be done. See the ReadColoradoHydroBaseRest documentation.

[smalers]

This release has been out for a while. The State has been using. I'm going to close this issue.