Metadata fix #45

arezoorn · 2018-05-12T05:41:23Z

I have added the min_valid and max_valid as attributes to the output files from NWM output routines. To do so, I added KHOUR to the nlst_rt, since the simulation duration was not accessible in other routines.

I also added one global attribute called model_total_valid_times which is basically the total number of outputs for a given file format (for example CHRTOUT) during the simulation.

I also resolved NCAR/wrf_hydro_nwm-issues#139 . The NoahMP time stamp was wrong (when the time stamp between LSM and Hydro differed). I added ITIME which is the LSM time step to the output routine and used that instead of out_counts which is Hydro time step. I also ran the model with LSM only, checked that the time is correct.

There was also a non reported issue/bug that is fixed with this change, the time variable was wrong if HYDRO was active and the LSM output timestep was not a multiplicative of the Hydro output timestep. For example if user was setting the LSM output timestep to 36000 (10hours) and Hydro output timestep to 180 (3 hours) the time variable in the LSM output was wrong. This was not a reported bug, since in our operational runs and our tests, we usually define the LSM output timestep a miltuplicative factor of the LSM output.

…the output files

Since the NWM routine was using the timestep of the hydro model, the LSM outputs time stamp was not right in cases that the time steps of LSM and HYDRO differs. I added ITIME to the routine, to be able to calculate the correct time in the output routines.

…ribute calcualted the time in the LSM output routine based on the LSM time step and not the HYDRO timestep. Also fixed the valid_max to present the time of the last exising file and not the end of the simulation.

wrf-hydro-nwm-bot · 2018-05-12T05:42:12Z

Waiting for private pull request queue to clear.

…del outputs

wrf-hydro-nwm-bot · 2018-05-14T20:01:27Z

Authentication required - Linked to private pull request #197, link: https://github.com/NCAR/wrf_hydro_nwm/pull/197

wrf-hydro-nwm-bot · 2018-05-14T20:01:28Z

Pull request submitted to private repo. Waiting for reviewer(s) to approve.

jmccreight · 2018-05-17T12:15:16Z

There is no comment that the PR passes any kind of regression test.
No test results have been placed in-line.
Can someone please comment on why the checks are failing? If the report is hard to read, then state that. We can make it better. Right now this is completely being ignored. To me, it looks like it's probably all metadata differences but I think we need to clearly separate metadata and value differences in the evaulation and I would open an issue about that.

jmccreight · 2018-05-17T12:50:02Z

Given that the restart files all check and the CHRTOUT and CHANOBS files fail, I'm going to assume those differences are meta-data only.
I filed #50 to request separation of data and metadata checks for output files.

jmccreight · 2018-05-17T12:51:53Z

trunk/NDHMS/Land_models/NoahMP/IO_code/module_NoahMP_hrldas_driver.F

@@ -1518,7 +1523,7 @@ subroutine land_driver_ini(NTIME_out,wrfits,wrfite,wrfjts,wrfjte)
       print*, "t0OutputFlag: ", t0OutputFlag  
 #endif        
 !#ifdef HYDRO_REALTIME
-   if(t0OutputFlag .eq. 1) call ldas_output()
+   if(t0OutputFlag .eq. 1) call ldas_output(0)


I love how this is so opaque. What was the assumed value when nothing was passed, what does 0 represent. Is there no good variable that can be used?

hmm, that is the initialization timestep, actually I could use ITIME as I used it in the other routines. I will check why I did not use it, and it possible I would replace it with ITIME.

jmccreight · 2018-05-17T12:53:34Z

trunk/NDHMS/Land_models/NoahMP/IO_code/module_NoahMP_hrldas_driver.F

-subroutine  ldas_output()
+subroutine  ldas_output(itime)
+integer, intent(in)  :: itime ! time step of the LSM, had to explicitly declare it since 
+                              ! it did not exist in the land_driver_ini 


👍 The comment is good, but notice how this kind of comment goes stale "had to explicitly..."
I love keywords. I wish they were slightly easier to use in fortran.

jmccreight · 2018-05-17T12:57:14Z

trunk/NDHMS/Routing/module_NWM_io.F

+
+   ! calculate total_valid_time
+   fileMeta%totalValidTime = int(nlst_rt(1)%khour * 60 / nlst_rt(1)%out_dt)  ! # number of valid time (#of output files)
+


I get why these time calculations are here. I would just like to point out that these kinds of calculations should go in a module_time.F and the results should be requested here (not calculated).

jmccreight · 2018-05-17T13:08:36Z

This PR needs a bit more explicit description of what the changes were. @arezoorn please correct if any of this is wrong:

"I have added the min_valid and max_valid as attributes to the output files from NWM output routines. " Specifically, in module_NWM_io_dict:

integer :: timeValidMin ! the minimum time each configuration can have, time of the first output file
integer :: timeValidMax ! the maximum time each configuration can have, time of the last output file
integer :: totalValidTime ! # number of valid time (#of output files)

These are now metadata attributes in the CHANOBS and CHRTOUT (same in both) with the following names:

iret = nf90_put_att(ftn,timeId,'valid_min',fileMeta%timeValidMin)
iret = nf90_put_att(ftn,timeId,'valid_max',fileMeta%timeValidMax)
iret = nf90_put_att(ftn,NF90_GLOBAL,"model_total_valid_times",fileMeta%totalValidTime)

jmccreight

See my comments. I'm approving with a lot of noise and whining. 🎊

kafitzgerald · 2018-05-17T14:15:13Z

Definitely agree that it would be nice to have some additional info regarding test failures in the log.

Just ran the tests again offline to confirm (file diffs below) and as expected we just see differences in the attributes.

CHRTOUT

DIFFER : NAME OF GLOBAL ATTRIBUTE : model_total_valid_times : GLOBAL ATTRIBUTE DOESN'T EXIST IN NWM/run_reference/201108281400.CHRTOUT_DOMAIN1
DIFFER : VARIABLE "time" IS MISSING ATTRIBUTE WITH NAME "valid_min" IN FILE "NWM/run_reference/201108281400.CHRTOUT_DOMAIN1"
DIFFER : VARIABLE "time" IS MISSING ATTRIBUTE WITH NAME "valid_max" IN FILE "NWM/run_reference/201108281400.CHRTOUT_DOMAIN1"
DIFFER : NUMBER OF ATTRIBUTES : VARIABLE : time : 3 <> 5```

CHANOBS
```DIFFER : NUMBER OF GLOBAL ATTRIBUTES : 16 <> 17
DIFFER : NAME OF GLOBAL ATTRIBUTE : model_total_valid_times : GLOBAL ATTRIBUTE DOESN'T EXIST IN NWM/run_reference/201108281400.CHANOBS_DOMAIN1
DIFFER : VARIABLE "time" IS MISSING ATTRIBUTE WITH NAME "valid_min" IN FILE "NWM/run_reference/201108281400.CHANOBS_DOMAIN1"
DIFFER : VARIABLE "time" IS MISSING ATTRIBUTE WITH NAME "valid_max" IN FILE "NWM/run_reference/201108281400.CHANOBS_DOMAIN1"
DIFFER : NUMBER OF ATTRIBUTES : VARIABLE : time : 3 <> 5```

jmccreight · 2018-05-17T14:21:29Z

@arezoorn
An issue needs referenced. If one does not exist, please create one.

jmccreight · 2018-05-17T14:22:05Z

Katelyn, thanks for doing that. I know it's currently a pain. should be less painful soon!!

Added model configuration to compile options json

arezoorn added 6 commits May 10, 2018 01:24

added KHOUR to the type

567311f

assigned the KHOUR to nlst_rt after reading it from namelist

5386ad6

adding min_valid and max_valid as the attributes of time variable to …

d5aa40a

…the output files

adding min_valid and max_valid as the attributes of time variable to …

534a8ec

…the output files

kafitzgerald requested review from logankarsten and kafitzgerald May 12, 2018 16:49

arezoorn added 2 commits May 12, 2018 11:16

adding the total_valid_time to the LDASOUT

660f2d7

adding the model_total_valid_times to the global attributes of the mo…

6754e43

…del outputs

kafitzgerald removed their request for review May 14, 2018 20:00

kafitzgerald assigned jmccreight and unassigned jmccreight May 15, 2018

kafitzgerald requested a review from jmccreight May 15, 2018 17:09

jmccreight mentioned this pull request May 17, 2018

Testing: checks for data and metadata should be separated #50

Closed

jmccreight reviewed May 17, 2018

View reviewed changes

jmccreight approved these changes May 17, 2018

View reviewed changes

logankarsten approved these changes May 17, 2018

View reviewed changes

kafitzgerald merged commit 687bacd into NCAR:master May 17, 2018

arezoorn deleted the metadata_fix branch May 21, 2018 00:44

jmccreight pushed a commit to jmccreight/wrf_hydro_nwm_public that referenced this pull request Nov 13, 2018

Merge pull request NCAR#45 from jmills-ncar/master

9b35d97

Added model configuration to compile options json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metadata fix #45

Metadata fix #45

arezoorn commented May 12, 2018 •

edited

wrf-hydro-nwm-bot commented May 12, 2018

wrf-hydro-nwm-bot commented May 14, 2018

wrf-hydro-nwm-bot commented May 14, 2018

jmccreight commented May 17, 2018 •

edited

jmccreight commented May 17, 2018

jmccreight May 17, 2018

arezoorn May 20, 2018

jmccreight May 17, 2018

jmccreight May 17, 2018

jmccreight May 17, 2018

jmccreight commented May 17, 2018

jmccreight left a comment

kafitzgerald commented May 17, 2018

jmccreight commented May 17, 2018

jmccreight commented May 17, 2018


		! calculate total_valid_time
		fileMeta%totalValidTime = int(nlst_rt(1)%khour * 60 / nlst_rt(1)%out_dt) ! # number of valid time (#of output files)

Metadata fix #45

Metadata fix #45

Conversation

arezoorn commented May 12, 2018 • edited

wrf-hydro-nwm-bot commented May 12, 2018

wrf-hydro-nwm-bot commented May 14, 2018

wrf-hydro-nwm-bot commented May 14, 2018

jmccreight commented May 17, 2018 • edited

jmccreight commented May 17, 2018

jmccreight May 17, 2018

Choose a reason for hiding this comment

arezoorn May 20, 2018

Choose a reason for hiding this comment

jmccreight May 17, 2018

Choose a reason for hiding this comment

jmccreight May 17, 2018

Choose a reason for hiding this comment

jmccreight May 17, 2018

Choose a reason for hiding this comment

jmccreight commented May 17, 2018

jmccreight left a comment

Choose a reason for hiding this comment

kafitzgerald commented May 17, 2018

jmccreight commented May 17, 2018

jmccreight commented May 17, 2018

arezoorn commented May 12, 2018 •

edited

jmccreight commented May 17, 2018 •

edited