Minor differences running same model on different systems

[KevinShook]

Running gcc version on my desktop and on graham cluster.
Minor differences in some variables. Might just be due to different compiler versions, I guess.
Noticeable that the differences occurred near the end of the runs - perhaps due to accumulated error?
Forcings appear to be the same, so no errors there:

Very little difference in SWE on the cultivated HRU:

Small differences in depressional storage in wetland HRU:

Differences appear to be near the end of the run:

Same with basinflow:

The weirdest difference is that the programs executed for different lengths of time.
The dates in the .prj file were

######
1980 01 03
2018 12 29

The last line in the desktop output was
2018-12-29T00:00,0.01938389,0,0,0,0,0,0,0,0,258.9114,311.069,60,60,0,0.4854615,46.26881,46.2689

The last line in the graham output was
2018-12-28T23:00,0.01938389,0,0,0,0,0,0,0,0,94.62073,300.6478,45.94271,60,0,0.4665215,46.21881,46.2189

I was wondering if the time zone of graham might be a factor, as it is not the same as the .obs files.
I just checked and here's the graham info:

timedatectl
      Local time: Thu 2022-07-07 17:47:48 EDT

[jhs507]

The weirdest difference is that the programs executed for different lengths of time.

This might be related to this error which I have yet to solve. For your forcing files what format are the dates in?
#246

[jhs507]

As for the differences in output since they are small and found towards the end of a long run that does point towards accumulation of small errors.

I will investigate further when I have some time.

[KevinShook]

I'm uploading the .prj and .obs files
7120275310_RDRS_v21.obs.zip
7120275310_class_2_RDRS_PHyDAP.prj.zip

[lawfordp2017]

@KevinShook , @jhs507
I'd like to put in my two cents here as this bug is affecting me as well.
I believe Kevin's suggestion regarding the timezone of graham is correct. Also, Kevin mentioned that running his project on graham (allianace canada cluster) is very slow and I think this is the same issue (I've had the same experience). Kevin is using the Y M D h m format for specifying time in the observation file (may be important).

the 'Julian' call in the grow_crop macro is called at every time step for every hru, and performs a date conversion. There are also other places where date conversion is called. But when converting dates between unix timestamp format and Y M D format, one direction assumes UTC and the other direction assumes local time. When the local time call 'mktime' is used, the timezone file on disk is opened. On graham this file is stored on a slow network drive, so every call for every hru and every time step is very slow. Also, graham is located in Ontario, so local timezone is EST/EDT. Dealing with daylight savings I think means that reading the observation data file becomes unreliable.

I'll post a suggested correction in #379, and a stack trace that hints at the problem below.

Kevin, you may want to mention this issue regarding /cvmfs/soft.computecanada.ca/gentoo/2020/etc/localtime to the alliance canada group in case they think they could move this file to a faster local storage.

[lawfordp2017]

Program received signal SIGINT, Interrupt.
__GI___xstat (vers=vers@entry=1, 
    name=name@entry=0x2aaaab094ba8 "/cvmfs/soft.computecanada.ca/gentoo/2020/etc/localtime", 
    buf=buf@entry=0x7fffffff76a0) at ../sysdeps/unix/sysv/linux/wordsize-64/xstat.c:35
35          return INLINE_SYSCALL (stat, 2, name, buf);
(gdb) bt
#0  __GI___xstat (vers=vers@entry=1, 
    name=name@entry=0x2aaaab094ba8 "/cvmfs/soft.computecanada.ca/gentoo/2020/etc/localtime", 
    buf=buf@entry=0x7fffffff76a0) at ../sysdeps/unix/sysv/linux/wordsize-64/xstat.c:35
#1  0x00002aaaaafc8a41 in __tzfile_read (
    file=file@entry=0x2aaaab094ba8 "/cvmfs/soft.computecanada.ca/gentoo/2020/etc/localtime", extra=extra@entry=0, 
    extrap=extrap@entry=0x0) at tzfile.c:154
#2  0x00002aaaaafc85fd in tzset_internal (always=always@entry=1) at tzset.c:405
#3  0x00002aaaaafc8706 in __tzset () at tzset.c:551
#4  0x00002aaaaafc7549 in __GI_mktime (tp=0x7fffffff77c0) at mktime.c:529
#5  0x000000000056bbb4 in StandardConverterUtility::Calculate_TdateTime_Offset ()
    at /home/lawfordp/scratch/crhmcode_official/crhmcode/crhmcode/src/core/StandardConverterUtility.cpp:83
--Type <RET> for more, q to quit, c to continue without paging--
#6  0x000000000056bcb1 in StandardConverterUtility::EncodeDateTime (Year=1979, Month=12, Day=31, hour=0, min=0)
    at /home/lawfordp/scratch/crhmcode_official/crhmcode/crhmcode/src/core/StandardConverterUtility.cpp:107
#7  0x0000000000470ac0 in Julian (when=...)
    at /home/lawfordp/scratch/crhmcode_official/crhmcode/crhmcode/src/core/ClassCRHM.cpp:325
#8  0x0000000000487898 in Fjulian ()
    at /home/lawfordp/scratch/crhmcode_official/crhmcode/crhmcode/src/core/ClassMacro.cpp:25
#9  0x000000000056f3e1 in VarCHRM::get (this=0xc292e0)
    at /home/lawfordp/scratch/crhmcode_official/crhmcode/crhmcode/src/core/VarCHRM.cpp:98
#10 0x00000000004946f7 in execVget::operator() (this=0xcd2110)
    at /home/lawfordp/scratch/crhmcode_official/crhmcode/crhmcode/src/core/exec/execVget.h:43
#11 0x000000000048d07f in ClassMacro::run (this=0xc2b730)

[KevinShook]

Very nice work Peter!
Thanks

[jhs507]

This certainly explains the issues. I will review your solution on Monday Peter. This would have taken me quite a while to find if I had found it at all.

[lawfordp2017]

Thanks, along with Kevin's clues I had some experience from my own efforts to port Tom's Borland code to gcc previously. Time conversion was one of the biggest headaches.
I think the EncodeDateTime/DecodeDateTime routines may still be incorrect. If you look at the stack trace above it mentions EncodeDateTime (Year=1979, Month=12, Day=31, hour=0, min=0) but Kevin's output begins at 1980-01-03 so why is it referencing a date outside the analysis period? I haven't looked into this, but if you have time on monday it would be interesting to check.