Test cases using ifort produce NaNs unless compiled without optimization

[rcwxok]

Version of MPAS:
2.0. Git repository pulled within past day.

Description of issue:
Log output shows NaNs after the first timestep:

Begin timestep 0000-01-01_00:00:03
...
global min, max w NaN NaN

Plotted fields at initial time look reasonable, but subsequent times fail to plot.

Program continues to run after NaNs are printed.

If compiled with DEBUG=true, the NaNs go away. Did not check netCDF.

Platform used (System Specs, as well as name, and operating system if possible):
Head node: RHEL 6.0, 16 x Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 16 GB RAM
Compute node: CentOS 6.3, 8 x Intel(R) Xeon(R) CPU X5560 @ 2.80GHz, 24 GB RAM

Compiler used (With version):
mpiifort/ifort 13.0.1
Had to modify Makefile, under ifort, to use mpiifort rather than mpif90,
and mpiicc rather than mpicc.

NetCDF version used:
4.1.3

Parallel NetCDF version used:
1.4.1

Parallel I/O version used:
1.7.1

Number of processors used:
Tried running separately on head and compute nodes.
Tried running witout MPI, or with 1 or 8 processors specified.

Test case used (if applicable):
Tried both supercell and jw_baroclinic_wave.

Is this issue reproducible in debug mode (make ... DEBUG=TRUE):
No. NaNs are not present in the log output when compiled with DEBUG=true, or with -O0.

If possible, please provide a backtrace for this issue:

[mgduda]

@rcwxok Apologies for not addressing this issue years ago! Since this issue is now 3 years old, I'll go ahead and close it; but, if you're still finding model failures with the most recent MPAS code, please feel free to re-open this issue or to create a new issue.