Tons of output on every processor to log file causes problems

[ekluzek]

There's tons of output that's ending up in the cesm.log file for FATES. In one of Jackie's cases I saw 5.9GBytes of data being spewed to the cesm.log file. In this case her simulation hangs, and I think in part that's due to the contention in every processor trying to write tons of output to the same log file. It certainly is a problem that needs attention and will help out. I suppose some of this output is there for debugging, but it would be best to have the ability to turn it on or off with a namelist or some type of switch.

@jkshuman @rgknox @bandre-ucar @rosiealice

[jkshuman]

Erik and I updated two write statements to be debug only (Fire/SFMainMod "SF currentCohort%c_area" and Biogeochem/EDPhysio "using high bl cap"). This reduced the cesm log file from 5.9GBytes to 1.2GBytes. The run still got hung up at the same time point. @rgknox Any suggestions of what else to look over for this issue?

[rgknox]

@ekluzek, you are write (pun intended) that these statements should be locked behind a debug flag of some sort, thanks for bringing up this issue.

I think the strategy is to look through logs and grep the code for the offending statements. I noticed that "using high bl cap" is a very common statement. I can look through my logs too and search for rogue log writing.

[jkshuman]

@ekluzek and I are still working on this issue. Within fates/main/EDMainMod there is a flux in and out error "net_flux-dstock". The error occurs quite frequently. Can this be made less sensitive? @rgknox any thoughts on this?
This error occurs quite frequently:
total error: call index: 1 in: 2.17201904430866 out:
16: 4.015798410291849E-002 net: 2.13186106020574 dstock:
16: 2.14862956004862 error=net_flux-dstock: 1.676849984287898E-002
16: biomass,litter,seeds 318.015646960220 34.7350155452888
16: 6.069999692981771E-004
16: lat lon 70.0000000000000 260.000000000000

[rgknox]

@jkshuman : In #264 we added in the correction to the seed fluxes that was proposed by @rosiealice that never made it into master.

c61e633

While this has seemed to reduce the frequency of which imbalances are reported, we still have plenty of instances where carbon is not balanced. However, the amount of logging (file logging, not tree logging) may be more tolerable.

I think the underlying reason for the reporting is a serious issue that should be prioritized. The carbon budge should never break even at small tolerances. Hopefully in the near future I will be able to identify which processes are still generating the errors, as the budget usually closes now.

[rgknox]

Carbon imbalances now generate a fail, not a warning, as per #282. However, with #282 it is also expected to close within 1e-6 as relative error of total biomass.

We are still generating plenty of warnings on radiation budget though, so I will leave this issue up.

[rgknox]

A message is written to the land-log that fates dynamics is taking place. The good: it only write to the master proc and once per day. The bad: it writes the message for every column/site on the proc.

see line 98 of EDMainMod.F90