Njeffery debug zbgc3 d #207

njeffery · 2018-10-12T23:49:09Z

@eclare108213 : zbgc now runs and passes the gx1 test. I needed to increase the queue time to 2 hours.

Corrected blocks in ice_forcing_bgc.F90
Moved bgc initializations in CICE_InitMod.
Uncommented some parameter definitions related to bgc in init_zbgc.
Completed test successfully with flags -g gx1 -m wolf -e intel -s bgcgx1 -p 32x1

Adds missing namelist fields. Also added gx1 tests for zbgc (bgc,bgcgx1) and skl (bgcskl,sklgx1) Required nutrient forcing filenames are hard coded (nitrate_climatologyWOA_gx1v6f.nc and silicate_climatologyWOA_gx1v6f.nc) Namelist field bgc_data_dir needs to be properly specified. Note *gx1 zbgc test is really slow and fails on wolf*

Corrected blocks in ice_forcing_bgc.F90 Moved bgc initializations in CICE_InitMod. Uncommented some necessary parameter definitions related to bgc in init_zbgc.

apcraig · 2018-10-13T13:00:08Z

travis is failing because ftp is down for semi-annual maintenance. I will trigger the travis build again once the ftp site is back up.

apcraig

Just a couple quick comments. It looks like there are some indentation issues. Can you check that you are not using tabs. Second, it looks like this needs to updated to master. Last week we merged the tracer cpps changes onto master and those are now namelist and not cpps, both cice.settings and ice_in have changed and that needs to be reflected in the set_env and set_nml files. Sorry about that. If you need any help, let me know.

Merge branch 'master' of https://github.com/CICE-Consortium/CICE into njeffery_debugZbgc3D

eclare108213 · 2018-10-17T17:35:09Z

@njeffery: nslyr=5 for the bgcISPOL and bgcNICE cases. Is that correct?

eclare108213 · 2018-10-17T18:05:50Z

I've updated the environment and namelist files for dynamic allocation changes in the code, and smoke-tested bgcgx1 and sklgx1 -- they pass, but I have not looked at actual results yet.
A number of namelist flags are redundant for the bgc tracers, e.g. I don't think we don't need both trzaero (on/off) and n_zaero (number of aerosols, 0=none could also mean off). I'll run a base_suite and continue working to simplify/clean up the code. @njeffery please do longer gx1 runs for both zbgc and skl and check results, as soon as you have a chance.

njeffery · 2018-10-17T20:25:37Z

Elizabeth, Snow layers? I think I've only run with 1. n

…

________________________________ From: Elizabeth Hunke <notifications@github.com> Sent: Wednesday, October 17, 2018 11:35:09 AM To: CICE-Consortium/CICE Cc: Jeffery, Nicole; Mention Subject: Re: [CICE-Consortium/CICE] Njeffery debug zbgc3 d (#207) @njeffery<https://github.com/njeffery>: nslyr=5 for the bgcISPOL and bgcNICE cases. Is that correct? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#207 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AH1K6mdxo90Lc_-Bpkmdy3bSknsdlWQlks5ul2pNgaJpZM4XaV11>.

eclare108213 · 2018-10-17T20:28:32Z

That's what I thought. I'll fix this.

eclare108213 · 2018-10-17T20:53:05Z

base_suite results are here:
https://github.com/CICE-Consortium/Test-Results/wiki/abd1d6613c.pinto.intel.181017.180843
The 2 pending tests are because we don't have 40 pes available to run. All of the other tests passed.

apcraig · 2018-10-18T00:17:39Z

Should I run a full test suite on conrad with 4 compilers? Is now a good time or is there still work to do? Just let me know if that is desired and when.

eclare108213 · 2018-10-18T01:07:48Z

This branch seems to be working, from a testing standpoint. I have not checked the actual test output to see if the tests are doing reasonable things, only that they pass the new bgc tests now available in base_suite.
@njeffery please run these yourself and look to see if the tests are reasonable. I suspect that the ISPOL and NICE tests are not running in the correct time period and I do not know what data those are using -- we are mainly interested in exercising the code here, but let's not do something that gives our users fits.
@apcraig I changed the way RUNLENGTH is set in the scripts. It was very unintuitive to me, and it wasn't documented anywhere that I could find. Now its value essentially corresponds to the number of wallclock hours requested.
@duvivier I added documentation for RUNLENGTH and also the available grids. Please make sure that I didn't screw it up!

eclare108213 · 2018-10-18T01:09:16Z

We might not want to keep all of these new bgc tests, especially since a few of them take a while to run.

eclare108213 · 2018-10-18T01:15:17Z

@apcraig I think it would be helpful to run the tests on conrad now, since this part of the code hasn't really been subject to that level of testing yet.
I still want (someone, maybe me) to clean up the bgc namelist flags, but that can be a separate PR.

apcraig · 2018-10-18T16:21:05Z

I tested on conrad and 4 compilers and everything is fine EXCEPT the new bgc cases. sklgx1 and bgcgx1 both fail. The main reason is the input data. The filenames are hardwired in the code and are not consistent with the data in the consortium,

../../cicecore/cicedynB/general/ice_forcing_bgc.F90: nit_file = trim(bgc_data_dir)//'nitrate_climatologyWOA_gx1v6f.nc'
../../cicecore/cicedynB/general/ice_forcing_bgc.F90: sil_file = trim(bgc_data_dir)//'silicate_climatologyWOA_gx1v6f.nc'

while the data files in the consortium tar file have _20150107 in the filename. These filenames have to be moved to namelist! Second, the bgc_data_dir is also hardwired to a path on some machine. This has to be generalized. I think the right change is to add

bgc_data_dir = 'ICE_MACHINE_INPUTDATA/CICE_data/forcing/gx1/WOA/MONTHLY/'

in set_nml.gx1 and to remove the bgc_data_dir setting in set_nml.sklgx1 and set_nml.bgcgx1.

Finally, I do not believe the sklgx1 and bgcgx1 need to be tied to the gx1 resolution. We should be able to move the data_dir and data filenames to set_nml.gx1 and the for the skl and bgc to NOT be grid dependent. I also prefer if the set_nml files be prepended with bgc. So something like

set_nml.bgcskl
set_nml.bgczbgc

or something like it would be good.

I can test again or even help with the implementation of these changes if you like. Just let me know.

njeffery · 2018-10-18T21:35:41Z

add fsal. Testing them both now.

…

________________________________ From: Elizabeth Hunke <notifications@github.com> Sent: Thursday, October 18, 2018 2:59:17 PM To: CICE-Consortium/CICE Cc: Jeffery, Nicole; Mention Subject: Re: [CICE-Consortium/CICE] Njeffery debug zbgc3 d (#207) Chatting with @njeffery<https://github.com/njeffery> -- She suspects that f_doc is not being passed properly. That could be causing initialization problems, seg faults, etc. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#207 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AH1K6qNAcNWHaB7hLWOYtNddq4r_-BHcks5umOulgaJpZM4XaV11>.

eclare108213 · 2018-10-19T01:30:09Z

I changed the options files for bgc. ISPOL and NICE are appropriate for testing Icepack but not CICE, so those options files have been removed. There is still an 'ISPOL' forcing option in the code but the Consortium does not have the necessary data files. I created bgcsklclim and bgczclim option files for use with the gx1 grid (the 'clim' option is only available for gx1) and set the bgc data_types to 'default' in the bgcskl and bgcz option files, for use with either gx3 or gx1.

The new tests in base_suite all pass for me, except the one which is commented out -- it does a comparison that is not BFB. Comparing the diagnostic output is interesting, maybe this provides some clues for what's going on with the roundrobin cases? or maybe not. I don't understand why the distribution type is different! Of course, I'm probably doing something stupid, assigning 4x1 and 'thread'......

 smoke          gx3     4x1        bgcz,thread        smoke_gx3_8x2_bgcz



 <   Processors:  total    =      4
 ---
 >   Processors:  total    =      8
 181c185
 <   Distribution type:            cartesian
 ---
 >   Distribution type:           roundrobin
 189c193
 <   max_blocks =                 4
 ---
 >   max_blocks =                10
 194,199c198,202
 <  ice_domain work_unit, max_work_unit =         3170          10
 <  ice_domain nocn =         4491       31691      261744
 <  ice_domain work_per_block =            3          11         106
 < (proc_decomposition)  Processors (X x Y) =    2 x    2
 <  ice: total number of blocks is          16
 <   Block size:  nx_block =     27
 ---
 >  ice_domain work_unit, max_work_unit =          739          10
 >  ice_domain nocn =            0        7385      261744
 >  ice_domain work_per_block =            0          11         475
 >  ice: total number of blocks is          79
 >   Block size:  nx_block =      7
 208,209d210
 <  min/max ULON:  0.000000000000000E+000   360.000005165255     
 <  min/max ULAT:  -76.1620403824018        89.9000004922619

Note that I attempted to fix the problem with the gx1 ocean forcing file in this PR but didn't test it.

duvivier · 2018-10-19T02:42:50Z

doc/source/user_guide/ug_case_settings.rst

+   " ", "0", "15 minutes (default)", " "
+   " ", "1", "59 minutes", " "
+   " ", "2", "2 hours", " "
+   " ", "other N$<$8", "N hours", " "


Should be:
" ", "other :math:2 < N < 8", "N hours", " "

The block of math text is set off with the ` on either side. Just to be clear. Same comment for below.

duvivier · 2018-10-19T02:43:22Z

doc/source/user_guide/ug_case_settings.rst

   "ICE_ACCOUNT", " ", "batch account number", "set by cice.setup, .cice_proj or by default"
   "ICE_QUEUE", " ", "batch queue name", "set by cice.setup or by default"
-   "ICE_THREADED", "true,false", "force threading in compile, will always compile threaded if NTHRDS is gt 1", "false"
+   "ICE_THREADED", "true, false", "force threading in compile, will always compile threaded if NTHRDS is gt 1", "false"


Should be:
"ICE_THREADED", "true, false", "force threading in compile, will always compile threaded if ICE_NTHRDS :math:> 1", "false"

duvivier

Just two minor errors in the user guide documentation. I tested these offline with my sphinx build and the suggestions I request fix the format.

duvivier · 2018-10-19T03:11:34Z

doc/source/user_guide/ug_case_settings.rst

   "ICE_ACCOUNT", " ", "batch account number", "set by cice.setup, .cice_proj or by default"
   "ICE_QUEUE", " ", "batch queue name", "set by cice.setup or by default"
-   "ICE_THREADED", "true,false", "force threading in compile, will always compile threaded if NTHRDS is gt 1", "false"
+   "ICE_THREADED", "true, false", "force threading in compile, will always compile threaded if ICE_NTHRDS :math: > 1", "false"


You'll have to add ` signs around the >1 (this didn't render before in my comment, sorry!)

duvivier · 2018-10-19T03:12:13Z

doc/source/user_guide/ug_case_settings.rst

+   " ", "0", "15 minutes (default)", " "
+   " ", "1", "59 minutes", " "
+   " ", "2", "2 hours", " "
+   " ", "other :math: 2 < N < 8", "N hours", " "


You'll have to add signs around the 2<N<8 (this didn't render before in my comment, sorry! If I do it in the comment it formats it instead of showing characters. See2<N<8`)

eclare108213 · 2018-10-19T03:15:53Z

trying again... is this better?

duvivier · 2018-10-19T03:23:31Z

Yes, that is perfect. :)

njeffery · 2018-10-23T17:07:00Z

@eclare108213 : I've added the missing bgc paramters. Ran a 10 week test with bgcz on the gx3 that passed.

apcraig · 2018-10-23T18:58:04Z

I just ran the bgc tests on conrad with 4 compilers, hash acbd01a at https://github.com/CICE-Consortium/Test-Results/wiki/cice_by_hash_forks. The results are better. The cray compiler fails all cases with errors like

lib-4961 : WARNING
Subscript 1 is out of range for dimension 1 for array
'aerosno(*,,)' at line 298 in file 'ice_step_mod.F90' with bounds 1:0.

lib-4961 : WARNING
Subscript 1 is out of range for dimension 1 for array
'aeron' at line 1069 in file 'icepack_shortwave.F90' with bounds 1:0.
(skl_biogeochemistry)initial sk_bgc < 0, nn,nbtrcr,cinit(nn) 15, 16, -2.99999999999999989E-2
(skl_biogeochemistry) cinit < c0
(icepack_warnings_setabort) T :file /p/home/apcraig/cice-consortium/cice.debugZbgc3D/icepack/columnphysics/icepack_algae.F90 :line 592
(icepack_warnings_aborted) ... (sklbio)
(icepack_warnings_aborted) ... (icepack_biogeochemistry)

(z_biogeochemistry)very large bgc value
(z_biogeochemistry)k,m,hbri,hbri_old,bio_tmp,biomat_cons(k,m),ocean_bio(m)
(z_biogeochemistry) 1, 15, 6.5519641720615418E-2, 7.14807665163326444E-2, 1.97942537215752571E+198, 1.29693004287869
203E+197, 2.45710724577836671E+198
(z_biogeochemistry)react(k,m),iphin_N(k),biomat_brine(k,m)
(z_biogeochemistry) -5.76394390422860599E+193, 0.46685667284947335, 4.23995713595056273E+198
(z_biogeochemistry) very large bgc value
(icepack_warnings_setabort) T :file /p/home/apcraig/cice-consortium/cice.debugZbgc3D/icepack/columnphysics/icepack_alga
e.F90 :line 1341
(z_biogeochemistry)trcrn(nt_zbgc_frac+m-1): 1.
(z_biogeochemistry)in_init_cons(k,m): 1.40508727472858092E+197
(z_biogeochemistry)trcrn(bio_index(m) + k-1)
(z_biogeochemistry) 1.96568579662269342E+198
(z_biogeochemistry)Category,m: 1, 15

The pgi cases fails with an error message,

0: Null pointer for a3db (/p/home/apcraig/cice-consortium/cice.debugZbgc3D/cicecore/cicedynB/analysis/ice_history_bgc.F90:
2548)

The intel case fails with an undefined seg fault,

forrtl: severe (174): SIGSEGV, segmentation fault occurred

The gnu case fails with a seg fault in what seems to be a threaded region, although not totally sure,

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x60816f in ???
at /usr/src/packages/BUILD/glibc-2.11.3/nptl/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
#1 0x593fd0 in ???
#2 0x594936 in ???
#3 0xb7bddd in gomp_thread_start
at ../../../cray-gcc-6.3.0-201701050407.93fe37becc347/libgomp/team.c:119
#4 0xb4e3b5 in start_thread
at /usr/src/packages/BUILD/glibc-2.11.3/nptl/pthread_create.c:301
#5 0xbe2128 in ???
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#0 0x60816f in ???
at /usr/src/packages/BUILD/glibc-2.11.3/nptl/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
#1 0x593fd0 in ???
#2 0x594936 in ???
#3 0xb7bddd in gomp_thread_start
at ../../../cray-gcc-6.3.0-201701050407.93fe37becc347/libgomp/team.c:119
#4 0xb4e3b5 in start_thread
at /usr/src/packages/BUILD/glibc-2.11.3/nptl/pthread_create.c:301
#5 0xbe2128 in ???
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
_pmiu_daemon(SIGCHLD): [NID 01753] [c9-0c0s6n1] [Tue Oct 23 17:48:36 2018] PE RANK 0 exit signal Segmentation fault

The first step might be to fix the cray errors as those are clearly pointing at something concrete. Maybe the others fix too? At some point, we can also turn on debugging and test with/without openmp/mpi for a few cases to see if that provides any extra clues, although I'm not sure we're quite there yet.

eclare108213 · 2018-10-23T23:36:10Z

base_suite test results are here:
https://github.com/CICE-Consortium/Test-Results/wiki/acbd01a5c7.pinto.intel.181023.195142
The 2 failing tests are expected on pinto; otherwise everything passes.
I have a test suite sitting in the queue with the code at hash abcd01a, to see if the debug runs throw any errors as on conrad. If not, I'm not sure how to debug those issues. Could it be an optimization issue?

eclare108213 · 2018-10-24T02:37:50Z

My abcd01a tests also compiled and ran as advertised. @apcraig your errors look like a configuration problem to me, like maybe the namelist isn't getting read correctly so array allocations aren't being made. This could conceivably happen due to our having multiple, redundant namelist options for tracers, although the code checks at initialization ought to catch inconsistencies.
To start, could you post or email to me the files from at least one of the failing runs, which have the config information in them? ice_in, case settings, diagnostic log file if the case started, etc. I'll compare with my versions.

Update history field control, place control logicals inside init and accum subroutines for consistency. Fix some indent issues in history subroutines. Comment out OMP loop that was causing problems on conrad_cray. Fix intent inout arguments in ice_forcing_bgc.F90, init_bgc_data for fed1, fep1. Modify ice_arrays_column so more variables are dynamically allocated using icepack query routines.

… into njeffery_debugZbgc3D

eclare108213

base_suite passes on pinto: https://github.com/CICE-Consortium/Test-Results/wiki/9f47dacbd0.pinto.intel.181026.131608
I did not check whether answers change compared with master. If they do not, then I'm in favor of merging this and then moving forward with additional PRs as needed.

apcraig · 2018-10-26T15:33:23Z

I have updated the test results from last night. 4 tests are still in the queue for some strange reason, but otherwise, everything is running, passing, and bit-for-bit. hash 9f47dac, https://github.com/CICE-Consortium/Test-Results/wiki/cice_by_hash_forks.

I will merge the PR now.

njeffery added 2 commits October 10, 2018 15:47

Debugged zbgc on gx1 for bgcgx1 test.

44474a2

Corrected blocks in ice_forcing_bgc.F90 Moved bgc initializations in CICE_InitMod. Uncommented some necessary parameter definitions related to bgc in init_zbgc.

apcraig reviewed Oct 13, 2018

View reviewed changes

apcraig mentioned this pull request Oct 13, 2018

Corrects icepack initialization for bgc. #175

Closed

Updating debugZbgc3D branch to CICE master

7535243

Merge branch 'master' of https://github.com/CICE-Consortium/CICE into njeffery_debugZbgc3D

env and nml options files for dyn alloc

abd1d66

set nslyr=1

d0d46ed

apcraig requested a review from eclare108213 October 18, 2018 00:16

add bgc tests to base_suite

dfebc28

eclare108213 requested a review from duvivier October 18, 2018 00:50

eclare108213 self-assigned this Oct 18, 2018

eclare108213 added Type: Bug Testing Documentation Scripts labels Oct 18, 2018

eclare108213 added this to To do in CICE v6 release via automation Oct 18, 2018

alignment in namelists

de13aee

eclare108213 mentioned this pull request Oct 18, 2018

document case settings #210

Closed

data dir and filenames for bgc

1c6dacb

eclare108213 added 2 commits October 18, 2018 18:58

expanding bgc options

8599bbc

remove ISPOL and NICE bgc options for CICE

100d6c9

duvivier reviewed Oct 19, 2018

View reviewed changes

duvivier requested changes Oct 19, 2018

View reviewed changes

correct doc

cdf2081

duvivier reviewed Oct 19, 2018

View reviewed changes

CICE v6 release automation moved this from To do to In progress Oct 19, 2018

duvivier previously approved these changes Oct 19, 2018

View reviewed changes

add apostrophes to doc

31d18de

eclare108213 dismissed duvivier’s stale review via 31d18de October 19, 2018 03:14

duvivier mentioned this pull request Oct 19, 2018

Nml settings #208

Merged

Add two missing parameters to bgc initialization.

acbd01a

resolve conflicts

1c0012d

apcraig added 2 commits October 26, 2018 05:37

Merge branch 'njeffery_debugZbgc3D' of https://github.com/njeffery/cice…

9f47dac

… into njeffery_debugZbgc3D

eclare108213 approved these changes Oct 26, 2018

View reviewed changes

apcraig approved these changes Oct 26, 2018

View reviewed changes

apcraig merged commit e6b9fee into CICE-Consortium:master Oct 26, 2018

CICE v6 release automation moved this from In progress to Done Oct 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Njeffery debug zbgc3 d #207

Njeffery debug zbgc3 d #207

njeffery commented Oct 12, 2018

apcraig commented Oct 13, 2018

apcraig left a comment

eclare108213 commented Oct 17, 2018

eclare108213 commented Oct 17, 2018

njeffery commented Oct 17, 2018 via email

eclare108213 commented Oct 17, 2018

eclare108213 commented Oct 17, 2018

apcraig commented Oct 18, 2018

eclare108213 commented Oct 18, 2018

eclare108213 commented Oct 18, 2018

eclare108213 commented Oct 18, 2018

apcraig commented Oct 18, 2018

njeffery commented Oct 18, 2018 via email

eclare108213 commented Oct 19, 2018

duvivier Oct 19, 2018

duvivier Oct 19, 2018

duvivier Oct 19, 2018

duvivier left a comment

duvivier Oct 19, 2018

duvivier Oct 19, 2018

eclare108213 commented Oct 19, 2018

duvivier commented Oct 19, 2018

njeffery commented Oct 23, 2018

apcraig commented Oct 23, 2018

eclare108213 commented Oct 23, 2018

eclare108213 commented Oct 24, 2018

eclare108213 left a comment

apcraig commented Oct 26, 2018

Njeffery debug zbgc3 d #207

Njeffery debug zbgc3 d #207

Conversation

njeffery commented Oct 12, 2018

apcraig commented Oct 13, 2018

apcraig left a comment

Choose a reason for hiding this comment

eclare108213 commented Oct 17, 2018

eclare108213 commented Oct 17, 2018

njeffery commented Oct 17, 2018 via email

eclare108213 commented Oct 17, 2018

eclare108213 commented Oct 17, 2018

apcraig commented Oct 18, 2018

eclare108213 commented Oct 18, 2018

eclare108213 commented Oct 18, 2018

eclare108213 commented Oct 18, 2018

apcraig commented Oct 18, 2018

njeffery commented Oct 18, 2018 via email

eclare108213 commented Oct 19, 2018

duvivier Oct 19, 2018

Choose a reason for hiding this comment

duvivier Oct 19, 2018

Choose a reason for hiding this comment

duvivier Oct 19, 2018

Choose a reason for hiding this comment

duvivier left a comment

Choose a reason for hiding this comment

duvivier Oct 19, 2018

Choose a reason for hiding this comment

duvivier Oct 19, 2018

Choose a reason for hiding this comment

eclare108213 commented Oct 19, 2018

duvivier commented Oct 19, 2018

njeffery commented Oct 23, 2018

apcraig commented Oct 23, 2018

eclare108213 commented Oct 23, 2018

eclare108213 commented Oct 24, 2018

eclare108213 left a comment

Choose a reason for hiding this comment

apcraig commented Oct 26, 2018