Where can I find the pre-processed fortran source files used in the lfric_apps rose-stem lfric_atm build?

[MichaelWhitall]

Hi folks,

I'm embarking on my first attempt to debug a failing lfric_apps rose-stem run of lfric_atm :)

I have some linked UM / lfric_apps tickets:
https://code.metoffice.gov.uk/trac/um/ticket/7407
https://code.metoffice.gov.uk/trac/lfric_apps/ticket/481

The UM tests look fine, and so do most of my latest go at lfric_apps tests, but I'm getting run-time fails in the comorph_dev apps that I've modified to test the new functionality, only for the SCM spice_intel_fast-debug compile (the same configuration run on xc40 CCE runs fine, including a larger 3D run).

The failing spice intel SCM comorph_dev run gives an error complaining about a floating divide-by-zero in one of the UM physics source files added by the linked UM ticket:

forrtl: error (73): floating divide by zero
Image              PC                Routine            Line        Source             
lfric_atm          0000000003520294  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC8526CA630  Unknown               Unknown  Unknown
lfric_atm          0000000001A2A070  bm_entrain_parcel         366  bm_entrain_parcel.F90

My problem now is that L366 of that file in my UM branch doesn't even contain a divide. Presumably the line number given by the error message refers to the pre-processed source file, not the original file in my UM branch, and the pre-processor has added or removed lines so that they don't match-up. I'm familiar with this issue when debugging run-time errors in the UM; one simply needs to go and dig out the pre-processed fortran source files from
cylc-run/vnXX.Y_my_branch/share/fcm_make_spice_whatever/preprocess-atmos/src
and find the line-number the error refers to in there.

How do I do this in lfric_apps?
I've had a good rummage through the files contained in the appropriate cylc-run directories on spice (where the compile for the failing job was performed), but have so-far failed to find the pre-processed fortran source. Does anyone know where the LFRic build squirrels these away?

Cheers!
Mike

[iboutle]

I'd be keen to know the answer to this too - we've had discussions before about the need to access this for exactly the purpose you describe.

As a workaround, if you follow the command line build (https://code.metoffice.gov.uk/trac/lfric/wiki/GhaspSupport/PhysicsHandsOn#a2.Commandlinebuildandrun) you will be able to look at the pre-processed code in the /working directory - and running your comorph_dev scm case from the command line will also probably make it much easier to debug.

[MichaelWhitall]

Awesome thanks Ian, I think I'll try getting the command-line build to run and editing the lfric_atm configuration.nml file to have my new functionality turned on; that sounds like an ideal framework for debugging if I can reproduce the problem there :)

Cheers!
Mike

[Steve Mullerworth (stevemullerworth)]

Hi Michael, LFRic compiles take place on temporary directories for optimisation reasons, which is why they don't appear in the cylc run output.

I do advocate working out how to build and test from the command line because turnaround can be much quicker as you benefit from incremental builds rather than having to rebuild each time. You may need to construct your own submission script if you want to run in batch.

For your problem above, the only preprocessing that currently happens to (capital F) .F90 files is for #if defined lines, and there are none in the file you mention. The line number is the start of a loop. I'm wondering if the issue is that it's an OpenMP loop that's causing the compiler to give a confusing line number. What happens if you remove the openmp directives?

[iboutle]

The issue with files that come from an external repo (UM, Jules etc) is that they get all of their blank lines stripped out, which really messes up the line numbers and makes it impossible to figure out what the traceback means. I'm not sure if we can stop this happening somehow? I guess it will partly get better when the UM code is in the lfric_apps repo, but will still be the case for Jules, Socrates etc.

Is it really the case that $TMPDIR is much faster than more persistent options (thinking especially about new HPC and new Spice), as if there were similarly fast (or only marginally slower) options that allowed the code to hang around even for a few days I think that would be a massive advantage for trying to debug issues with larger jobs (that can't be run at the command line)?

[Steve Mullerworth (stevemullerworth)]

I was only recently debugging an lfric_atm failure that occurred on line 605 of bm_cld (on EXA), and I didn't have any problems with the line numbers being wrong.

[MichaelWhitall]

I can confirm that if I run the command-line compile of LFRic (using ifort on my VDI), the pre-processor has removed all the blank lines from my UM source files so that the line-numbers have changed. This also seems to be the case for the spice ifort builds in rose-stem. But I guess it might be different for other compilers / platforms?

[iboutle]

That's interesting - I wonder if something different is happening on exa! See for example the command line build of bm_cld at: /net/home/h04/frib/fcm_lfric/vn1.2_ral_fixes/applications/lfric_atm/working/build_lfric_atm/science/src/um/src/atmosphere/large_scale_cloud/bm_cld.F90 (old spice) vs the trunk version at: https://code.metoffice.gov.uk/trac/um/browser/main/trunk/src/atmosphere/large_scale_cloud/bm_cld.F90 for the differences which arise.

I do seem to remember when trying to figure out the same issue for https://code.metoffice.gov.uk/trac/lfric_apps/ticket/355 that azspice still had the issue of the compiled line numbers not being the same as the source code.

[Steve Mullerworth (stevemullerworth)]

Sorry, I am wrong. Yes the blank lines were processed out in my local source. It was line 505 in my local source and 605 when I looked at the unprocessed source. I didn't notice the difference of exactly 100 when I checked the original!

[Matthew Hambley (MatthewHambley)]

The working directory for build tasks is controlled by rose-stem/suite/meto/common/suite_config_xc40.cylc and is set to a temporary directory. (BUILD_ROOT) If you want to interrogate working files there are two options.

You could change where the working directory is located. Remember that it must be somewhere the compute nodes can see.

Alternatively you could set up a post-script to the compile task which uses rsync to copy the contents of the working directory somewhere for inspection.

This is not normally an issue for us as we develop interactively at the command line where the working directory is next to the source. Which wont help you if your bug only reveals itself in configurations too large to run on the desktop.

[MichaelWhitall]

Many thanks all!

In my case, running the model from the command-line compile has reproduced the problem and I've found the bug :)

The problem actually turned out to be in the LFRic physics interface to explicit boundary-layer, which was leaving wvar unset in the top model-level; see my fix here:
https://code.metoffice.gov.uk/trac/lfric_apps/changeset?reponame=&new=6465%40main%2Fbranches%2Fdev%2Fmichaelwhitall%2Fvn2.0.1_improve_bimodal&old=6443%40main%2Fbranches%2Fdev%2Fmichaelwhitall%2Fvn2.0.1_improve_bimodal
The UM physics routine bdy_expl2 sets bl_w_var on all levels 2:nlayers+1 (remembering that the boundary-layer scheme's thermodynamic levels are offset by 1 level so that the surface is at k=1). But bl_exp_kernel_mod.F90 only copies the values from 2:nlayers into wvar.

I'm confused that in my failing test wvar was zero in the top level, not uninitialised. Is there a line somewhere that initialises the whole wvar array to zero before the UM physics? Anyhow, my new code divides things by sqrt(wvar) on the assumption that it will always be positive (which should be the case because bdy_expl2 imposes a minimum value of 1.0E-12 on the whole array for safety). Hence I was hitting a divide-by-zero at the top model-level but only in LFRic.

Cheers!
Mike

[iboutle]

Well spotted Mike! wvar is initialised to zero in https://code.metoffice.gov.uk/trac/lfric_apps/browser/main/trunk/science/um_physics_interface/source/algorithm/init_turbulence_fields_alg_mod.x90, but you're right that the copy happening in bl_exp_kernel is wrong and should go to nlayers+1

[MichaelWhitall]

Cool thanks no problem, that's good to know :)
Is the initialisation of wvar in init_turbulence_fields_alg_mod redundant once I've fixed the level-indexing in the copy in bl_exp_kernel_mod? Or does it still need to be initialised to zero for other reasons?

Cheers!
Mike

[iboutle]

I'd leave it, as I suspect it was done for cases where bimodal or something like that isn't being used and so it isn't being set in the BL code