Building GOCART in UFS with spack-stack: cannot find esmf.mod

[benjamin-cash]

Hello,

Hopefully this is the right place to ask this question. :) Like the title says, I am trying to build GOCART as part of the ufs, using spack-stack, and I am running into the following error:

/work2/02441/bcash/frontera/ufs-weather-model/develop/GOCART/ESMF/Shared/Chem_AeroGeneric.F90:18:7:

18 | use ESMF
| 1
Fatal Error: Cannot open module file ‘esmf.mod’ for reading at (1): No such file or directory

I can build the model in many other configurations without issue, it is specifically when I try to build with GOCART that this issue arises. Do I need to do something beyond loading the esmf module to have it found by GOCART? Thanks!

@climbfuji - Just tagging you in since you were the first one to run into this.

[benjamin-cash]

@rsdunlapiv - Since I see you come up here, any thoughts on what could be happening here?

[rsdunlapiv]

@benjamin-cash it looks like an issue with the UFS build system using spack-stack. I think this should go to whomever is maintaining the spack-stack and the UFS build system itself.

[mathomp4]

@benjamin-cash What version of GOCART are you trying to use? I want to say Spack and GOCART might only work in develop at this point. If so, I might need to backport some CMake changes into GOCART 2.0.

For example, these PRs:

#115
#126

are only in develop I believe. It wouldn't be too hard I think to backport all the changes needed.

[mathomp4]

Also this:

CMakeFiles/ufs_model.dir/link.txt

makes me think figuring this out will be "fun". I mean, the CMake for that seems to be:

###############################################################################
### UFS executable
###############################################################################
add_executable(ufs_model driver/UFS.F90)
add_dependencies(ufs_model ufs)
target_link_libraries(ufs_model ufs esmf w3emc::w3emc_d)

which means a nice long trek through the CMake.

And of course ufs-weather-model has five FindESMF.cmake that I know of. And maybe spack-stack is pulling in more?

Are you able to try building a "GOCART-less" version of UFS? Maybe if removing GOCART fixes this, we know it's somehow on our end...

[benjamin-cash]

Would you like access? I have a startup allocation on Frontera specifically to enable porting UFS there and benchmarking the code. I'd be happy to add you or someone else.

[benjamin-cash]

@mathomp4 - It is definitely related to GOCART. I've built the ATM, S2S, and S2SW apps without hitting this snag. It comes up when I try to build S2SWA and GOCART.

[benjamin-cash]

@mathomp4 - Just poking my head in to see if this ever got resolved.

[mathomp4]

@benjamin-cash I've built all of GEOSgcm inside of spack so this should be doable. Have you tried GOCART v2.1.0?

If that still doesn't work, you could try changing this chunk of code in the CMakeLists.txt:

  if (NOT ESMF_FOUND)
    find_package (ESMF REQUIRED)
  endif ()

to:

  if (NOT TARGET esmf)
    if (DEFINED ENV{esmf_ROOT})
      message (STATUS "Found esmf_ROOT in environment: $ENV{esmf_ROOT}")
      find_path(ESMF_CMAKE_PATH FindESMF.cmake HINTS "$ENV{esmf_ROOT}/cmake")
      message (STATUS "Found FindESMF.cmake in: ${ESMF_CMAKE_PATH}")
      if (ESMF_CMAKE_PATH)
        message (STATUS "Appending to CMAKE_PREFIX_PATH: ${ESMF_CMAKE_PATH}")
        list (APPEND CMAKE_MODULE_PATH ${ESMF_CMAKE_PATH})
      endif ()
    endif ()

    find_package(ESMF MODULE REQUIRED)

    # ESMF as used in MAPL requires MPI
    # NOTE: This looks odd because some versions of FindESMF.cmake out in the
    #       world provide an "esmf" target while others provide "ESMF". So we
    #       need this ugliness to support both.
    if (TARGET esmf)
      target_link_libraries(esmf INTERFACE MPI::MPI_Fortran)
    else()
      target_link_libraries(ESMF INTERFACE MPI::MPI_Fortran)
      # MAPL and GEOS use lowercase target due to historical reasons but
      # the latest FindESMF.cmake file from ESMF produces an ESMF target.
      add_library(esmf ALIAS ESMF)
    endif()
  endif ()

That bit of horribleness is what I came up with in my current testing of trying spack install geosgcm

[climbfuji]

@mathomp4 @benjamin-cash I think I can spend some time this afternoon trying to decipher this.

[benjamin-cash]

It looks like I can't test on Orion at this point. The installation of spack-stack there does not include w3emc so the model build fails, and when I tried to build spac-stack myself it died almost immediately with an error about the C compiler and "github-actions-v0.1 raised ValueError: source is not trusted". So it will have to wait until someone updates that on Orion or @climbfuji untangles it.