Update ESMF from spack-stack fork: fix netcdf static libs and variant combinations

[climbfuji]

Description

This PR updates the ESMF package from the version in spack-stack's spack fork:

1. Fix the call to nc-config for static builds.
2. Simplify the version constraints (@8.4.0:8.4.99 --> @8.4 etc.)
3. Include logic for commonly used beta snapshots so that spack users can work with versions of the ufs-weather-model, for example, that sometimes point to these beta snapshots.
4. Fix a bug in versions up to 8.4.1 on Cray machines (see The ESMF build system strikes back: failure to build on Narwhal JCSDA/spack-stack#517 for the full details)

[spackbot-app]

@theurich can you review this PR?

This PR modifies the following package(s), for which you are listed as a maintainer:

• esmf

[theurich]

The --static for netcdf ~shared looks good to me. But the parallelio section seems to bring back an old version of that section. That part should be updated.

[climbfuji]

@theurich Thanks for the pointer. I wasn't sure about the timeline and which version was the correct/better one. I will revert this block of changes, and to make sure it doesn't happen again, update our fork again immediately after this PR is merged.

[climbfuji]

@theurich I reverted the change and used the simplified syntax for the versions. I would like to ask @AlexanderRichert-NOAA and @tldahlgren for confirmation that this syntax is indeed equivalent.

[theurich]

Looks good to me now. Thanks!

[jedwards4b]

@theurich thanks for the clarification

[climbfuji]

Thanks @jedwards4b and @theurich for the rapid turnaround!

[tldahlgren]

How is the snapshot variant being used in the package?

[tldahlgren]

I don't see this variant being passed to the build process. Am I missing something?

[AlexanderRichert-NOAA]

It's a workaround for the fact that in modules.yaml, spack doesn't distinguish between, say, "@1.2.3" and "@1.2.3b09". I discussed this at some length with Todd once upon a time and there wasn't really another good solution. So this is essentially a dummy variant that let's us distinguish ESMF's "b??" subversions when we customize the modulefile names of one of our dependencies (MAPL).

[theurich]

One question to consider - and I do not know the answer - is how much longer will anything depend on a specific snapshot tag like v8.3.0b09? I think it was unfortunate that some models started depending on that specific snapshot tag, but hopefully that is a thing of the past now?

[climbfuji]

If I was the one to make this decision, I would say that any submitted ufs-weather-model code must use an official release of ESMF. But as long as UFS WM developers want to test with a beta snapshot (and have the mapl dependency be built with it, too), and spack(-stack) is used to install that version, this code needs to stay.

[climbfuji]

It's ok of course to leave this out of mainline spack and keep that "feature" in our fork, just one more difference to carry around.

[theurich]

Thanks @climbfuji for clarifying. I have no issue with this "feature" moving into the mainline spack ESMF package. Generally I think the fewer differences between your fork and the mainline, the better.

[scheibelp]

I don't want to hold up the PR on this but would like to understand the notion better: the module file should be able to access the version of dependencies (e.g. 8.3.0b09). I can see how you might not want to alter all module file names just to account for this.

That being said I think this variant could be named more-clearly, like:

variant(ufs-weather-model)

And furthermore if it's a dummy variant which derives from the version, it might also be useful to encode a conflict

conflicts(+ufs-weather-model, when=)

admittedly this would be difficult to encode given that we don't support not.

[scheibelp]

We do have require come to think of it, so you can do

require(@8.3.0b09, when=+ufs-weather-model)

[scheibelp]

I don't want to hold up the PR on this but would like to understand the notion better: the module file should be able to access the version of dependencies (e.g. 8.3.0b09). I can see how you might not want to alter all module file names just to account for this.

That being said I think this variant could be named more-clearly, like:

variant(ufs-weather-model)

And furthermore if it's a dummy variant which derives from the version, it might also be useful to encode a conflict

conflicts(+ufs-weather-model, when=)

admittedly this would be difficult to encode given that we don't support not.

[AlexanderRichert-NOAA]

@scheibelp FWIW-- I conferred with Todd on this some time ago, because the problem we ran into is that when Spack considers version in modules.yaml, it doesn't distinguish between, e.g., 8.3.0 and 8.3.0b09, in that if we have entries for both, both modifications will be applied. At first we considered a boolean variant, but in principle there can be multiple beta snapshots per version number, so that's why we went with the named variant (with an entry in modules.yaml for each).

[climbfuji]

I am not sure I understand the question/comment @scheibelp. The reasoning was to be able to encode the beta snapshot in the module names, which requires a bit of logic in the modules.yaml section of the environment when we build multiple combinations of dependencies, for example for mapl:

      mapl:
        suffixes:
          ^esmf@8.2.0~debug snapshot=none: 'esmf-8.2.0'
          ^esmf@8.2.0+debug snapshot=none: 'esmf-8.2.0-debug'
          ^esmf@8.3.0b09~debug snapshot=b09: 'esmf-8.3.0b09'
          ^esmf@8.3.0b09+debug snapshot=b09: 'esmf-8.3.0b09-debug'
          ^esmf@8.3.0~debug snapshot=none: 'esmf-8.3.0'
          ^esmf@8.3.0+debug snapshot=none: 'esmf-8.3.0-debug'
          ^esmf@8.4.0~debug snapshot=none: 'esmf-8.4.0'
          ^esmf@8.4.0+debug snapshot=none: 'esmf-8.4.0-debug'
          ^esmf@8.4.1~debug snapshot=none: 'esmf-8.4.1'
          ^esmf@8.4.1+debug snapshot=none: 'esmf-8.4.1-debug'
          ^esmf@8.4.2~debug snapshot=none: 'esmf-8.4.2'
          ^esmf@8.4.2+debug snapshot=none: 'esmf-8.4.2-debug'
          ^esmf@8.5.0~debug snapshot=none: 'esmf-8.5.0'
          ^esmf@8.5.0+debug snapshot=none: 'esmf-8.5.0-debug'

[scheibelp]

why wouldn't

      mapl:
        suffixes:
          ^esmf+ufs-weather-model: esmf-ufs-wm

work? or, stripping the variant entirely and just pinning it to versions:

      mapl:
        suffixes:
          ^esmf@=8.3.0b09: esmf-ufs-wm

that when Spack considers version in modules.yaml, it doesn't distinguish between, e.g., 8.3.0 and 8.3.0b09

I think I see what you mean, if you have a listing like

          ^esmf@8.3.0b09: '...'
          ^esmf@8.3.0: '...'

then, as stated, the second entry matches both:

• Firstly, I thought module files used ordered matching
• Secondly, my initial suggestions in this comment avoid an extensive per-version listing (you should be able to conceptually generate the suffixes based on the different aspects of the spec - version, variant, etc. - vs. handling them combinatorially)
• Lastly, one thing that has likely changed since your conversation is that you can more-precisely specify versions with @=, where @=8.3.0 would not match @8.3.0b09

Does any of that suggest a means to eliminate or clarify the use of this snapshot variant?

[climbfuji]

Within one release / one environment that we build (each quarter), we have to support multiple combinations of mapl and esmf, for example

mapl@2.40.3 ^esmf@8.5.0
mapl@2.35.2 ^esmf@8.5.0
mapl@2.35.2 ^esmf@8.4.2
mapl@2.42.7 ^esmf@8.5.1b03

(I made up the last one)

and since we don't want hashes in modules, we need module names like this:

mapl/2.40.3-esmf-8.5.0
mapl/2.35.2-esmf-8.5.0
mapl/2.35.2-esmf-8.4.2
mapl/2.42.7-esmf-8.5.1b03

[scheibelp]

I think my note about @= accounts for your latter need. Also, according to https://spack.readthedocs.io/en/latest/module_file_support.html#customize-the-naming-of-modules you could do something like

suffixes:
  ^esmf: esmf-{^esmf.version}

so you don't have to list them out individually.

[climbfuji]

I think my note about @= accounts for your latter need. Also, according to https://spack.readthedocs.io/en/latest/module_file_support.html#customize-the-naming-of-modules you could do something like

suffixes:
  ^esmf: esmf-{^esmf.version}

so you don't have to list them out individually.

That's a neat idea, we should try that next. Thanks @scheibelp !