Inconsistent CRTM fix files

[AlexanderRichert-NOAA]

From NOAA-EMC/GSI#447 (comment) :

The crtm/2.4.0 coefficients associated with crtm-fix/2.4.0_emc from /lfs/h1/emc/nceplibs/noscrub/spack-stack/spack-stack-1.4.1/envs/unified-env-intel22/install/modulefiles/Core. are not correct. While ctests run with these CRTM coefficients do not seg fault, the radiance penalties differ from develop in the 2nd digit.

The spack-stack crtm-fix/2.4.0_emc sets CRTM_FIX=/lfs/h1/emc/nceplibs/noscrub/spack-stack/spack-stack-1.4.1/envs/unified-env-intel22/install/intel/2022.0.2.262/crtm-fix-2.4.0_emc-2i4pkrg/fix.

The production crtm/2.4.0 sets CRTM_FIX=/apps/ops/prod/libs/intel/19.1.3.304/crtm/2.4.0/fix.

The spack-stack CRTM_FIX contains 1706 files. The production CRTM_FIX cotnains 1410 files. A diff of the two directories finds 445 $sensor.TauCoeff.bin files differ.

The ctests reported above bypassed this problem by explicitly pointing at CRTM coefficients in /apps/ops/prod/libs/intel/19.1.3.304/crtm/2.4.0/fix.

We need to get the correct CRTM coefficients in the spack-stack CRTM_FIX.

[AlexanderRichert-NOAA]

Appears to be due to contents of ftp://ftp.ssec.wisc.edu/pub/s4/CRTM/fix_REL-2.4.0_emc.tgz changing over time.

[arunchawla-NOAA]

who manages the CRTM fix files ? We need to make sure that the appropriate fix files are associated with the correct version. We would not have found this if it was not due to Russ's diligence

[AlexanderRichert-NOAA]

@BenjaminTJohnson can you shed some light on the evolution of ftp://ftp.ssec.wisc.edu/pub/s4/CRTM/fix_REL-2.4.0_emc.tgz in recent months? Seems to be creating some confusion since there are some significantly different values.

Based on the spack recipe changes, it looks like the checksum changed November or so of last year (https://github.com/JCSDA/spack/pull/195/files), and in early March 2023 (JCSDA/spack@e3d1410).

[BenjaminTJohnson]

A few files have been updated per EMC/NCO requests, mostly surrounding atms_n21.SpcCoeff files and the addition of tropics files.

The release branch code has also been updated.

We should probably quickly decide on a better mechanism for version control of binary tarballs.

If we decide that a given tarball name should be immutable, then we need a predictable pattern for naming.

I'm open to recommendations.

Ben

[arunchawla-NOAA]

Hi Ben

My preference is that all tars/tags be immutable as much as possible (that said sometimes when the tag is just released you may want to update a couple of times in the first weeks and that is ok). In general anytime we change we should change version numbers or we get into a mess

[arunchawla-NOAA]

That said, can you provide a tag with the appropriate set of fixed files that spack stack can deploy on all platforms ? We want to make sure that GSI runs properly on all the platforms

[climbfuji]

@BenjaminTJohnson I think it would be good if we settled this in spack-stack develop during our technical debt code sprint in the next weeks, do you agree? And also add the new versions that you communicated to me via email.

[climbfuji]

@BenjaminTJohnson Do you think we'll be able to resolve this for the existing tarballs within the last two days of the tech debt sprint?

For the record, we (Ben. J. and I) did have a meeting with Arun shortly before he left and we agreed on a tarball versioning convention going forward, and in particular that there will be no "re-tagging" (i.e. recreation of tarballs with the same name) in the future.

[BenjaminTJohnson]

Ignore my previous comment, which you might have received by email.

Yes I can start doing that now with the current version(s) of CRTM, is there any need to update the release branch? If so which versions of CRTM do you want this for?

[climbfuji]

@AlexanderRichert-NOAA @aerorahul Can you help us to identify if and if so, what, needs to be done for the existing CRTM fix files (this is not about future tarballs, we have a solution for that)? Thank you ...

Yes I can start doing that now with the current version(s) of CRTM, is there any need to update the release branch? If so which versions of CRTM do you want this for?

[climbfuji]

@alexrichert Will your PR that bumps crtm fix to 2.4.0.1_emc address this issue?

[climbfuji]

@AlexanderRichert-NOAA Did your PR for crtm 2.4.0.1 fix this issue?

[AlexanderRichert-NOAA]

Not specifically, but I believe the agreement is to not overwrite existing fix files so we don't have unexpected inconsistencies. cc'ing @BenjaminTJohnson- if we're in agreement there then we can close this.

[climbfuji]

@AlexanderRichert-NOAA What you wrote above is my understanding, too, and matches what was discussed between @BenjaminRuston, Arun and myself a few months ago. Closing. Thanks everyone!