Build static libjpeg if requested, use hpc-stack libjpeg if built in jasper and NCEPLIBS; add NCAR Cheyenne #38

climbfuji · 2020-10-26T20:21:32Z

This PR fixes a few issues with libjpeg that surfaced when trying to install libjpeg on Cheyenne, see #39.

It also adds the configuration files for Cheyenne with Intel and GNU, and a separate NCAR YAML stack config. The latter differs from the NOAA config in building a static version of libjpeg.

Limitations:

I fixed the jasper cmake build only, i.e. only for newer versions of jasper.
The ufs-weather-model executable now launches, i.e. the library load problem is solved, but terminates immediately after the ESMF init. It also prints the ESMF banner ntasks number of times. I have seen this before, presumably ESMF is not built correctly on SGI MPI systems. I can fix this in this or a follow-up PR, whatever is better.

…peg lib in jasper and g2

aerorahul

Looks good.
Just a couple minor comments.

Also, can you please add shared: NO in stack_noaa.yaml section for jpeg.

See bugfix/jpeg branch. it will save us another PR.

Thanks.

aerorahul · 2020-10-26T20:36:13Z

libs/build_jasper.sh

@@ -56,13 +56,18 @@ else
    useCmake=NO
 fi

+if [[ "${STACK_jpeg_build}" == "YES" ]]; then
+    module load jpeg


Would just a module try-load jpeg work? Because if one installed jpeg previously and now just updating jasper, the if block would return false.

Yes, I came across that only later when I looked at the NCEPLIBS scripts. Should be fixed now, please check.

aerorahul · 2020-10-26T20:37:26Z

libs/build_jpeg.sh

 cmake $sourceDir \
  -DCMAKE_INSTALL_PREFIX=$prefix \
  -DBUILD_TESTS=ON \
  -DBUILD_EXECUTABLES=ON \
-  -DCMAKE_BUILD_TYPE=RELEASE
+  -DCMAKE_BUILD_TYPE=RELEASE \
+  ${shared_flags}


This is fine.
Can you see bugfix/jpeg and please add the check_flags logic from that branch. It will save us another PR.

Should be done now, please check if I missed something.

…wer comments

aerorahul · 2020-10-26T22:47:03Z

config/config_cheyenne_gnu.sh

+export STACK_EXIT_ON_FAIL=Y
+export WGET="wget -nv"
+
+# Load these basic modules for Hera


you mean Cheyenne

fixed for both intel and gnu

aerorahul

A few more comments.

aerorahul · 2020-10-26T22:47:22Z

config/config_cheyenne_intel.sh

+export STACK_EXIT_ON_FAIL=Y
+export WGET="wget -nv"
+
+# Load these basic modules for Hera


Cheyenne here too.

aerorahul · 2020-10-26T22:48:30Z

config/stack_ncar.yaml

@@ -0,0 +1,243 @@
+cmake:


how different is stack_ncar.yaml to stack_noaa.yaml?
Just curious.

Just the build-jpeg YES|NO, correct.

aerorahul · 2020-10-26T22:49:59Z

config/stack_noaa.yaml

@@ -4,6 +4,7 @@ cmake:

 jpeg:
  build: NO
+  shared: NO


If build: NO is the only difference between stack_noaa.yaml and stack_ncar.yaml, do you see any harm in turning jpeg to YES in stack_noaa.yaml?
I don't. May be @kgerheiser has some thoughts.

I would prefer building it per default on NOAA platforms, too. In that case we could ditch the stack_ncar.yaml file again.

I would in that case suggest just build jpeg on NOAA platforms and ditch stack_ncar.yaml

aerorahul · 2020-10-26T22:50:36Z

libs/build_jasper.sh

@@ -56,13 +56,18 @@ else
    useCmake=NO
 fi

+# Load jpeg module if created by hpc-stack; requires setting
+# MAKE_POLICY_DEFAULT_CMP0074 to new below so that JPEG_ROOT is searched
+module try-load jpeg


This has to be up top in the if $MODULES block.

It hopefully is now.

aerorahul

All good.
Thanks for the fixes and porting to Cheyenne.
Look at the comment on jpeg for noaa. We can build jpeg on noaa too and ditch stack_ncar.yaml.
I am fine with having both too.

aerorahul · 2020-10-26T23:00:27Z

config/stack_noaa.yaml

@@ -4,6 +4,7 @@ cmake:

 jpeg:
  build: NO
+  shared: NO


I would in that case suggest just build jpeg on NOAA platforms and ditch stack_ncar.yaml

climbfuji · 2020-10-26T23:03:41Z

All good.
Thanks for the fixes and porting to Cheyenne.
Look at the comment on jpeg for noaa. We can build jpeg on noaa too and ditch stack_ncar.yaml.
I am fine with having both too.

Ok, thanks. I'll wait if @kgerheiser has anything to add. I am also trying a fix for the ESMF build on Cheyenne right now.

climbfuji · 2020-10-27T02:17:28Z

I just pushed the esmf configuration update for SGI MPT (used on Cheyenne). Works just fine. After deciding what to do with building jpeg on NOAA platforms or not (and potentially removing the NCAR config), this PR is ready to merge.

aerorahul · 2020-10-27T04:08:59Z

config/config_cheyenne_gnu.sh

@@ -0,0 +1,21 @@
+#!/bin/bash


One last comment otherwise happy to merge.
Do we need config_cheyenne_gnu.sh and config_cheyenne_intel.sh
Can we just have one config_cheyenne.sh
See config_hera.sh for example.

I remember we "slacked" about this before. I want to minimize the amount of work and reduce the potential of making mistakes when rolling out a new version of the stack. All it should take is a no-brainer exercise of checking out the code, running the seup_modules script and the build_stack script. No editing of whatsoever files. This should be a failsafe exercise at 3am in the morning when somebody calls you that the libraries got deleted accidentally and they need them back right now. We should do the same for every other platform for which we support multiple compilers, in my opinion. Right now only hera, i.e. config_hera_gnu.sh and config_hera_intel.sh. And possibly append _intel for all the other platforms.

Thanks, appreciate it. All CI tests passed, let's wait to hear from @kgerheiser about the jpeg build on NOAA platforms.

or maybe not ;-)

climbfuji added 2 commits October 26, 2020 14:13

Fix jpeg build to create static library if requested, use hpc-stack j…

70c4d33

…peg lib in jasper and g2

Configuration for NCAR Cheyenne with Intel and GNU

41efeac

climbfuji marked this pull request as ready for review October 26, 2020 20:24

aerorahul reviewed Oct 26, 2020

View reviewed changes

climbfuji added 2 commits October 26, 2020 16:44

Updates to libs/build_jasper.sh and libs/build_jpeg.sh based on revie…

b693d34

…wer comments

config/stack_noaa.yaml: build static jpeg if requested to build at all

0ebcfdb

aerorahul reviewed Oct 26, 2020

View reviewed changes

Cheyenne is Cheyenne and not Hera

f0e4a75

aerorahul reviewed Oct 26, 2020

View reviewed changes

libs/build_jasper.sh: move jpeg module load intto $MODULES block

b69d363

aerorahul approved these changes Oct 26, 2020

View reviewed changes

libs/build_esmf.sh: add configuration option for SGI MPT

0c5558d

aerorahul reviewed Oct 27, 2020

View reviewed changes

aerorahul merged commit 2585235 into NOAA-EMC:develop Oct 27, 2020

climbfuji mentioned this pull request Oct 27, 2020

libjpeg not built statically if requested and not used by applications depending on it #39

Closed

This was referenced Nov 13, 2020

feature/hpc-stack-nco: merge Cheyenne updates from develop #62

Merged

Install feature/hpc-stack-nco on Cheyenne #64

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build static libjpeg if requested, use hpc-stack libjpeg if built in jasper and NCEPLIBS; add NCAR Cheyenne #38

Build static libjpeg if requested, use hpc-stack libjpeg if built in jasper and NCEPLIBS; add NCAR Cheyenne #38

climbfuji commented Oct 26, 2020 •

edited

Loading

aerorahul left a comment

aerorahul Oct 26, 2020

climbfuji Oct 26, 2020

aerorahul Oct 26, 2020 •

edited

Loading

climbfuji Oct 26, 2020

aerorahul Oct 26, 2020

climbfuji Oct 26, 2020

aerorahul left a comment

aerorahul Oct 26, 2020

aerorahul Oct 26, 2020

climbfuji Oct 26, 2020

aerorahul Oct 26, 2020

climbfuji Oct 26, 2020

aerorahul Oct 26, 2020

aerorahul Oct 26, 2020

climbfuji Oct 26, 2020

aerorahul left a comment

aerorahul Oct 26, 2020

climbfuji commented Oct 26, 2020

climbfuji commented Oct 27, 2020

aerorahul Oct 27, 2020

climbfuji Oct 27, 2020

aerorahul Oct 27, 2020

climbfuji Oct 27, 2020

climbfuji Oct 27, 2020

Build static libjpeg if requested, use hpc-stack libjpeg if built in jasper and NCEPLIBS; add NCAR Cheyenne #38

Build static libjpeg if requested, use hpc-stack libjpeg if built in jasper and NCEPLIBS; add NCAR Cheyenne #38

Conversation

climbfuji commented Oct 26, 2020 • edited Loading

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul Oct 26, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

climbfuji commented Oct 26, 2020

climbfuji commented Oct 27, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

climbfuji commented Oct 26, 2020 •

edited

Loading

aerorahul Oct 26, 2020 •

edited

Loading