[Announcement] PSyclone 3.0.0 upgrade in LFRic Core and Apps

[Iva Kavcic (TeranIvy)]

Guidance to PSyclone 3.0.0 upgrade

PSyclone 3.0.0 was released on Friday 6 December 2024, delivering new functionality for supporting LFRic science and performance optimisation. There are, however, some non-backward compatible changes in PSyclone APIs which impact how LFRic Apps and Core are built and how they configure PSyclone.

PSyclone 3.0.0 is incorporated into the LFRic software stack on the Met Office platforms as well as the LFRic Core and Apps source as of Thursday 12 December 2024. This is part of the preparation for the LFRic Apps vn2.0 release.

Changes in PSyclone 3.0.0 relevant for LFRic developers and users

More detailed information about new and modified functionality in PSyclone 3.0.0 can be found here. The most important changes relevant to LFRic are outlined below.

• PSyclone LFRic API is now called lfric instead of the legacy dynamo0.3. Also, the concept of a “default” API has been removed from PSyclone, meaning that psyclone command must specify API option. This impacts the implementation of psyclone and psyad commands in LFRic and therefore the LFRic build process. See "The psyclone command" in the PSyclone user guide and "The psyad command" in the PSyAD user guide.

• LFRic PSy-layer now looks up nlayers for each kernel in an invoke call individually, instead of passing the nlayers from the first kernel/built-in call to all kernels in the invoke. Note, nlayers for each kernel is derived from the first field in the kernel metadata, regardless of its read/write access. See "Rules for General-Purpose Kernels" in the PSyclone user guide.

• Support for user-defined kernels that operate on DoFs. See "Rules for DoF kernels and "operates_on" in the PSyclone user guide.

• Support for kernels that operate on halos (for FFSL in LFRic). See "operates_on" in the PSyclone user guide. Note, this functionality is still being developed.

• Improvements to kernel-extraction and driver-generation (PSyKE) functionality (including MPI support). See "PSy Kernel Extractor (PSyKE)" in the PSyclone user guide.

• NaN-checking support is generalised so that kernel inputs can be checked for validity against user-provided ranges. See "Value Range Check" in the PSyclone user guide.

• Updates to support new operator index ordering in LFRic. See "Rules for General-Purpose Kernels" in the PSyclone user guide.

• Comments generated for built-ins in the PSy-layer to denote which built-in call the code is generated for.

• Derive module and subroutine names of adjoint tests from supplied filename for LFRic API. See the PSyAD user guide.

Guidance for LFRic developers and users

Although the changes are non-backward compatible, they are mainly focused on the LFRic build process and the LFRic software environments. Running the Rose test suites on Met Office platforms will continue to work as we are using specific versions of LFRic software environments. The command-line builds may be affected due to change of default environments.

Met Office developers

• No changes are required if you are running Rose test suite (uses versioned environments).

• If you are building code from the command line, please load the specific environment versions instead of "default" ones (see "LFRic software environments" below for specific Met Office platforms).

• It is advisable to wait for the LFRic Apps vn2.0 release to be finalised before creating new development branches.

External developers

• No changes are required unless you are building LFRic from the command line on MONSooN. In that case, please load the specific environment versions instead of "default" ones (see "LFRic software environments" below for specific Met Office platforms).

• It is advisable to wait for the LFRic Apps vn2.0 release to be finalised before creating new development branches and upgrading environments to PSyclone 3.0.0. See "Installation" in the PSyclone user guide for installation instructions.

What changed in LFRic

LFRic Core

The changes below were implemented in the revision 51572.

• As said above, the LFRic API is now called lfric instead of legacy dynamo0.3. This affects the LFRic Core build process, with the impacted source:

  • etc/psyclone.cfg (LFRic-specific PSyclone configuration file),
  • infrastructure/build/psyclone/psyclone.mk (LFRic implementation of the psyclone command).

• LFRic software environments use PSyclone 3.0.0 (see "LFRic software environments" below for specific Met Office platforms). The impacted source is:

  • infrastructure/rose-stem/inc/targets.rc,
  • rose-stem/site/meto/common/ directory (specifically suite_config_spice.cylc, suite_config_xc40.cylc and suite_config_ex1a.cylc).

LFRic Apps

The changes below were implemented in the revision 5632.

• As in the LFRic Core, the PSyclone API changes affect LFRic Apps build process, with the impacted source:

  • rose-stem/app/psyclone/bin/psyclone.sh (LFRic implementation of the psyclone command),
  • science/adjoint/build/psyad.mk (LFRic implementation of the psyad command).

• Use of PSyclone 3.0.0 in software environments (see "LFRic software environments" below for specific Met Office platforms). The impacted source is in rose-stem/site/meto/common/ directory (specifically suite_config_spice.cylc, suite_config_xc40.cylc, suite_config_lfricinputs.cylc, suite_config_lfric_coupled.cylc and suite_config_ex1a.cylc).

• Adjoint model patches are modified to reflect recent developments in PSyAD and LFRic API. This impacts the source in science/adjoint/patches/algorithm/ and science/adjoint/patches/kernel/ directories.

LFRic software environments

PSyclone 3.0.0 is now integrated into the LFRic software environments on the legacy/current (Spice, Crays XCE/F/S) and new (EXA/B/Z) Met Office platforms.

Spice/Desktop (legacy/current)

Compiler environments that use PSyclone 3.0.0 are:

• Intel 19.0.1 LFRic environment: environment/lfric/ifort/19.0_64/9 (Core and all Apps),
• Gnu 11.2.0 LFRic environment: environment/lfric/gcc/11.2.0/9 (Core and all Apps except for LFRic Inputs),
• Gnu 6.1.0 LFRic environment: environment/lfric/gcc/6.1.0/10 (LFRic Inputs App).

Note, if the specific version of a compiler environment is not loaded (e.g. module load environment/lfric/ifort), command-line builds may fail when the build system defaults are changed. If you wish to continue to use PSyclone 2.5.0 for command-line development and testing, please load:

• Intel 19.0.1 LFRic environment: environment/lfric/ifort/19.0_64/7 (Core and all Apps, PSyclone 2.5.0),
• Gnu 11.2.0 LFRic environment: environment/lfric/gcc/11.2.0/7 (Core and all Apps except for LFRic Inputs, PSyclone 2.5.0),
• Gnu 6.1.0 LFRic environment: environment/lfric/gcc/6.1.0/9 (LFRic Inputs App, PSyclone 2.5.0).

Cray XCE/F (legacy/current)

Compiler environments that use PSyclone 3.0.0 are:

• Intel 17.0.1 LFRic environment: meto-environment/lfric/intel/17.0.0.098/10 (Core and all Apps except for LFRic Coupled),
• Cray 8.7.0 LFRic environment: meto-environment/lfric/cce/8.7.0/7 (Core and all Apps except for LFRic Coupled),
• Intel 17.0.1 MOCI environment: GC5-PrgEnv/intel/17.0.0.098/2024-12-ifort/5427 (LFRic Coupled App).

Cray XCS/MONSooN (legacy/current)

Compiler environments that use PSyclone 3.0.0 are:

• Intel 17.0.1 LFRic environment: meto-environment/lfric/intel/17.0.0.098/10 (Core and all Apps except for LFRic Coupled),
• Cray 8.7.0 LFRic environment: meto-environment/lfric/cray/8.7.0/8 (Core and all Apps except for LFRic Coupled),
• Intel 17.0.1 MOCI environment: GC5-PrgEnv/intel/17.0.0.098/2024-12-ifort/5425 (LFRic Coupled App).

Note, if the specific version of a compiler environment is not loaded (e.g. module load meto-environment/lfric/intel), command-line builds may fail when the build system defaults are changed. If you wish to continue to use PSyclone 2.5.0 for command-line development and testing, please load:

• Intel 17.0.1 LFRic environment: meto-environment/lfric/intel/17.0.0.098/9 (Core and all Apps except for LFRic Coupled, PSyclone 2.5.0),
• Cray 8.7.0 LFRic environment: meto-environment/lfric/cray/8.7.0/7 (Core and all Apps except for LFRic Coupled, PSyclone 2.5.0),
• Intel 17.0.1 MOCI environment: GC5-PrgEnv/intel/17.0.0.098/2024-02-ifort/5103 (LFRic Coupled App, PSyclone 2.5.0).

Cray EXA/B (porting in progress)

Please note that porting of LFRic Core and Apps to the new Cray EXA/B is in progress, and not all Apps are working on the new platforms. Compiler environments that use PSyclone 3.0.0 are:

• Gnu 12.1.0 LFRic environment: lfric-gnu/12.2.0/0.5,
• Cray 15.0.0 LFRic environment: lfric-cray/15.0.0/0.5.

Azure Spice (porting in progress)

Building of the LFRic software environments and porting of the LFRic Core and Apps is in progress.

Iva, on behalf of Sim IT - with thanks to the STFC PSyclone developers and Sim IT SSEs (releases are a lot of work!) 😃

[Iva Kavcic (TeranIvy)]

Update

I set the environment defaults on legacy Spice and Crays XCE/F/S to PSyclone 3.0.0.

Porting of LFRic Apps to Azure Spice and LFRic Core to Crays EXA/B is in progress. They will use environments with PSyclone 3.0.0.