ESMF 8.6.1

Overview

The 8.6.1 release of ESMF is a patch release to the previously released 8.6.0, introducing 2 minor feature enhancements and 14 fixes in various areas such as ESMF Config, HConfig, and ESMF Field.

Due to user requests, ESMF Config was enhanced to remove the single line limitation of 1024 characters. In addition, the ESMF library target within the FindESMF module has been changed to conform to the imported library standard. Both feature enhancements allow more flexibility and usability.

Several corrections were introduced, such as the returning of index space information by ESMF_FieldGet(), start_index attribute in ESMFMesh unstructured mesh file in edge cases, interpretation of ESMF_RUNTIME_ environment variable keys by ESMF_Initialize(), ESMF_NVML=OFF build option, building of ESMF with ESMF_NO_GETHOSTID, equality operators in the ESMF_HConfig and ESMF_HConfigIter, and ISO_C_BINDING symbols export via the ESMF Fortran module.

Incremental progress was also made in several other areas of the framework. In ESMF_Config, several enhancements and fixes were made to cover more edge cases such as dealing with string length, special characters, and blank values.

Note: ESMF_Config will be deprecated in a future major release for the preferred ESMF_HConfig class, which supports YAML coded configuration files. Deprecation has not been scheduled at this time due to the high impact it will have on existing configuration files.

Release Notes

• This patch release is backward compatible with ESMF 8.6.0. There were no API changes in this release.
• No bit-for-bit changes are expected for this release compared to ESMF 8.6.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• The index space information (minIndex, maxIndex, elementCount) returned by method ESMF_FieldGet() is now correct for a Field on any ESMF_STAGGERLOC (Grid) or ESMF_MESHLOC (Mesh) location.
• The ESMF_Config class has been enhanced by removing the 1024 character limit. It is now possible to set strings to values longer than 1024 characters. The maximum size is determined by the ESMF_Config buffer size, which is 256k.
• Multiple bugs in the implementation of ESMF_Config have been fixed.
  • A bug has been fixed that prevented changing values to a different string length.
  • It is now possible to enclose number sign (#) characters within single or double quotes in order to include this character in string values.
  • It is now possible to enclose apostrophes / single quote (‘) characters within double quotes in order to include this character in string values.
  • It is now possible to enclose double quote (“) characters within single quotes in order to include this character in string values.
  • It is now possible to set values to blank and change blank values to non-blank values.
  • Note: ESMF_Config will be deprecated in a future major release for the preferred ESMF_HConfig class, which supports YAML coded configuration files. Deprecation has not been scheduled at this time due to the high impact it will have on existing configuration files.
• The ESMF library target within the FindESMF module for CMake has been changed to ‘ESMF::ESMF’. This conforms to the imported library standard as seen in modules provided by Kitware. For backwards compatibility ‘ESMF’ has been added as an alias target.
• The issue of inadvertently exporting ISO_C_BINDING symbols via the ESMF Fortran module has been addressed.
• ESMF_RUNTIME_ environment variable keys that are set on the top level of the configuration file ingested by the ESMF_Initialize() method are now correctly interpreted.
• The ESMF_NVML=OFF build option now works correctly as documented.
• The build of libesmftrace_preload.dylib on some Darwin (Mac) systems with OpenMPI has been fixed. (Incorrect link flags were being used in this build.)
• The problem of building ESMF with macro ESMF_NO_GETHOSTID defined has been fixed.
• The ESMFMESH unstructured mesh file format now correctly begins the connection index calculation at the start_index value when the start_index attribute is attached to the elementConn variable. Previously, beginning in ESMF 8.3.0, this attribute had been incorrectly ignored.
• The ESMF_HConfig and ESMF_HConfigIter equality operators have been fixed. Previously, on some platforms, identical objects could be detected as not equal which could lead to problems with iterations not terminating correctly, etc.

Known Issues

• Same as 8.6.0.

Documentation

• ESMF Reference Manual for Fortran
• ESMF Reference Manual for C
• ESMF User Guide
• NUOPC Layer Reference
• Building a NUOPC Model
• ESMPy Doc

Tables

• Regridding Status Table
• Supported Platform Table
• API Test Coverage

ESMF 8.6.0

Overview

The 8.6.0 release of ESMF introduces two major new features: spherical vector regridding and integrated accelerator device management. Both features are provided at an early stage of the implementation, expected to be fleshed out over the course of the next several releases. In addition to these new developments, the release includes a number of improvements of existing functionality. Highlights of the 8.6.0 release are outlined in the following paragraphs, followed by a comprehensive list of release notes.

Support for spherical vector regridding has been requested by a number of groups over the years, and a basic implementation of the feature is now available in ESMF 8.6.0. One of the advantages of mapping vectors through 3D Cartesian space, as opposed to regridding the individual components separately, is improved accuracy of the results, especially in the polar regions. The initial implementation of this feature included with 8.6.0 is limited to 2D tangential vectors (expressed in terms of east and north components) on a spherical geometry (e.g. an ESMF_Grid with ESMF_COORDSYS_SPH_DEG) and requires that the vector components are stored in a single ungridded Field dimension of size 2. We expect these and other restrictions to be relaxed in future releases and welcome feedback from the community to help prioritize.

Accelerator device solutions that are capable of managing multi-component applications are likely to become critical as coupled systems are run on hardware that provides most of the compute power in the form of GPUs. ESMF 8.6.0 provides novel development in this area, building on the experience gained from CPU resource management for multi-component ESMF applications based on the petList concept, and its recent extension to ESMF-managed threading. The user has now the option to associate an analogous list for accelerator devices (devList) with each component. ESMF implements the required bookkeeping, providing component context specific device information. Paired with standard accelerator programming paradigms, such as OpenACC or OpenMP, this allows component code to target the desired devices for offloading. The current implementation is considered a proof of concept, and refinements of the feature are expected in future releases.

Among the areas of the framework that received incremental improvements is ESMX, where it is now possible to modify ESMF run-time behavior via ESMF_RUNTIME_* options in the ESMX run-time configuration file for convenience. Small functional improvements also went into the Array and Field Create() methods that now allow creation from the slice of an existing object. Handling this situation on the ESMF level can significantly simplify user code.

Further progress was made for multi-tile I/O support for Fields and Arrays. It is now possible to perform I/O for multi-tile Arrays and Fields with layouts other than 1 DE per PET. Related to I/O, the "normalization" attribute in mapping files written by the ESMF regrid weight generation methods and applications was corrected to set the value “N/A” for anything other than the conservative regridding methods.

Incremental progress, bug fixes, portability and performance improvements were made in several other areas of the framework. This includes full support for the NVHPC compiler suite, tracing support for MPI calls on all platforms, resolving crashes observed with the Darshan and Cray Performance Analysis Tools, and finally addition of a public C API for the ESMF trace feature. Please see the release notes below for a detailed list of changes.

Release Notes

• The public Fortran API in this release is backward compatible with the previous release, ESMF 8.5.0. There were a few API changes, none of which require user code changes. The complete list of API changes is summarized in a table showing interface changes since ESMF 8.5.0. The table includes the rationale and impact for each change.
• No bit-for-bit changes were observed for this release compared to ESMF 8.5.0. This is based on test runs with the Intel compilers using options "-O2 -fp-model precise". However, changes were made to the implementation of the conservative weight calculation to remove a problem caused by the same calculation being optimized differently in two places in the code. Bit-for-bit changes compared to ESMF 8.5.0 from this change are possible when using regridding methods ESMF_REGRIDMETHOD_CONSERVE or ESMF_REGRIDMETHOD_CONSERVE_2ND and not using strict floating point compiler options.
• No changes that affect the status of existing regridding methods were made to the ESMF regrid weight generation methods and applications. However, the tables summarizing the ESMF regridding status were extended to cover the spherical vector regridding option added in this release.
• Keys to match all of the ESMF_RUNTIME_* environment variables were added to the App Options of the ESMX run configuration. This feature provides a convenient way to control the run-time behavior of the ESMX executable from its standard configuration file.
• Basic accelerator device management for multi-component applications was implemented. This feature allows the user to assign accelerator devices via the “devList” option to individual components. "devList” is available as a new argument to ESMF_GridCompCreate() and ESMF_CplCompCreate(), and is also accessible on the ESMX level as Component Label Option. Once specified, ESMF handles the device bookkeeping, and provides the user with context specific device information through the ESMF_VMGet() API. This feature can be leveraged in connection with popular accelerator programming paradigms like OpenACC, OpenMP, or standard language approaches to offload component code to the desired devices.
• Basic support for vector regridding was added to the ESMF_FieldRegridStore() method. The initial implementation of this feature is limited to 2D tangential vectors (expressed in terms of east and north components) on a spherical geometry (e.g. an ESMF_Grid with ESMF_COORDSYS_SPH_DEG) and requires that the vector components are stored in an ungridded Field dimension. The advantage of using this capability, which maps vectors through 3D Cartesian space, instead of regridding both components separately, is that it provides more accurate results, particularly in the polar regions.
• The "normalization" attribute in mapping files written by the ESMF regrid weight generation methods and applications was changed to now correctly set “N/A” for anything other than the conservative regridding methods. User code that accesses the “normalization” attribute in ESMF mapping files might need to be adjusted accordingly.
• A new entry point was added to the ESMF_FieldCreate() interface allowing creation from an existing Field object. The interface supports slicing with respect to trailing ungridded dimensions.
• The ESMF_ArrayCreate() entry point that allows creation from an existing Array object was extended to support slicing with respect to trailing undistributed dimensions.
• The Read and Write operations for multi-tile Arrays and Fields (e.g., for representing a cubed sphere grid as a six-tile grid) have been extended to permit I/O for Arrays / Fields with layouts other than 1 DE per PET. This change applies to the ESMF_ArrayRead(), ESMF_ArrayWrite(), ESMF_ArrayBundleRead(), ESMF_ArrayBundleWrite(), ESMF_FieldRead(), ESMF_FieldWrite(), ESMF_FieldBundleRead(), and ESMF_FieldBundleWrite() methods. The remaining limitations of the implementation are discussed in the Restrictions and Future Work section of the I/O Capability in the reference m...

ESMF 8.5.0

Overview

The 8.5.0 release of ESMF comes with a few big new developments as well as improvements of existing functionality. The highlights of the 8.5.0 release are outlined in the following paragraphs, followed by a detailed list of release notes.

One major new development available with 8.5.0 is the addition of a hierarchical configuration class: ESMF_HConfig. This class provides YAML 1.2 support through the ESMF Fortran API. The ESMF_HConfig class integrates with the existing configuration class ESMF_Config for backward compatibility, and allows ESMF user code to seamlessly access, modify, and create information in YAML format. This new capability is leveraged within ESMF and NUOPC to provide a YAML alternative when specifying ESMF_Initialize() parameters, NUOPC attribute, petList, or run sequence information.

Another area that has seen a lot of new development is the Earth System Model eXecutable (ESMX) layer. ESMX has greatly matured in many areas since it was first introduced in the previous ESMF release and we believe it is now ready to be used by early adopters for “real world” applications. To this end, several new build options were added to the ESMX build configuration, and component building is now an integrated feature. The new ESMX_Builder command line tool eliminates the need for direct user interaction with the CMake system and improves the overall usability.

ESMF 8.5.0 adds a generic geometry class (ESMF_Geom) to support user code that deals generically with the existing geometry classes: ESMF_Grid, ESMF_Mesh, ESMF_LocStream, and ESMF_XGrid. Using the ESMF_Geom type, code will function regardless of the underlying geometry class. In this release only a few ESMF operations, such as Field creation, are overloaded to support the generic Geom type. Additional support will be added in future releases as the need arises.

Progress was also made in the area of multi-tile I/O support for Fields and Arrays, eliminating one of the restrictions of the previous release. Multi-tile Fields with ungridded dimensions, and multi-tile Arrays with undistributed dimensions are now fully supported.

With respect to installation and portability of the library, support for Spack and Docker continues to be an important focus. ESMF core team members are now maintainers of the official ESMF Spack package, and automated daily testing ensures continued support of this build option. Docker images are available and maintained under the ESMF organization on Docker Hub.

Incremental progress, bug fixes, and performance improvements were made in several other areas of the framework, including the ESMF_RegridWeightGen application, Mesh creation, Regridding, RouteHandle re-use, and others. Please see the release notes below for a comprehensive list of changes.

Release Notes

• The public Fortran API in this release is backward compatible with the last release of ESMF, 8.4.0, and patch releases 8.4.1 and 8.4.2. There were a few minor API changes, most of which do not require user code changes. The exception is a change to the Read() and Write() methods for multi-tile Arrays or Fields, where the tile number placeholder character in the fileName argument was changed from “#” to “*”. A number of new interfaces were added. The complete list of API changes is summarized in a table showing interface changes since ESMF 8.4.0, including the rationale and impact for each change.
• Some bit-for-bit changes are expected for this release compared to ESMF 8.4.0, and patch releases 8.4.1 and 8.4.2. This is based on test runs with the Intel compiler suite using options “-O2 -fp-model precise”, and due to the following changes:
  • Fixed a problem in the first-order conservative weight calculation that might occur in some situations when a destination cell is a concave quadrilateral. The fix was made to both the 2D Cartesian and 2D spherical weight conservative calculation. Expected bit-for-bit changes:
    • Small changes to first-order conservative weights for destination cells that are concave. These changes can occur for either 2D Cartesian or 2D spherical Grids or Meshes.
  • Updated the ESMF cubed sphere coordinate generation algorithm to use R8 reals. This makes the algorithm more accurate and also aligns it better with the same algorithm used by NASA. Expected bit-for-bit changes:
    • Small changes to the coordinate values produced by ESMF_GridCreateCubedSphere() or ESMF_MeshCreateCubedSphere(), and also changes to anything which depends on those values (e.g. regridding weights to/from an ESMF generated cube sphere Grid).
• No changes that affect regridding status were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• All of the fixes and improvements released with patches 8.4.1 and 8.4.2 are included in release 8.5.0.
• Significant improvements have been made to the Earth System Model eXecutable (ESMX) layer:
  • Component building has been integrated into the ESMX build system.
  • The build configuration has been enhanced through new build options.
  • The ESMX_Builder script has been added to simplify building ESMX applications.
  • The ESMX run configuration file has been updated to YAML formatting.
  • The names of the ESMX build and run configuration files can be customized.
  • The ESMX Data Component, which allows users to define import/export fields and prescribe data, has been added to the ESMX Layer.
• A hierarchical configuration class (ESMF_HConfig) with Fortran API was implemented. ESMF_HConfig covers the YAML 1.2 standard and integrates with the existing configuration class ESMF_Config for backward compatibility.
• The ESMF_Initialize() method now supports reading startup configurations from a YAML file.
• The ESMF_InfoSet() interface was overloaded to allow setting ESMF_Info key value pairs from an ESMF_HConfig object.
• The NUOPC methods that implement ingestion of PetList, RunSequence and Attribute information have been overloaded to support the YAML format through the ESMF_HConfig class.
• A new named constant, ESMF_STATEINTENT_INTERNAL, for type(ESMF_StateIntent_Flag) was added. This new option allows a state to be marked exclusively for component internal access.
• A generic geometry class (ESMF_Geom) was added to ESMF. A Geom object can be created from any object of the existing geometry classes (ESMF_Grid, ESMF_Mesh, ESMF_LocStream, and ESMF_XGrid). Having this class allows a user to pass a generic geometry object through a coupled system without knowing its exact type. To support greater flexibility in coupled systems, some ESMF operations are now supported on Geom (e.g. Field creation). More will be added as the need arises.
• Creating a Mesh, or adding nodes to an existing Mesh, without specifying the nodeOwners argument now works, even if there are PETs with zero nodes. Previously this condition led to a hang inside the respective ESMF method.
• Creating a Mesh without specifying node information now works, even if there are PETs with zero elements. Previously this condition led to a hang inside the respective ESMF method.
• Grids that contain DEs of zero width are now supported in regridding. Previously using a Grid of this type with some regridding methods (e.g. ESMF_REGRIDMETHOD_CONSERVE) would result in an error.
• The srcTermProcessing argument was added to the version of ESMF_FieldRegridStore() that operates on an XGrid. This fixes the issue where in some cases bit-for-bit reproducibility wasn’t available in that version of ESMF_FieldRegridStore().
• The RouteHandle reuse optimization in ESMF_FieldBundleRegridStore() was extended to include the Grid-to/from-Mesh and Mesh-to/from-Mesh combinations. This optimization is leveraged by the [NUOPC_Connector](https://earthsystemmodelin...

ESMF 8.4.2

Overview

ESMF 8.4.2 is a patch release that fixes a few minor issues that were discovered in 8.4.1 after it was released. Unless a user experiences any of the issues listed below, there is no need to upgrade to this release from patch 8.4.1.

As a patch release, 8.4.2 does not introduce new features beyond 8.4.0 or 8.4.1. Applications that have already moved to ESMF 8.5.0 beta tags should not go to the 8.4.2 patch. For these cases an upgrade to 8.5.0, when officially released, is the recommended next step. All of the fixes that come with patch release 8.4.2 are also available in beta tag v8.5.0b20 and newer.

Release Notes

• This patch release is backward compatible with ESMF 8.4.0 and 8.4.1.
• No bit-for-bit changes are expected for this release compared to ESMF 8.4.0 and 8.4.1.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• The ESMF profiler mode (activated by setting environment variable ESMF_RUNTIME_PROFILE=ON) has been made compatible with user code that calls the standard ESMF component methods from within OpenMP threaded regions. This is a rare scenario. However, it can be encountered e.g. under NOAA’s Unified Forecast System (UFS) when enabling the threading option for NASA’s GOCART chemistry component.
• A linking issue reported for the Darwin.gfortranclang.default build configuration, when building ESMF under the Spack package manager, has been fixed.

Known Issues

• Same as 8.4.1.

Documentation

• ESMF Reference Manual for Fortran
• ESMF Reference Manual for C
• ESMF User Guide
• NUOPC Layer Reference
• Building a NUOPC Model
• ESMPy Doc

Tables

• Regridding Status Table
• Supported Platform Table

ESMF 8.4.1

Overview

ESMF 8.4.1 is a patch release that fixes a major issue that was discovered in 8.4.0 after it was released. The identified bug can lead to memory corruption problems, which are often hard to detect and diagnose by their nature. We strongly recommend upgrading applications that are currently using 8.4.0 to patch release 8.4.1.

As a patch release, 8.4.1 does not introduce new features beyond 8.4.0. Besides the major bug fix mentioned above, a number of smaller bug fixes and maintenance work is included with the patch release. See the detailed release notes below. Applications that have already moved to ESMF 8.5.0 beta tags do not need to upgrade to the 8.4.1 patch. For these cases an upgrade to 8.5.0, when officially released, is the recommended next step. All of the fixes that come with patch release 8.4.1 are also available in beta tag v8.5.0b18 and newer.

Release Notes

• This patch release is backward compatible with ESMF 8.4.0.
• No bit-for-bit changes are expected for this release compared to ESMF 8.4.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• A bug in the implementation of method ESMF_FieldGet() was fixed. The problematic code was accessing the optional, intent(out) “name” argument without the proper present() check. As a consequence, code not specifying an actual “name” argument when making this call was at risk of suffering from memory corruption issues. Due to the fact that the ESMF library internally is making calls to ESMF_FieldGet() without passing the “name” argument, it must be assumed that all user code is at risk of memory corruption issues when using ESMF 8.4.0.
• A problem of incorrect ESMPy project metadata was fixed. This issue was causing failures during installation when using python’s setuptools module v67.1.0 or later: (“configuration error: project.maintainers[{data__maintainers_x}] must not contain {'author'} properties”). The fix allows ESMPy installation to work with the latest setuptools. We recommend this release for anyone doing a new installation of ESMPy.
• The NUOPC_Driver code responsible for stepping components through the initialization protocol has been optimized to reduce the amount of synchronization needed between components. This allows for a greater level of concurrency between components during initialization, resulting in a reduction of execution time.
• A number of code adjustments were made to support Intel’s LLVM based OneAPI compiler suite under the common ESMF_COMPILER=intel setting. Classic and OneAPI Intel compiler flavors are now supported under the same setting.

Known Issues

• Same as 8.4.0.

Documentation

• ESMF Reference Manual for Fortran
• ESMF Reference Manual for C
• ESMF User Guide
• NUOPC Layer Reference
• Building a NUOPC Model
• ESMPy Doc

Tables

• Regridding Status Table
• Supported Platform Table

ESMF 8.4.0

Overview

The 8.4.0 release of ESMF implements a number of exciting new features. Some are incremental in nature while others explore new territory. Highlights of the 8.4.0 release are outlined in the following paragraphs. A detailed list of release notes is provided further down.

ESMX, a new layer built on top of the ESMF and NUOPC APIs, is being introduced in this release. ESMX stands for the Earth System Model eXecutable (ESMX) layer. The goals of ESMX are (1) to simplify standing up new NUOPC-based systems, (2) promote hierarchical model component testing, (3) reduce the cost of maintaining NUOPC-based modeling systems, (4) improve alignment and interoperability between different NUOPC-based systems, and (5) provide a fast and coordinated roll-out strategy for new ESMF/NUOPC features.

Exploration and experimentation with ESMX is strongly encouraged. However, please recognize that it is a new development effort that comes with the typical rough edges. For example, no formal reference manual is available yet. A good starting point to learn about the layer is the ESMX/README.md. We expect development of ESMX to continue throughout the next several release cycles.

Progress was made in the area of I/O. Multi-tile Arrays and Fields can now be used in Read and Write operations. A common use case is the six-tile representation of the cubed sphere grid. There are a number of restrictions that apply to the multi-tile I/O support made available in this release. Please see the associated release note below for more information. The plan is to relax restrictions in future releases.

On the ESMF object level a new concept of “Named Aliases” was introduced. This concept builds on the existing ESMF alias concept for deep objects (objects explicitly created and destroyed by Create() and Destroy() methods). Named aliases manage their own private name without affecting the name of the aliased object. This allows the same object to be known under different names inside of different components.

Incremental progress was also made in the areas of XGrid, LocStream, and Grid. Here user feedback identified a number of issues that have been addressed in this release.

With the increasing importance of ARM based architectures, such as Apple’s M1, etc., work was done in this release to support the native development environment on Darwin (Mac OS X). The compiler combination GFortran + Clang is now considered fully supported on Darwin. The release has been fully tested on an Apple M1 system with MPIUNI, MPICH, and OpenMPI.

ESMPy users please note that the Python module has been renamed from "ESMF" to "esmpy" for better alignment with Python Enhancement Proposal (PEP) guidelines. Please check out the respective release note for details and potential impact on user code.

Release Notes

• The public Fortran API in this release is backward compatible with the last release of ESMF 8.3.0 and patch release ESMF 8.3.1. There were a few minor API changes, none of which require user code changes. The list of API changes is summarized in a table showing interface changes since ESMF 8.3.0, including the rationale and impact for each change.
• The Python module name for ESMPy was changed in this release from “ESMF” to “esmpy” for better alignment with Python Enhancement Proposal (PEP) guidelines. This change will require user code changes! ESMPy users will have to do a global search and replace on their scripts to adjust to the module name change. Also notice that the internal ESMF source tree directory structure has changed, moving ./src/addon/ESMPy to ./src/addon/esmpy. This might affect anybody keeping an ESMF repository clone, and maintaining scripts that depend on the internal directory naming.
• No bit-for-bit changes were observed for this release compared to release ESMF 8.3.0. This is based on test runs with the Intel compilers using options “-O2 -fp-model precise”.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• All of the fixes and improvements released with patch 8.3.1 are also included in release 8.4.0. Of particular note is support for messages above the 2GiB limit by the ESMF VM communication layer.
• The Earth System Model eXecutable (ESMX) layer was added to ESMF. ESMX removes technical hurdles that impede hierarchical NUOPC model testing. It includes a unified executable capable of driving multiple configurations of model components: single component, one-way forced, and fully dynamic two-way coupled. A simple YAML file is used to list the component build dependencies (CMake based). A standard NUOPC run configuration file is used to specify the processor layout, run sequence, the field dictionary, and model component options. ESMX supports hierarchical testing by allowing decentralized testing at the component level before integration into larger coupling systems, as well as testing on the integration level.
• The concept of “Named Aliases” has been introduced. Regular aliases of deep ESMF objects continue being created using a simple assignment operator. Changing the name of a regular alias affects the object and thus all other aliases. Named aliases are created using the new ESMF_NamedAlias() function. Named aliases manage their own private name. Changing the name of a named alias does not affect the object or other aliases.
• Using a Mesh that contains one or more elements with greater than 4 sides when creating an XGrid is now supported. Previously doing so would result in an error when data was transferred into or out of the XGrid.
• LocStreams can now be created from a Cartesian Mesh file. Previously the attempt to do so would result in an error.
• Attempting to create a Grid from a GRIDSPEC mosaic file that doesn’t contain variables with standard_name set to "grid_tile_spec", "geographic_latitude", or "geographic_longitude" now triggers clear ERROR log messages before returning with an error. Previously no ERROR messages were logged, making this situation hard to debug.
• Read and Write operations are now permitted for multi-tile Arrays and Fields. A common use case for this is representing a cubed sphere grid as a six-tile grid. This change applies to ESMF_ArrayRead(), ESMF_ArrayWrite(), ESMF_ArrayBundleRead(), ESMF_ArrayBundleWrite(), ESMF_FieldRead(), ESMF_FieldWrite(), ESMF_FieldBundleRead(), and ESMF_FieldBundleWrite(). Each of these methods reads or writes to multiple files in the multi-tile case. See the respective API documentation for details of how the tile-specific file names are constructed.
  Current limitations are:
  • For I/O of ArrayBundles and FieldBundles, all Arrays / Fields in the bundle must contain the same number of tiles;
  • I/O is not yet permitted for multi-tile Arrays / Fields with ungridded / undistributed dimensions;
  • I/O is currently only permitted for multi-tile Arrays / Fields with 1 DE per PET.
• ESMF profiles now include a summary of the full application run time for convenience. Previously user instrumentation was required to introduce an end-to-end profiling level.
• The configuration for ESMF_COMPILER=gfortranclang now works on Darwin systems (Mac OS X). Note that on Darwin, the system-level g++ invokes clang++, and unless you ensure that you have a true g++ early in your path (or build the MPI compiler wrappers to ensure that they wrap the true g++), you will end up using clang++ even if you think you are using the GNU C++ compiler. Setting ESMF_COMPILER=gfortranclang is correct for this typical situation. Attempting to use ESMF_COMPILER=gfortran when the C++ compiler is actually clang++ now issues an error message.

Known Issues

• Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.

Platform-specific issues:

• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default, ESMF uses the C++11 standard and cannot be downgraded. If you run into build issues due to the C++11 dependency, you must make sure a GCC>=4.8 is loaded.
• On Darwin, with the GNU gfortran+gcc combination, when building MPICH3 from source, it is important to s...

ESMF 8.3.1

Overview

ESMF 8.3.1 is a patch release that fixes a number of issues that were noticed after the 8.3.0 release. While most of the issues are minor, they have been reported as problems under specific user applications. The 8.3.1 patch release provides a path for affected applications to upgrade and use an official ESMF release instead of a beta tag in the 8.4.0 series currently in development.

As a patch release, 8.3.1 does not introduce new features. Applications that work fine with 8.3.0, or are already on a 8.4.0 beta tag, need not upgrade to the 8.3.1 patch. For these cases an upgrade to 8.4.0, when officially released, is the recommended next step. All of the fixes in the 8.3.1 release are also available in beta tag v8.4.0b11 and newer.

Release Notes

• This patch release is backward compatible with ESMF 8.3.0.
• No bit-for-bit changes are expected for this release compared to ESMF 8.3.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• The ESMF communication layer now supports single messages that are above the previous 2GiB limit. This applies to the direct usage of ESMF_VM communication calls, but also extends to ESMF_RouteHandle based communication methods: Regrid, Redist, Halo, and SMM. User applications have been observed to push over the previous 32-bit message limit when each PET addresses a substantial amount of memory and calls into the ESMF communication methods. Before this fix, a user could experience application crashes in the MPI layer due to ESMF attempting to send messages that exceed the 32-bit size limit.
• An issue inside ESMF_Info was fixed in which the data type and precision of attributes was not properly set when querying for an attribute through ESMF_InfoGet(). This led to a downstream issue of writing NetCDF attributes with a precision inconsistent with the attribute's precision inside the ESMF_Info object.
• Fixed an issue in which the IO layer was incorrectly querying for the number of compute cores on a node. In some cases, the bug led to application hangs during IO operations (e.g. ArrayWrite) that span multiple nodes.
• An issue was fixed in the IO layer that occurred when reading through PNetCDF into a destination with repeating elements (e.g. halo points). When used to read a Mesh from file, this problem caused bad coordinate values to be set in the created mesh. This in turn led to very poor performance when subsequently using the Mesh in a regrid weight generation operation.
• The IO performance and memory requirement for the Read() operation was significantly improved by eliminating a costly check only relevant for Write() operations.
• An issue observed under Darwin M1 systems with ESMP_Initailize() called by ESMPy was fixed.
• The ESMPy syntax was corrected to allow for calls with property accessor Manager.local_pet.
• The internal MOAB library included with ESMF now builds under old GCC 5.4.x.

Known Issues

• Same as ESMF 8.3.0.

Documentation

• ESMF Reference Manual for Fortran
• ESMF Reference Manual for C
• ESMF User Guide
• NUOPC Layer Reference
• Building a NUOPC Model
• ESMPy Doc

Tables

• Regridding Status Table
• Supported Platform Table

ESMF 8.3.0

Overview

The 8.3.0 release of ESMF implements a number of incremental improvements and bug fixes across the library. Highlights of the 8.3.0 release are outlined in the following paragraphs. A detailed list of release notes is provided further down.

On the code management side, ESMF has aligned its tagging scheme with the standard convention used by many other packages on GitHub. Standard tags now start with the lowercase letter “v”, followed by the version triplet. For example, the tag for release 8.3.0 on the ESMF GitHub repository is “v8.3.0”. Beta snapshots leading to a future release have the same root, followed by the lowercase letter “b” and a two digit snapshot number. E.g. v8.3.0b17 was the last 8.3.0 beta snapshot tag before the official release tag.

ESMF uses a library called ParallelIO (PIO) for its internal I/O operations, such as reading in mesh files and writing out fields. During this release, the version of PIO used internally was upgraded from a very outdated 1.x version to version 2.5. A new option was also added to the ESMF build system to allow linking to an external build of the PIO library.

Built on top of the PIO upgrade, the ESMF_MeshCreate() method that reads a mesh from file was re-implemented. It now reads the mesh coordinate information in a fully distributed way. This reduces the memory footprint dramatically, allowing the creation of much larger meshes from file than before.

Further progress was made toward the full adoption of MOAB as the internal mesh representation in ESMF. The internal MOAB library, included with ESMF, was updated to version 5.3, and combinatorial testing was added to the ESMF testing framework to ensure consistency and backward compatibility between the native mesh implementation and the MOAB-based implementation. Several consistency issues were resolved as a result of the new testing. By default, ESMF 8.3.0 still uses the native mesh implementation internally. As in previous releases, users can enable the MOAB-based implementation at run-time by calling ESMF_MeshSetMOAB().

Support for dynamically changing grid coordinates (e.g. storm following grids) was added to ESMF. The ESMF_GridCreate() method that creates a new Grid from an existing Grid with new DistGrid was extended to optionally return a RouteHandle object. The RouteHandle allows subsequent calls into the new ESMF_GridRedist() method to efficiently redistribute the coordinate values from the original source grid to the new destination grid. NUOPC_Connector support for handling changing grid coordinates is not available in this release but will be added in a future release.

Two issues were encountered and addressed in the ESMF_XGrid implementation. First, element areas can now be set in a Field built on an XGrid by using the ESMF_FieldRegridGetArea() method. Previously, this capability was only supported for a Field built on a Grid or Mesh. Second, the algorithm used to generate interpolation weights was improved to guarantee that exchange grid cells are exactly overlapping exactly one cell on each side. Prior to this change, small numerical errors prevented this property from holding and resulted in small remapping errors.

Release Notes

• This release is backward compatible with the last release ESMF 8.2.0, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. The entire list of API changes is summarized in a table showing interface changes since ESMF 8.2.0, including the rationale and impact for each change.
• No bit-for-bit changes were observed for this release compared to release ESMF v8.2.0 with Intel compilers using “-O2 -fp-model precise”.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding.
• A new section was added to the NUOPC Reference Manual describing the use of NUOPC_AddNestedState() for the coupling of multiple nests or multiple data sets between components.
• The option to profile the execution time of each individual iteration through a NUOPC run sequence has been implemented in the Driver Component Metadata Profiling attribute. Setting the appropriate profiling bit results in a profile where the timing for each individual run sequence iteration is reported in the timing profile under a unique label. This information can be helpful for cases where the cost per iteration changes throughout the execution.
• An issue in the ESMF_StateReconcile() method used by ESMF and NUOPC to generate a consistent object view across multiple components was fixed. The optimization implemented in v8.1.0 introduced the unintended behavior of switching out geom objects (Grid, Mesh, etc.) for Fields contained in States that are used in multiple ESMF_StateReconcile() operations. The incorrect association of geom objects with Fields resulted in unexpected results during subsequent operations using those Fields, such as creating a RouteHandle for regridding.
• Progress was made in full adoption of MOAB as the internal mesh representation in ESMF. This includes updating the internal MOAB library included with ESMF to version 5.3.1 and the addition of combinatorial testing designed to ensure consistency and backward compatibility between the native mesh implementation and the MOAB-based implementation. Several consistency issues were resolved as a result of the new testing.
• The previous version of ESMF_MeshCreate() from a file used to read all the node coordinate information on every processor. For large mesh files this global read can lead to high memory consumption and prevent reading in certain large meshes entirely. To reduce the memory footprint, a fully distributed read of mesh coordinate information was implemented. This change allows the creation of much larger meshes from file.
• The nodeOwners argument for method ESMF_MeshCreate() and ESMF_MeshAddNodes() was made optional. This allows the user to defer specification of node ownership to ESMF in cases where a specific ownership assignment is not needed to match the application data distribution. When this argument is absent, ESMF generates a consistent assignment of node owners.
• The ESMF_GridCreate() method that creates a new Grid from an existing Grid with new DistGrid was extended to optionally return a RouteHandle object. The RouteHandle allows subsequent calls into the new ESMF_GridRedist() method to redistribute the coordinate values from the original source grid to the new destination grid. This feature supports efficient handling of dynamically changing grids between components.
• The implementation of the exchange grid (ESMF_XGrid) class that supports efficient conservative regridding between multiple grids on source and destination sides has been improved:
  • Element areas can now be set in a Field built on an XGrid by using the ESMF_FieldRegridGetArea() method. Previously, this capability was only supported for a Field built on a Grid or Mesh.
  • The algorithm used to generate interpolation weights was improved to guarantee that exchange grid cells are exactly overlapping exactly one cell on each side. Prior to this change, small numerical errors prevented this property from holding.
• Added the optional --checkFlag argument to ESMF_RegridWeightGen application. This flag allows the user to turn on more expensive error checking that may not be appropriate for an operational run. Initially this flag turns on a check for grid self-intersection during conservative regridding.
• The VM Epoch implementation now provides an option to reduce the memory pressure on the sending side PETs. By default, internal send buffers, once allocated, are kept until the VM is destroyed. This can lead to high memory pre...

ESMF 8.2.0

Overview

Starting with version 8.2.0, the ESMF team has moved to a more frequent release cadence with new releases anticipated approximately every six months. This approach helps to ensure that new features, bug fixes, and optimizations are available more frequently in official releases of ESMF.

Highlights of the 8.2.0 release are outlined below. A detailed list of release notes is also provided below.

The NUOPC run sequence feature has proven a viable formalism to capture and express the control- and data-flow among the components of a wide range of coupled applications. Recent application work has demonstrated the need for more succinctly specifying conditional execution of run sequence elements. This release extends the NUOPC RunSequence syntax to include Alarm Blocks. Alarm blocks allow the user to specify if certain run sequence elements should be called less frequently than the parent timestep.

Several groups have started implementing exchange grids in their modeling systems, including within NUOPC Mediators. To facilitate these efforts, the ESMF_XGrid support was extended in this release. It now includes the use of all ESMF regridding methods (bilinear, patch, etc.) and options (extrapolation, regridding status, etc.) when regridding to or from Fields built on exchange grids.

A number of issues were uncovered during the deployment of the ESMF-managed threading and resource control features. This release addresses these issues, and the NUOPC level support for resource control and handling of threaded components is now more robust and has been demonstrated in several large-scale applications. This feature allows model components to independently set OpenMP threading levels so that all components in a coupled system are best utilizing available HPC resources, based on their individual scaling profiles.

The VMEpoch feature is an important communication optimization used by the NUOPC_Connector, and by some applications directly. This release fixes a problem that was encountered when using VMEpoch with any of the Redist() methods. The release also addresses an out-of-memory issue that can be triggered when the sending side runs many iterations ahead of the receiving side, by introducing automatic message throttling. Finally, a new reference manual section is available that describes the use of VMEpoch for asynchronous RouteHandle communications.

The process of replacing the native ESMF mesh implementation with the MOAB library, developed by the U.S. Department of Energy, is continuing. This release makes the MOAB mesh backend available to ESMPy users by calling Manager.set_moab(). This option allows users to test the impacts of using the MOAB mesh backend instead of the default native mesh through ESMPy.

Release Notes

• This release is backward compatible with the last major release update, ESMF 8.1.0 and patch release ESMF 8.1.1, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. The entire list of API changes is summarized in a table showing interface changes since ESMF_8_1_0, including the rationale and impact for each change.
• No bit-for-bit changes were observed for this release compared to release ESMF 8.1.0 and patch release ESMF 8.1.1, with Intel compilers using “-O2 -fp-model precise”. However, the release contains code changes to the regridding implementation that have the potential to lead to bit-for-bit changes in regridding weights. Any release item with the potential to introduce a bit-for-bit change is indicated in the respective release note.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding.
• The NUOPC RunSequence syntax was extended to support Alarm Blocks. An alarm block specifies the time interval at which the elements within the block are executed. This adds additional flexibility to the RunSequence approach, e.g. to write restart files at certain intervals that are multiples of the parent timestep.
• Fields created on XGrids can now be used as either source, destination, or both when calling the general ESMF regrid methods (ESMF_FieldRegridStore(), ESMF_FieldRegrid(), ESMF_FieldBundleRegridStore(), ESMF_FieldBundleRegrid()). This enables the use of all ESMF regridding methods (bilinear, patch, etc.) and options (extrapolation, regridding status, etc.) when regridding to or from Fields on an XGrid. Prior to this release, regridding to or from Fields on an XGrid was only supported when going from one of the grids used to originally create the XGrid. Also, only conservative methods were supported.
• A change in the 3D spherical bilinear weight calculation to handle more complex cells lead to a decrease in performance in releases 8.0.0, 8.1.0, and 8.1.1. The current release restores the performance to the level of ESMF 7.1.0r, and better, while retaining support for the complex cells. (Note that this change has the potential to introduce round off level changes in weights calculated for the 3D spherical bilinear method compared to previous ESMF releases. However, bit-for-bit testing with the Intel compiler using “-O2 -fp-model precise” did not detect any changes.)
• A number of issues that were found with ESMF-managed threading under real application usage, as released with ESMF 8.1.0, have been addressed: (1) PETs that execute a threaded component are no longer instantiated as Pthreads by default but instead execute under the original MPI process. This resolves the issue of not being able to set an unlimited stack size. (2) Issues within the automatic garbage collection of ESMF objects have been resolved, which lead to memory corruption issues during ESMF_Finalize() when Grids or Meshes were transferred between threaded components. (3) Thread affinities and number of OpenMP threads are reset when exiting from a threaded component method, and global resource control can be turned on/off via the optional argument globalResourceControl during ESMF_Initialize().
• It is now possible to override the defaults of a number of global ESMF settings by specifying an ESMF_Config file during ESMF_Initialize(). This is particularly useful for adjusting log specific settings, or to turn on/off resource control on the global VM.
• A new section was added to the ESMF Reference Manual that discusses use of VMEpoch for asynchronous RouteHandle communications.
• The VMEpoch feature allows sending PETs to fill the message queue up to the limit set by the MPI implementation. For message sizes where an MPI implementation chooses to use the EAGER protocol, this can lead to memory exhaustion on the receiving PETs. To prevent this issue, VMEpoch now limits the number of outstanding send cycles to ten by default. This default can be overridden by the user through the optional argument throttle to ESMF_VMEpochEnter().
• The process of replacing the native ESMF mesh implementation with the MOAB library is continuing. The MOAB mesh backend is now available to ESMPy by calling Manager.set_moab(). This allows the user to test ESMPy regridding features with the new MOAB backend in preparation for MOAB becoming the default. Manager.moab returns a boolean value to indicate if the MOAB backend is currently in use. The default is to use the native ESMF mesh backend.

Known Bugs

• The ESMF_XGrid construction can lead to degenerate cells in cases where the source and destination grids have edges that are almost the same. Often these cells don't produce weights and are benign, but when weights are produced, they can lead to low accuracy results when transferring data to/from the XGrid.

• Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.

  
  Platform-specific bugs:

• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default, ESMF uses the C++11 standard a...

ESMF 8.1.1

Overview

The 8.1.1 is a patch release that fixes a major regression in 8.1.0. The regression is in the area of user level OpenMP thread handling, where we received several reports of reduced performance. Performance in these cases has been restored with ESMF 8.1.1.

Release Notes

• This patch release is backward compatible with ESMF 8.1.0.
• No bit-for-bit changes are expected for this release compared to ESMF 8.1.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• This patch release corrects a regression in the handling of user level OpenMP threads. See the Known Bugs section for details.

Known Bugs

• Same as ESMF_8_1_0 with the following exceptions:
  • The issue with OpenMP thread count being reset to 1 within all ESMF components has been fixed.
  • The PETs of all ESMF components, and any potentially created OpenMP threads under such PETs, are no longer pinned automatically to any specific PEs.
• Platform-specific bugs:
  • The same as ESMF_8_1_0.