Possible issue in MeshCreateFromFile() with PIO_IOTYPE_PNETCDF reading CDF5 mesh files

[baoshanning]

Requirements

• Reviewed ESMF User's Guide
• Searched GitHub Discussions

Affiliation(s)

National space science center, Chinese Academy of Science

ESMF Version

v8.9.1

Issue

Hello ESMF developers and users,

I recently started using CTSM and encountered a problem that appears to originate from the ESMF mesh I/O path. After some debugging, I was able to narrow it down to a specific code path, but I'm not sure whether this is a bug, a configuration issue, or expected behavior. I would appreciate any guidance.

Environment:
OS: WSL-Ubuntu 24.04
Compiler: gfortran
MPI: OpenMPI

Build configuration:
ESMF_PIO=internal
ESMF_PNETCDF=standard
ESMF_NETCDF=split
NetCDF:
netCDF-C 4.9.2
netCDF-Fortran (Ubuntu package)
PnetCDF: 1.12.3

Problem:
While running the CTSM regression test "./create_test SMS_Ln9.f09_g17.X", the model aborts during ESMF_MeshCreateFromFile() with the following error:
PIO/PNetCDF error = Unknown Error:
Unrecognized error code -500

The error originates from "src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C" around the call to PIOc_openfile(). The mesh being read is "fv0.9x1.25_141008_polemod_ESMFmesh.nc". The file appears to be valid since "ncdump -k" reports 'cdf5'. Both "ncdump" and "ncmpidump" can read the file successfully without any errors.

Investigation

I traced the failure into "src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C" where the code currently selects

**
#ifdef ESMF_PNETCDF
    int pio_type = PIO_IOTYPE_PNETCDF;
#else
    int pio_type = PIO_IOTYPE_NETCDF;
#endif
**

before calling "PIOc_openfile(...)".

As an experiment, I changed the two occurrences in MeshCreateFromFile() to
**int pio_type = PIO_IOTYPE_NETCDF;**
and I rebuilt ESMF, and reran the CTSM regression test.
After this change, the test completed successfully without any additional modifications.

This suggests that, at least in my environment, the failure is specific to the PIO_IOTYPE_PNETCDF path, while using PIO_IOTYPE_NETCDF works correctly for the same CDF5 mesh file.
However, I do not intend to maintain this local modification; I used it only as a debugging experiment to help isolate the problem.

Questions:
Is this a known issue?
Has this already been fixed in ESMF 8.9.2, 8.9.3, or the current development branch?
Is PIO_IOTYPE_PNETCDF expected to support reading CDF5 mesh files in this code path?
Is there a recommended runtime or build configuration that avoids this problem without modifying the ESMF source?
If this behavior is expected, could you explain why PIO_IOTYPE_PNETCDF is preferred here over PIO_IOTYPE_NETCDF?

I'd appreciate any suggestions for the recommended solution.
Thank you very much for your time.

@oehmke

[oehmke]

Hi @baoshanning, This isn't a known issue. We can look at that error code and see if we can figure out what's going on. Would you run: make info at the top of the ESMF library and send us the output, so we can see the full build configuration?

It's possible that there's an issue or inconsistency with the PNetCDF library. After running make info as described above, would you also mind trying the unmodified 8.9.1 ESMF library and rebuild with ESMF_PNETCDF not set and see if that works with your file? That should essentially do what you did by your ESMF modifications, so I want to make sure that's a good work around. Thanks!

[baoshanning]

Hi Oehmke,
Thank you very much for your reply.
As requested, I've attached the output of make info from the original ESMF build with PNETCDF enabled: make_info.txt
I also followed your suggestion and rebuilt the unmodified ESMF 8.9.1 with ESMF_PNETCDF unset:

unset ESMF_PNETCDF
unset ESMF_PNETCDF_INCLUDE
unset ESMF_PNETCDF_LIBPATH

make distclean
make -j4

The corresponding make info output is also attached:
make_info_noPNETCDF.txt
. After rebuilding, I reran the CTSM test: ./create_test SMS_Ln9.f09_g17.X --machine ubuntu-wsl --compiler gnu. The test completed successfully (PASS).
So I now have three reproducible cases:

1. Original ESMF 8.9.1 + ESMF_PNETCDF=standard

• Mesh I/O uses PIO_IOTYPE_PNETCDF
  

• CTSM Fails during ESMF_MeshCreateFromFile() with:
  

PIO/PNetCDF error = Unknown Error: Unrecognized error code -500

2. Original ESMF 8.9.1 with ESMF_PNETCDF unset

• No source code changes
  

• Mesh I/O falls back to NetCDF
  

• CTSM test passes successfully.
  

3. Original ESMF 8.9.1 with ESMF_PNETCDF=standard, but forcing PIO_IOTYPE_NETCDF in ESMCI_Mesh_FileIO.C

• CTSM test also passes successfully.
  

These suggest that the issue is specific to the PIO/PNETCDF code path when opening this CDF5 mesh file, rather than mesh file itself.
Do you have any suggestions on what I should investigate next?