ROMS Parallel IO

Introduction

Generally, writing is a more frequent and complicated operation than reading. There are four strategies for writing:

1. Single file, single writer: Serial I/O in non-parallel or parallel applications. It is the default strategy in ROMS using the NetCDF3 or NetCDF4 libraries.
2. Single file, multiple writers: Parallel I/O allows each partition tile (task) to write its data into a single file. This capability is achieved in ROMS by activating PARALLEL_IO and HDF5. It is only possible with NetCDF4/HDF5 libraries. However, the performance is questionable because option (1) is faster in some applications.
3. Single file, collective writers: Parallel I/O with either one or a subset of process performing I/O operations. It can be synchronous or asynchronous. In ROMS, this capability uses the Paralle-IO (PIO) library developed at NCAR. It is available when the PIO_LIB CPP option is activated.
4. Multiple files, multiple writers: Parallel I/O in which each distributed-memory or shared-memory tile decomposition writes its data into separate NetCDF files. However, post-processing is required to pack the data into a single file. This feature is undesirable in applications running on hundreds of processors. This capability is unavailable in ROMS because it is inefficient and complicates ensemble and 4D-Var drivers.

PIO Library

The PIO library has two modes of parallel I/O, which are only possible with the MPI applications.

• Synchronous: MPI intra-communication mode. A subset of processes or all processors perform both I/O and computations. The users specify the number of I/O tasks, PIO_IOTASKS parameter, and how they are distributed across HPC nodes as a function of the ROMS MPI-communicato object, OCN_COMM_WORLD. It is often desirable to shift the first I/O tasks using the PIO_BASE parameter from the first computational task, since it has higher memory requirements than other processes. If the MPI processes are scattered across several computer nodes, it is highly recommended to spread all I/O tasks evenly across all nodes using the PIO_STRIDE parameter. Avoid all I/O processes occupying the same node. This strategy is illustrated below.

  • In the Box Rearrangement, data is continuously rearranged from computational to I/O processes according to the data ordering in the file. Since data ordering between computational and I/O partitions may differ, the rearrangement will require all-to-all MPI communications. Each computing tile may transfer data to one or more I/O processes.
  • In the Subset Rearrangement, each I/O process is associated with a subset of computing processes. The computing tile sends its data to a unique I/O process. The data on all I/O processes may be more fragmented than the ordering on disk, which may increase the MPI communication to the storage medium. However, this method scales better since all-to-all MPI communications are unnecessary.

• Asynchronous: MPI inter-communications mode. The I/O tasks are a disjoint set of dedicated I/O processes that do not perform computations. It is possible to have groups of computational units running separate models, like coupling, where all the I/O data is sent to dedicated processes. In ROMS, this I/O mode is activated with ASYNCHRONOUS_PIO and DISJOINTED communicators. It's not very easy and requires further work.

ROMS PIO Configuration and Implementation

The PIO configuration for a particular application is set in the ROMS standard input file roms.in. It depends on the application and the computational resources. The user needs to experiment with these parameters to evaluate the performance.

Notice that the standard NetCDF3/NetCDF4 and PIO libraries coexist in a ROMS executable. We can choose which library is used during reading (INP_LIB) and writing (OUT_LIB). The ROMS design is very flexible, as shown below, depending, for example, on the value of HIS(ng)%IOtype in the derived TYPE T_IO structure for each ROMS input or output file, which can have a value of either io_nf90 or io_pio.

I/O Descriptors

To accelerate the reading and writing with PIO, ROMS declares and initializes the I/O descriptors once at the beginning of the computations. They are used in the parallel decomposition mapping from computational to I/O processes and vice versa for all ROMS C-type variables, array ranks, and array kinds. It specifies how data in memory should be written to or read from disk. The I/O descriptors are declared in mod_pio_netcdf.F and initialized in module set_pio.F, routine set_iodecomp.

Delayed Synchronization

We could use option PIO_METHOD=3 in roms.in to activate parallel read and serial write of NetCDF4 files. However, the files are not synchronized (written) to disk until a call to PIO_closefile is made at the end of the simulation.

The reason PIO and NetCDF4 do not synchronize data (I/O buffering and caching) to disk until the file is closed is primarily to maximize performance by keeping data in memory. Writing to disk is slow; buffering allows multiple write operations to be batched into a single, efficient operation. Data is automatically synchronized when PIO_closefile is called.

The key reasons for delayed synchronization are:

• Performance Optimization: NetCDF4 caches data in memory, which is significantly faster than writing directly to the disk for every PIO_write_darray call.
• Metadata Management: The final file structure, including internal indexing and file system metadata, is best finalized at the end to ensure file integrity.
• Reduced Disk Load: Delaying the synchronization reduces the number of calls to the underlying file system, improving overall parallel I/O efficiency.

Alternatively, one could use nc_sync() to manually call the synchronization function to force pending writes to disk, although this can be slow. Or use NC_SHARE when opening or creating a file to enable more frequent synchronization, though this may incur a performance penalty.

Caution

If a program terminates abnormally before calling PIO_closefile, the data in the buffer may be lost.