Skip to content

ROMS Parallel IO

Jump to bottom
Hernan G. Arango edited this page Mar 3, 2026 · 2 revisions
image

Introduction

Generally, writing is a more frequent and complicated operation than reading. There are four strategies for writing:

  1. Single file, single writer: Serial I/O in non-parallel or parallel applications. It is the default strategy in ROMS using the NetCDF3 or NetCDF4 libraries.
  2. Single file, multiple writers: Parallel I/O allows each partition tile (task) to write its data into a single file. This capability is achieved in ROMS by activating PARALLEL_IO and HDF5. It is only possible with NetCDF4/HDF5 libraries. However, the performance is questionable because option (1) is faster in some applications.
  3. Single file, collective writers: Parallel I/O with either one or a subset of process performing I/O operations. It can be synchronous or asynchronous. In ROMS, this capability uses the Paralle-IO (PIO) library developed at NCAR. It is available when the PIO_LIB CPP option is activated.
  4. Multiple files, multiple writers: Parallel I/O in which each distributed-memory or shared-memory tile decomposition writes its data into separate NetCDF files. However, post-processing is required to pack the data into a single file. This feature is undesirable in applications running on hundreds of processors. This capability is unavailable in ROMS because it is inefficient and complicates ensemble and 4D-Var drivers.

PIO Library

The PIO library has two modes of parallel I/O, which are only possible with the MPI applications.

  • Synchronous: MPI intra-communication mode. A subset of processes or all processors perform both I/O and computations. The users specify the number of I/O tasks, PIO_IOTASKS parameter, and how they are distributed across HPC nodes as a function of the ROMS MPI-communicato object, OCN_COMM_WORLD. It is often desirable to shift the first I/O tasks using the PIO_BASE parameter from the first computational task, since it has higher memory requirements than other processes. If the MPI processes are scattered across several computer nodes, it is highly recommended to spread all I/O tasks evenly across all nodes using the PIO_STRIDE parameter. Avoid all I/O processes occupying the same node. This strategy is illustrated below.

    • In the Box Rearrangement, data is continuously rearranged from computational to I/O processes according to the data ordering in the file. Since data ordering between computational and I/O partitions may differ, the rearrangement will require all-to-all MPI communications. Each computing tile may transfer data to one or more I/O processes.
    • In the Subset Rearrangement, each I/O process is associated with a subset of computing processes. The computing tile sends its data to a unique I/O process. The data on all I/O processes may be more fragmented than the ordering on disk, which may increase the MPI communication to the storage medium. However, this method scales better since all-to-all MPI communications are unnecessary.
image image
  • Asynchronous: MPI inter-communications mode. The I/O tasks are a disjoint set of dedicated I/O processes that do not perform computations. It is possible to have groups of computational units running separate models, like coupling, where all the I/O data is sent to dedicated processes. In ROMS, this I/O mode is activated with ASYNCHRONOUS_PIO and DISJOINTED communicators. It's not very easy and requires further work.
image

ROMS PIO Configuration and Implementation

The PIO configuration for a particular application is set in the ROMS standard input file roms.in. It depends on the application and the computational resources. The user needs to experiment with these parameters to evaluate the performance.

image

Notice that the standard NetCDF3/NetCDF4 and PIO libraries coexist in a ROMS executable. We can choose which library is used during reading (INP_LIB) and writing (OUT_LIB). The ROMS design is very flexible, as shown below, depending, for example, on the value of HIS(ng)%IOtype in the derived TYPE T_IO structure for each ROMS input or output file, which can have a value of either io_nf90 or io_pio.

image image

I/O Descriptors

To accelerate the reading and writing with PIO, ROMS declares and initializes the I/O descriptors once at the beginning of the computations. They are used in the parallel decomposition mapping from computational to I/O processes and vice versa for all ROMS C-type variables, array ranks, and array kinds. It specifies how data in memory should be written to or read from disk. The I/O descriptors are declared in mod_pio_netcdf.F and initialized in module set_pio.F, routine set_iodecomp.

image

Delayed Synchronization

We could use option PIO_METHOD=3 in roms.in to activate parallel read and serial write of NetCDF4 files. However, the files are not synchronized (written) to disk until a call to PIO_closefile is made at the end of the simulation. Similar behavior applies for option PIO_METHOD=4 to activate parallel read and parallel write of NetCDF4 files.

The reason PIO and NetCDF4 do not synchronize data (I/O buffering and caching) to disk until the file is closed is primarily to maximize performance by keeping data in memory. Writing to disk is slow; buffering allows multiple write operations to be batched into a single, efficient operation. Data is automatically synchronized when PIO_closefile is called.

The key reasons for delayed synchronization are:

  • Performance Optimization: NetCDF4 caches data in memory, which is significantly faster than writing directly to the disk for every PIO_write_darray call.
  • Metadata Management: The final file structure, including internal indexing and file system metadata, is best finalized at the end to ensure file integrity.
  • Reduced Disk Load: Delaying the synchronization reduces the number of calls to the underlying file system, improving overall parallel I/O efficiency.

Note

If you issue an ncdump -k command while the model is running, you get the following error because the file is not properly closed yet. For example:

% ncdump -k usec3km_roms_his_20190827.nc
% ncdump: usec3km_roms_his_20190827.nc: NetCDF: HDF error

Alternatively, one could use nc_sync() to manually call the synchronization function to force pending writes to disk, although this can be slow. Or use NC_SHARE when opening or creating a file to enable more frequent synchronization, though this may incur a performance penalty.

Caution

If a program terminates abnormally before calling PIO_closefile, the data in the buffer may be lost.

USEC 3km Benchmark

The Nonlinear forward solution for the USEC 3km application (561x243x50) is used to benchmark ROMS I/O with the NetCDF NF90 and PIO libraries. It is run on a desktop Linux box (16 CPUs and 1 GPU with 384 CUDA cores, NVIDIA), with 12 CPUs and a 3x4 partition, compiled with ifort (Spack-Stack 1.9). Please investigate the proper configuration for your computer systems. The USEC 3km configuration has the following characteristics:

  • The ROMS NLM kernel is run in double precision, and the output files are written in double precision.
  • A three-day simulation with 720 timesteps.
  • The option GRID_EXTRACT is activated to extract a coarser 6km trajectory (281x122x50) by decimation to be used as a background trajectory in the 4D-Var data assimilation algorithm and written into usec6km_roms_fwd_20190827.nc.
  • The option OUT_NETCDF4 activates writing all output files with the NetCDF4/HDF5 format in ROMS. These files can be compressed by activating DEFLATE. In addition, the user can activate DELAYED_SYNC_NF90 to avoid flushing the buffers to disk each time an unlimited-dimension record is written, thereby improving efficiency. It is up to the NetCDF4/HDF5 libraries to determine their flushing frequency for performance. The files are accessible after they are closed.
  • The option VERIFICATION is activated to interpolate the model solution at the observation location, H(x) operator. They are written into the output file usec3km_roms_mod_20190827.nc.
  • The option DEFLATE may or may not be activated to compress only output NetCDF4/HDF5 files. It is not possible to compress NetCDF-3 files with a 64-bit offset.
  • The options DELAYED_SYNC_NF90 and DELAYED_SYNC_NF90 are activated to avoid synchronization to disk for each output unlimited time dimension records for efficiency. They are only possible on NetCDF-4/HDF5 output files.
  • ROMS supports both NetCDF NF90 (OUT_NETCDF4) and PIO (PIO_LIB) libraries which are controlled by the inp_lib and out_lib configuration parameters.
  • The history, quicksave, and extracted fields by decimation are saved hourly (73 records).
  • The table below shows the effects of file compression with the DEFLATE option and delayed synchronization. In Case 1, the size of all output files is reduced by 35% to 26.593 GB, down from 41.163 GB. However, the CPU (elapsed time) takes about 8.5 minutes longer, resulting in a 44% increase. Thus, the decision is about what is more important: disk space, file transfer bandwidth to storage, CPU time, or pricing in cloud computer services. As the applications become larger, the CPU penalty in compression tends to decrease. However, you'll need to evaluate your application benchmarks.
Case inp_lib out_lib PIO
Method		 File Type Deflate Delayed Sync Files Size Elapsed Time
mm:ss
1 2
PIO		 2
PIO		 3 netCDF-4 Yes PIO, NF90 26.593 GB 36:56
2 2
PIO		 2
PIO		 3 netCDF-4 No PIO, NF90 41.163 GB 25:36
3 2
PIO		 2
PIO		 2 64-bit offset N/A N/A 38.126 GB 28:12
4 2
PIO		 1
NF90		 2 netCDF-4 No N/A, NF90 41.163 GB 25:47
5 2
PIO		 1
NF90		 2 64-bit offset N/A N/A 40.721 GB 25:59
  • Case 1:
    • Options: DEFLATE, DELAYED_SYNC_NF90, DELAYED_SYNC_PIO, GRID_EXTRACT, OUT_DOUBLE, OUT_NETCDF4, PIO_LIB, VERIFICATION
    • PIO Parameters: inp_lib = 2, out_lib = 2, pio_method = 3, pio_NumIOtasks = 2
    • Compression Parameters: shuffle = 1, deflate = 1, deflate_level = 1
    • Elapsed time: 6:00:23 to 6:36:56 (36 min, 33 sec)
    • Profiling: CPU Average = 2041.357 sec, CPU Total = 24496.286 sec (12 PETs)
     5560607736 Feb 22 18:37 usec6km_roms_fwd_20190827.nc
      601028687 Feb 22 18:36 usec3km_roms_qck_20190827.nc
    21404576550 Feb 22 18:36 usec3km_roms_his_20190827.nc
       29536640 Feb 22 18:34 usec3km_roms_mod_20190827.nc
      288549636 Feb 22 18:34 usec3km_roms_rst_20190827.nc

    20.413G   usec3km_roms_his_20190827.nc   (PIO,  netCDF-4)
    29M	      usec3km_roms_mod_20190827.nc   (PIO,  netCDF-4)
    574M      usec3km_roms_qck_20190827.nc   (PIO,  netCDF-4)
    276M      usec3km_roms_rst_20190827.nc   (PIO,  netCDF-4)
    5.304G    usec6km_roms_fwd_20190827.nc   (NF90, netCDF-4)
    26.593G   total
  • Case 2:
    • Options: DELAYED_SYNC_NF90, DELAYED_SYNC_PIO, GRID_EXTRACT, OUT_DOUBLE, OUT_NETCDF4, PIO_LIB, VERIFICATION
    • PIO Parameters: inp_lib = 2, out_lib = 2, pio_method = 3, pio_NumIOtasks = 2
    • Compression Parameters: shuffle = 1, deflate = 1, deflate_level = 1 (no compression, DEFLATE is off)
    • Elapsed time: 12:24:36 to 12:50:12 (25 min, 36 sec)
    • Profiling: CPU Average = 1523.794 sec, CPU Total = 18285.524 sec (12 PETs)
     8395341139 Feb 23 12:50 usec6km_roms_fwd_20190827.nc
      973162965 Feb 23 12:50 usec3km_roms_qck_20190827.nc
    33296933285 Feb 23 12:50 usec3km_roms_his_20190827.nc
       29536640 Feb 23 12:50 usec3km_roms_mod_20190827.nc
      466897660 Feb 23 12:50 usec3km_roms_rst_20190827.nc

    31.755G   usec3km_roms_his_20190827.nc   (PIO,  netCDF-4)
    29M       usec3km_roms_mod_20190827.nc   (PIO,  netCDF-4)
    929M      usec3km_roms_qck_20190827.nc   (PIO,  netCDF-4)
    446M      usec3km_roms_rst_20190827.nc   (PIO,  netCDF-4)
    8.007G    usec6km_roms_fwd_20190827.nc   (NF20, netCDF-4)
    41.163G   total
  • Case 3:
    • Options: DEFLATE, DELAYED_SYNC_NF90, DELAYED_SYNC_PIO, GRID_EXTRACT, OUT_DOUBLE, OUT_NETCDF4, PIO_LIB, VERIFICATION
    • PIO Parameters: inp_lib = 2, out_lib = 2, pio_method = 2, pio_NumIOtasks = 2
    • Compression Parameters: shuffle = 1, deflate = 1, deflate_level = 1
    • Elapsed time: 6:40:32 to 7:08:44 (28 min, 12 sec)
    • Profiling: CPU Average = 1679.784 sec, CPU Total = 20157.407 sec (12 PETs)
     5560607736 Feb 22 19:08 usec6km_roms_fwd_20190827.nc
      972942020 Feb 22 19:08 usec3km_roms_qck_20190827.nc
    32953746000 Feb 22 19:08 usec3km_roms_his_20190827.nc
       29509212 Feb 22 19:08 usec3km_roms_mod_20190827.nc
      460977056 Feb 22 19:08 usec3km_roms_rst_20190827.nc

    31.428G   usec3km_roms_his_20190827.nc   (PIO,  64-bit offset)
    29M	      usec3km_roms_mod_20190827.nc   (PIO,  64-bit offset)
    928M      usec3km_roms_qck_20190827.nc   (PIO,  64-bit offset)
    440M      usec3km_roms_rst_20190827.nc   (PIO,  64-bit offset)
    5.304G    usec6km_roms_fwd_20190827.nc   (NF90, netCDF-4)
    38.126G   total
  • Case 4:
    • Options: DELAYED_SYNC_NF90, DELAYED_SYNC_PIO, GRID_EXTRACT, OUT_DOUBLE, OUT_NETCDF4, PIO_LIB, VERIFICATION
    • PIO Parameters: inp_lib = 2, out_lib = 1, pio_method = 2, pio_NumIOtasks = 2
    • Compression Parameters: shuffle = 1, deflate = 1, deflate_level = 1 (no compression, DEFLATE is off)
    • Elapsed time: 7:23:54 to 7:49:41 (25 min, 47 sec)
    • Profiling: CPU Average = 1541.751 sec, CPU Total = 18501.013 sec (12 PETs)
     8395341139 Feb 22 19:49 usec6km_roms_fwd_20190827.nc
      973165553 Feb 22 19:49 usec3km_roms_qck_20190827.nc
    33296935876 Feb 22 19:49 usec3km_roms_his_20190827.nc
       29536640 Feb 22 19:49 usec3km_roms_mod_20190827.nc
      466900251 Feb 22 19:49 usec3km_roms_rst_20190827.nc

    31.755G   usec3km_roms_his_20190827.nc   (NF90, netCDF-4)
    29M       usec3km_roms_mod_20190827.nc   (NF90, netCDF-4)
    929M      usec3km_roms_qck_20190827.nc   (NF90, netCDF-4)
    446M      usec3km_roms_rst_20190827.nc   (NF90, netCDF-4)
    8.007G    usec6km_roms_fwd_20190827.nc   (NF90, netCDF-4)
    41.163G   total
  • Case 5:
    • Options: GRID_EXTRACT, OUT_DOUBLE, PIO_LIB, VERIFICATION
    • PIO Parameters: inp_lib = 2, out_lib = 1, pio_method = 2, pio_NumIOtasks = 2
    • Compression Parameters: shuffle = 1, deflate = 1, deflate_level = 1 (no compression, DEFLATE is off)
    • Elapsed time: 11:17:48 to 11:43:47 (25 min, 59 sec)
    • Profiling: CPU Average = 1553.637 sec, CPU Total = 18643.648 sec (12 PETs)
       29509212 Feb 23 11:43 usec3km_roms_mod_20190827.nc
      460977080 Feb 23 11:43 usec3km_roms_rst_20190827.nc
      972942044 Feb 23 11:43 usec3km_roms_qck_20190827.nc
     8281000400 Feb 23 11:43 usec6km_roms_fwd_20190827.nc
    32953746024 Feb 23 11:43 usec3km_roms_his_20190827.nc

    31.428G   usec3km_roms_his_20190827.nc   (NF90, 64-bit offset)
    29M       usec3km_roms_mod_20190827.nc   (NF90, 64-bit offset)
    928M      usec3km_roms_qck_20190827.nc   (NF90, 64-bit offset)
    440M      usec3km_roms_rst_20190827.nc   (NF90, 64-bit offset)
    7.898G    usec6km_roms_fwd_20190827.nc   (NF90, 64-bit offset)
    40.721G   total

East Coast Community Ocean Forecast System (ECCOFS)

Clone this wiki locally