The HDF5 asynchronous I/O VOL connector (async I/O VOL) allows applications to fully or partially hide the I/O time by overlapping it with the computation. This page provides more information on how applications can take advantage of it.
The async I/O VOL can be used in explicit
and implicit
modes.
To use the implicit
mode, see Implicit Mode
The recommended mode for applications to use the async I/O VOL is ``explicit mode'', which is described below.
Using the async I/O VOL is ``explicit mode'' requires modifying the application code to use HDF5 asynchronous I/O APIs. This includes three steps:
- adding the EventSet API calls,
- switching the existing HDF5 I/O calls to their asynchronous version
- ensuring data consistency.
The EventSet APIs allows tracking and inspecting multiple asynchronous I/O operations and avoids the burden for the developer to manage the individual operation. An event set (ID) is an in-memory object that is created by the application and functions similar to a "bag" holding request tokens from one or more asynchronous I/O operations. Every asynchronous I/O operation must be associated with an event set, thus the application must create one before the first HDF5 operation:
es_id = H5EScreate();
The application can then use this event set ID (es_id
) for all subsequent HDF5 I/O operations. One can also create multiple event set IDs for different operations. Because the HDF5 functions are non-blocking when async VOL is enabled, the application should also wait for all asynchronous operations to finish before exiting, with:
H5ESwait(es_id, H5ES_WAIT_FOREVER, &n_running, &op_failed);
The H5ESwait is also required before closing an event set, with:
H5ESclose(es_id);
The HDF5-1.13 and later versions provide an asynchronous version of all the HDF5 I/O operations. A complete list of them can be found in the API section. Applications need to switch their existing HDF5 function calls to their asynchronous version, which can be done by adding "_async" to the end of the HDF5 function name and adding an event set ID as the last parameter to the function parameter list. One can also maintain both the original synchronous calls and asynchronous with a MACRO and decides which to use at compile time, e.g.,:
#ifndef ENABLE_HDF5_ASYNC H5Fcreate(fname, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); #else H5Fcreate_async(fname, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT, es_id); #endif
Once switched to use the event set and asynchronous APIs, the application developer must re-examine the code to ensure data consistency as all the HDF5 I/O calls are non-blocking.
For data write, the application must not modify or free the write buffer until the write operation has been executed by the background thread. This can be done by calling H5ESwait
before the buffer reuse or free, or by double buffering with alternating one buffer for the asynchronous write and another for new data. Users may also choose to have async VOL to perform the double buffering automatically by adding -DENABLE_WRITE_MEMCPY
to the CFLAGS
in the Makefile, however, this requires double the buffer size in memory while the I/O operation is queued, and should be used with caution (the extra memory copy is automatically freed after the operation completes).
For data read, an H5ESwait
function is needed before the data is accessed for the asynchronous operation to complete and fill the user's buffer. It is recommended to issue asynchronous read operations as early as possible and has their execution overlap with other tasks of the application to.
An application may have the following HDF5 operations to write data:
// Synchronous file create fid = H5Fcreate(...); // Synchronous group create gid = H5Gcreate(fid, ...); // Synchronous dataset create did = H5Dcreate(gid, ..); // Synchronous dataset write status = H5Dwrite(did, ..); // Synchronous dataset read status = H5Dread(did, ..); ... // Synchronous file close H5Fclose(fid); // Continue to computation
which can be converted to use async VOL as the following:
// Create an event set to track async operations es_id = H5EScreate(); // Asynchronous file create fid = H5Fcreate_async(.., es_id); // Asynchronous group create gid = H5Gcreate_async(fid, .., es_id); // Asynchronous dataset create did = H5Dcreate_async(gid, .., es_id); // Asynchronous dataset write status = H5Dwrite_async(did, .., es_id); // Asynchronous dataset read status = H5Dread_async(did, .., es_id); ... // Asynchronous file close status = H5Fclose_async(fid, .., es_id); // Continue to computation, overlapping with asynchronous operations ... // Finished computation, Wait for all previous operations in the event set to complete H5ESwait(es_id, H5ES_WAIT_FOREVER, &n_running, &op_failed); // Close the event set H5ESclose(es_id); ...
Note
More details on using the explicit
mode are available at Explicit Mode