Parallel write + compression still fails in append mode (libnetcdf 4.8.0; hdf5 1.10.6)

[piyueh]

To report a non-security related issue, please provide:

• version (all installed using conda from channel conda-forge):

  • libnetcdf: 4.8.0 mpi_openmpi_h13b1d62_1
  • netcdf4: 1.5.6 mpi_openmpi_py38hc4e43b6_3
  • hdf5: 1.10.6 mpi_openmpi_hd9270da_1014
  • mpi4py: 3.0.3 py38h3e8e7aa_7
  • openmpi: 4.1.1 hbfc84c5_0

• environmental information:

  • Arch Linux
  • Anaconda environment
  • Python 3.8.8 hdb3f193_5

• a description of the issue with the steps needed to reproduce it

  The API documentation says:

  You can write compressed data in parallel only with netcdf-c >= 4.7.4 and hdf5 >= 1.10.3 (although you can read in parallel with earlier versions).

  However, with required library versions, I still wasn't able to do parallel write with compression if the dataset is opened in a mode (append mode). The code throws RuntimeError: NetCDF: HDF error.
  Here's the code to reproduce:

  #! /usr/bin/env python3
  from mpi4py import MPI
  from netCDF4 import Dataset
  
  dset = Dataset("./parallel_test.nc", "w", parallel=True, comm=MPI.COMM_WORLD)
  dset.createDimension("nx", 100)
  dset.createVariable("x", "f8", ("nx",), True)
  dset.close()
  
  dset = Dataset("./parallel_test.nc", "a", parallel=True, comm=MPI.COMM_WORLD)
  dset["x"][...] = 1.0
  dset.close()

  Tested with mpiexec -n 1 python test.py

[jswhit]

Confirmed.

[jswhit]

It is failing because netcdf-c is reporting that hdf5 doesn't support parallel filters (the return code from nc_def_var_deflate is NC_EINVAL). I don't understand why, since I thought this was enabled by default for parallel builds of HDF5 1.10.2 and later.

[jswhit]

In my case (using anaconda) when netcdf-c was built, it decided that the hdf5 library it was linking against did not support parallel filters. I suspect this is happening for you also.

[jswhit]

Upgrading the anaconda libnetcdf and hdf5 packages to libnetcdf 4.8.0-mpi_openmpi_h7cf8dfc_2 and hdf5 1.10.6-mpi_openmpi_hf18bdd3_1014 fixed this for me (on macos x). Here's a simple c program you can use to test whether your netcdf-c and hdf5 libraries support parallel filters

#include <stdio.h>
#include "netcdf.h"
#include "netcdf_par.h"
#include <mpi.h>

static int total_err = 0, err = 0;
/* This macro prints an error message with line number and name of
 * test program. */
#define ERR do {                                                        \
        fflush(stdout); /* Make sure our stdout is synced with stderr. */ \
        err++;                                                          \
        fprintf(stderr, "Sorry! Unexpected result, %s, line: %d\n",     \
                __FILE__, __LINE__);                                    \
        fflush(stderr);                                                 \
        return 2;                                                       \
    } while (0)

#define FILE "tst_parallel_zlib.nc"
#define NDIMS 3
#define DIMSIZE 24
#define QTR_DATA (DIMSIZE * DIMSIZE / 4)
#define NUM_PROC 4
#define NUM_SLABS 10

int
main(int argc, char **argv)
{
    /* MPI stuff. */
    int mpi_namelen;
    char mpi_name[MPI_MAX_PROCESSOR_NAME];
    int mpi_size, mpi_rank;
    MPI_Comm comm = MPI_COMM_WORLD;
    MPI_Info info = MPI_INFO_NULL;

    /* Netcdf-4 stuff. */
    int ncid, v1id, dimids[NDIMS];
    size_t start[NDIMS], count[NDIMS];

    int i, res;
    int slab_data[DIMSIZE * DIMSIZE / 4]; /* one slab */
    char file_name[NC_MAX_NAME + 1];

    /* Initialize MPI. */
    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &mpi_size);
    MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);
    MPI_Get_processor_name(mpi_name, &mpi_namelen);
    /*printf("mpi_name: %s size: %d rank: %d\n", mpi_name,
      mpi_size, mpi_rank);*/

    if (mpi_rank == 0)
    {
       printf("\n*** Testing parallel writes with zlib.\n");
       printf("*** testing simple write with zlib...");
    }

    /* Create phony data. We're going to write a 24x24 array of ints,
       in 4 sets of 144. */
    for (i = 0; i < DIMSIZE * DIMSIZE / 4; i++)
       slab_data[i] = mpi_rank;


    /* Create a parallel netcdf-4 file. */
    /*nc_set_log_level(3);*/
    /* sprintf(file_name, "%s/%s", TEMP_LARGE, FILE); */
    sprintf(file_name, "%s", FILE);
    if ((res = nc_create_par(file_name, NC_NETCDF4, comm, info, &ncid))) ERR;

    /* Create three dimensions. */
    if (nc_def_dim(ncid, "d1", DIMSIZE, dimids)) ERR;
    if (nc_def_dim(ncid, "d2", DIMSIZE, &dimids[1])) ERR;
    if (nc_def_dim(ncid, "d3", NUM_SLABS, &dimids[2])) ERR;

    /* Create one var. */
    if ((res = nc_def_var(ncid, "v1", NC_INT, NDIMS, dimids, &v1id))) ERR;

    /* Setting deflate only will work for HDF5-1.10.2 and later
     * versions. */
    res = nc_def_var_deflate(ncid, 0, 0, 1, 1);
    if (mpi_rank == 0)
    {
    printf("\nreturn code from nc_def_var_deflate = %d\n",res);
    if (res) ERR;
    }

    /* Write metadata to file. */
    if ((res = nc_enddef(ncid))) ERR;
    /* Set up slab for this process. */
    start[0] = mpi_rank * DIMSIZE/mpi_size;
    start[1] = 0;
    count[0] = DIMSIZE/mpi_size;
    count[1] = DIMSIZE;
    count[2] = 1;
    /*printf("mpi_rank=%d start[0]=%d start[1]=%d count[0]=%d count[1]=%d\n",
      mpi_rank, start[0], start[1], count[0], count[1]);*/

    /* Not necessary, but harmless. */
    if (nc_var_par_access(ncid, v1id, NC_COLLECTIVE)) ERR;

    for (start[2] = 0; start[2] < NUM_SLABS; start[2]++)
    {
       nc_set_log_level(3);
       /* Write slabs of phoney data. */
       if (nc_put_vara_int(ncid, v1id, start, count, slab_data)) ERR;
    }

    /* Close the netcdf file. */
    if ((res = nc_close(ncid))) ERR;
    /* Shut down MPI. */
    MPI_Finalize();

    return 0;
}

which should produce

mpirun --oversubscribe -np 4 ./a.out

*** Testing parallel writes with zlib.
*** testing simple write with zlib...
return code from nc_def_var_deflate = 0

[jswhit]

I see now after reading the original post more carefully that the problem i was having was different than yours. After fixing the problem I described above, I still see the error you reported when using mode='a'. This error can be fixed by putting the variable in 'collective mode' before writing to it, like this

dset = Dataset("./parallel_test.nc", "a", parallel=True, comm=MPI.COMM_WORLD)
v = dset["x"]
v.set_collective(True)
v[...] = 1.0
dset.close()

I've updated the parallel IO docs to mention this in #1109

[piyueh]

Thanks for looking into the issue. Yes, setting it to collective mode works.

Just out of curiosity, why, even when using one MPI process, non-collective I/O still not working? I thought there's no difference between collective and non-collective I/O when using one single MPI process. (Sorry, I'm not familiar with parallel I/O)

Another question is, does using compression + collective I/O add overhead to time cost? For example, if each rank writes to a variable without overlapping (i.e., each rank owns a slice of the variable, and slices don't overlap), is it better to use non-compression + non-collective writing?

[jswhit]

The requirement that parallel compressed writes happen in collective mode comes from the HDF5 library. I don't know why, nor do I know what the performance implications are - that would be a question for the HDF5 group.

[jswhit]

closed by PR #1109