romio: make contig_access_count MPI_Count instead of int #6928

hzhou · 2024-02-27T23:38:30Z

Pull Request Description

It is potentially possible to have more than INT_MAX number of noncontig segments, thus contig_access_count need be typed as MPI_Count to prevent overflow.

Author Checklist

Provide Description
Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
Commits Follow Good Practice
Commits are self-contained and do not do two things at once.
Commit message is of the form: module: short description
Commit message explains what's in the commit.
Passes All Tests
Whitespace checker. Warnings test. Additional tests via comments.
Contribution Agreement
For non-Argonne authors, check contribution agreement.
If necessary, request an explicit comment from your companies PR approval manager.

hzhou · 2024-02-27T23:46:15Z

test:mpich/ch3/tcp
test:mpich/ch4/ofi

wkliao · 2024-03-01T21:35:38Z

Can ROMIO now handle a single request of size > INT_MAX made from a process rank?

hzhou · 2024-03-01T21:37:33Z

Can ROMIO now handle a single request of size > INT_MAX made from a process rank?

It is still in testing, but it should -- at least that's the goal :)

roblatham00 · 2024-03-01T21:38:00Z

@wkliao ROMIO has been able to handle large sizes for a few years now. Large counts are proving a bit harder. Requires lots of changes all over ROMIO, but I think I'm close. I'll push my changes to this branch.

Do you have a test case we should try out?

wkliao · 2024-03-01T22:05:52Z

Most of my tests are through PnetCDF. Because of its use of type conversion and nonblocking APIs, PnetCDF calls MPI-IO APIs with MPI_BYTE datatype for buffers.

Both MPI-IO and PnetCDF need to be modernized to add large count APIs.

roblatham00 · 2024-03-01T22:17:50Z

I just pushed a bunch of changes, but for review only. i'm trying to figure out how ADIOI_Heap_merge gets a bogus memory address with small requests, and large counts timeout after 30 minutes without actually doing any I/O

src/mpi/romio/adio/ad_daos/ad_daos_io_str.c

src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c

wkliao · 2024-03-02T23:35:19Z

Below is the test program with the assertion message when running 2 processes.

Assertion failed in file ../../../../mpich/src/mpi/romio/adio/common/ad_write_coll.c at line 898: (curr_to_proc[p] + len) == (unsigned) ((ADIO_Offset) curr_to_proc[p] + len)

% cat large_dtype.c
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <mpi.h>

#define ERROR(fname) \
    if (err != MPI_SUCCESS) { \
        int errorStringLen; \
        char errorString[MPI_MAX_ERROR_STRING]; \
        MPI_Error_string(err, errorString, &errorStringLen); \
        printf("Error at line %d when calling %s: %s\n",__LINE__,fname,errorString); \
    }

#define LEN 2048
#define NVARS 1100

/*----< main() >------------------------------------------------------------*/
int main(int argc, char **argv)
{
    char *filename;
    size_t i, buf_len;
    int err, rank, verbose=1, nprocs, psize[2], gsize[2], count[2], start[2];
    char *buf;
    MPI_File     fh;
    MPI_Datatype subType, filetype, buftype;
    MPI_Status   status;
    MPI_Offset fsize;
    int array_of_blocklengths[NVARS];
    MPI_Aint array_of_displacements[NVARS];
    MPI_Datatype array_of_types[NVARS];

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &nprocs);

    if (argc != 2) {
        if (!rank) printf("Usage: %s filename\n",argv[0]);
        MPI_Finalize();
        exit(1);
    }
    filename = argv[1];

    /* Creates a division of processors in a cartesian grid */
    psize[0] = psize[1] = 0;
    err = MPI_Dims_create(nprocs, 2, psize);
    ERROR("MPI_Dims_create");

    /* each 2D variable is of size gsizes[0] x gsizes[1] bytes */
    gsize[0] = LEN * psize[0];
    gsize[1] = LEN * psize[1];

    /* set subarray offset and length */
    start[0] = LEN * (rank / psize[1]);
    start[1] = LEN * (rank % psize[1]);
    count[0] = LEN - 1;   /* -1 to create holes */
    count[1] = LEN - 1;

    fsize = (MPI_Offset)gsize[0] * gsize[1] * NVARS - (LEN+1);
    if (verbose) {
        buf_len = (size_t)NVARS * (LEN-1) * (LEN-1);
        if (rank == 0) {
            printf("nprocs=%d NVARS=%d LEN=%d\n", nprocs, NVARS, LEN);
            printf("Expecting file size=%lld bytes (%.1f MB, %.1f GB)\n",
                   fsize, (float)fsize/1048576,(float)fsize/1073741824);
            printf("Each global variable is of size %d bytes (%.1f MB)\n",
                   gsize[0]*gsize[1],(float)gsize[0]*gsize[1]/1048576);
            printf("Each process writes %zd bytes (%.1f MB, %.1f GB)\n",
                   buf_len,(float)buf_len/1048576,(float)buf_len/1073741824);
        }
        printf("rank %3d: gsize=%4d %4d start=%4d %4d count=%4d %4d\n", rank,
               gsize[0],gsize[1],start[0],start[1],count[0],count[1]);
    }


    /* create 2D subarray datatype for fileview */
    err = MPI_Type_create_subarray(2, gsize, count, start, MPI_ORDER_C, MPI_BYTE, &subType);
    ERROR("MPI_Type_create_subarray");
    err = MPI_Type_commit(&subType);
    ERROR("MPI_Type_commit");

    /* create a filetype by concatenating NVARS subType */
    for (i=0; i<NVARS; i++) {
        array_of_blocklengths[i] = 1;
        array_of_displacements[i] = gsize[0]*gsize[1]*i;
        array_of_types[i] = subType;
    }
    err = MPI_Type_create_struct(NVARS, array_of_blocklengths,
                                        array_of_displacements,
                                        array_of_types,
                                        &filetype);
    ERROR("MPI_Type_create_struct");
    err = MPI_Type_commit(&filetype);
    ERROR("MPI_Type_commit");
    err = MPI_Type_free(&subType);
    ERROR("MPI_Type_free");

    /* Create local buffer datatype: each 2D variable is of size LEN x LEN */
    gsize[0] = LEN;
    gsize[1] = LEN;
    start[0] = 0;
    start[1] = 0;
    count[0] = LEN-1;  /* -1 to create holes */
    count[1] = LEN-1;

    err = MPI_Type_create_subarray(2, gsize, count, start, MPI_ORDER_C, MPI_BYTE, &subType);
    ERROR("MPI_Type_create_subarray");
    err = MPI_Type_commit(&subType);
    ERROR("MPI_Type_commit");

    /* concatenate NVARS subType into a buftype */
    for (i=0; i<NVARS; i++) {
        array_of_blocklengths[i] = 1;
        array_of_displacements[i] = LEN*LEN*i;
        array_of_types[i] = subType;
    }

    /* create a buftype by concatenating NVARS subTypes */
    err = MPI_Type_create_struct(NVARS, array_of_blocklengths,
                                        array_of_displacements,
                                        array_of_types,
                                        &buftype);
    ERROR("MPI_Type_create_struct");
    err = MPI_Type_commit(&buftype);
    ERROR("MPI_Type_commit");
    err = MPI_Type_free(&subType);
    ERROR("MPI_Type_free");

    /* allocate a local buffer */
    buf_len = (size_t)NVARS * LEN * LEN;
    buf = (char*) malloc(buf_len);
    for (i=0; i<buf_len; i++) buf[i] = (char)rank;

    /* open to create a file */
    err = MPI_File_open(MPI_COMM_WORLD, filename, MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &fh);
    ERROR("MPI_File_open");

    /* set the file view */
    err = MPI_File_set_view(fh, 0, MPI_BYTE, filetype, "native", MPI_INFO_NULL);
    ERROR("MPI_File_set_view");

    /* MPI collective write */
    err = MPI_File_write_all(fh, buf, 1, buftype, &status);
    ERROR("MPI_File_write_all");

    MPI_File_close(&fh);
    free(buf);

    err = MPI_Type_free(&filetype);
    ERROR("MPI_Type_free");
    err = MPI_Type_free(&buftype);
    ERROR("MPI_Type_free");

    MPI_Finalize();
    return 0;
}

roblatham00 · 2024-03-08T16:48:05Z

Wei-keng I always love to see new testscases: can I add this to test/mpi/io ?

wkliao · 2024-03-08T17:24:10Z

Wei-keng I always love to see new testscases: can I add this to test/mpi/io ?

Sure.

wkliao · 2024-03-08T20:48:48Z

FYI. To test reads, I added a call to MPI_File_read_all right after MPI_File_write_all in the test program (and file open mode changed to MPI_MODE_RDWR)

    /* MPI collective read */
    err = MPI_File_read_all(fh, buf, 1, buftype, &status);
    ERROR("MPI_File_read_all");

Then, I got this error when running just one MPI process.

Assertion failed in file ../../../../mpich/src/mpi/romio/adio/common/ad_read_str.c
at line 379: ((ADIO_Offset) num + size) == (unsigned) (num + size)

raffenet · 2024-03-15T20:12:12Z

Forgive me for parachuting in, but I'd like to get this PR merged and backported to 4.2.x for a release by the end of the month. I added a commit to hopefully address the issue currently blocking @wkliao. Could you take a look?

raffenet · 2024-03-15T20:12:20Z

test:mpich/ch4/most

wkliao · 2024-03-16T16:17:56Z

Hi, @raffenet

I tested your patch on a ufs using the above test program.
It passed when running one MPI process, but failed with two.
The same error message appeared

Assertion failed in file ../../../../mpich/src/mpi/romio/adio/common/ad_write_coll.c
at line 898: (curr_to_proc[p] + len) == (unsigned) ((ADIO_Offset) curr_to_proc[p] + len)

I can see your patch fixed file ad_read_str.c and ad_write_str.c, but the error
came from ad_write_coll.c.

In addition, there are other ADIO drivers that also implement ReadStrided/WriteStrided.
The same for the collective subroutines, WriteStridedColl/ReadStridedColl
They also need to be fixed.

wkliao · 2024-03-17T22:14:22Z

Please also update file /mpich/test/mpi/io/Makefile.am to add the new test program large_count.c, so it can run during CI.

raffenet · 2024-03-18T02:05:30Z

Thanks for testing. I also enabled -Wshorten-64-to-32 on my laptop and found a bunch more issues that need fixing. Will keep plugging at it when I have time.

roblatham00 · 2024-03-18T18:48:49Z

oh man last I checked (a few years ago) there were 15,000 -Wshorten-64-to-32 warnings

…

On Sun, Mar 17, 2024 at 9:05 PM Ken Raffenetti ***@***.***> wrote: Thanks for testing. I also enabled -Wshorten-64-to-32 on my laptop and found a bunch more issues that need fixing. Will keep plugging at it when I have time. — Reply to this email directly, view it on GitHub <#6928 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMIENFIRAYKQQRVL3ZDJTYYZDYBAVCNFSM6AAAAABD5ANR4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSG42DONZZGY> . You are receiving this because your review was requested.Message ID: ***@***.***>

roblatham00 · 2024-03-19T20:56:52Z

I can make ROMIO happy about this large count but not MPICH itself:

#0  0x00002b871f16a387 in raise () from /lib64/libc.so.6
#1  0x00002b871f16ba78 in abort () from /lib64/libc.so.6
#2  0x00002b870936444b in MPID_Abort.cold () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#3  0x00002b870a40f598 in MPIR_Handle_fatal_error () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#4  0x00002b870a40ffe7 in MPIR_Err_return_comm () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#5  0x00002b8709a648ea in internal_Waitall () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#6  0x00002b871959ada6 in ADIOI_W_Exchange_data () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#7  0x00002b8719598076 in ADIOI_Exch_and_write () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#8  0x00002b871959530c in ADIOI_GEN_WriteStridedColl () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#9  0x00002b87194d18e4 in MPIOI_File_write_all () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#10 0x00002b87194d12a7 in PMPI_File_write_all () from /home/robl/soft/mpich-master/lib/libmpi.so.0
#11 0x00000000004014d8 in main ()

(guess I need to rebuild with debug info)

roblatham00 · 2024-03-19T21:01:32Z

(gdb) where
#0  0x00002b871f16a387 in raise () from /lib64/libc.so.6
#1  0x00002b871f16ba78 in abort () from /lib64/libc.so.6
#2  0x00002b870936444b in MPID_Abort (comm=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, comm@entry=0x0, mpi_errno=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, mpi_errno@entry=0,
    exit_code=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, exit_code@entry=1010488083, error_msg=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
    error_msg@entry=0x7fff21c09330 "Fatal error in internal_Waitall: Invalid MPI_Request, error stack:\ninternal_Waitall(125): MPI_Waitall(count=4, array_of_requests=0x2051230, array_of_statuses=0x1) failed\ninternal_Waitall(49).: The sup"...)
    at ../src/mpid/ch4/src/ch4_globals.c:120
#3  0x00002b870a40f598 in MPIR_Handle_fatal_error (comm_ptr=comm_ptr@entry=0x0, fcname=fcname@entry=0x2b8719602480 <__func__.0> "internal_Waitall", errcode=1010488083) at ../src/mpi/errhan/errutil.c:747
#4  0x00002b870a40ffe7 in MPIR_Err_return_comm (comm_ptr=comm_ptr@entry=0x0, fcname=fcname@entry=0x2b8719602480 <__func__.0> "internal_Waitall", errcode=<optimized out>) at ../src/mpi/errhan/errutil.c:278
#5  0x00002b8709a648ea in internal_Waitall (count=<optimized out>, array_of_requests=<optimized out>, array_of_statuses=<optimized out>) at ../src/binding/c/request/waitall.c:129
#6  0x00002b871959ada6 in ADIOI_W_Exchange_data (fd=0x2042890, buf=0x2b872588d010, write_buf=0x2b883888e0b0 "", flat_buf=0x2050e40, offset_list=0x2b883988f0b0, len_list=0x2b883a9bce58, send_size=0x20517a8, recv_size=0x20517b0, off=4294967296, size=16777216, count=0x2051798,
    start_pos=0x20517d8, partial_recv=0x20517a0, sent_to_proc=0x20517b8, nprocs=2, myrank=0, buftype_is_contig=0, contig_access_count=2251700, min_st_offset=0, fd_size=4613733376, fd_start=0x2050ae0, fd_end=0x2050af0, others_req=0x20516a0, send_buf_idx=0x20517c0,
    curr_to_proc=0x20517c8, done_to_proc=0x20517d0, hole=0x7fff21c0aaf4, iter=256, buftype_extent=4613734400, buf_idx=0x2042ee0, error_code=0x7fff21c0adec) at ../../../../src/mpi/romio/adio/common/ad_write_coll.c:750
#7  0x00002b8719598076 in ADIOI_Exch_and_write (fd=0x2042890, buf=0x2b872588d010, datatype=-872415232, nprocs=2, myrank=0, others_req=0x20516a0, offset_list=0x2b883988f0b0, len_list=0x2b883a9bce58, contig_access_count=2251700, min_st_offset=0, fd_size=4613733376,
    fd_start=0x2050ae0, fd_end=0x2050af0, buf_idx=0x2042ee0, error_code=0x7fff21c0adec) at ../../../../src/mpi/romio/adio/common/ad_write_coll.c:461
#8  0x00002b871959530c in ADIOI_GEN_WriteStridedColl (fd=0x2042890, buf=0x2b872588d010, count=1, datatype=-872415232, file_ptr_type=101, offset=0, status=0x7fff21c0af50, error_code=0x7fff21c0adec) at ../../../../src/mpi/romio/adio/common/ad_write_coll.c:186
#9  0x00002b87194d18e4 in MPIOI_File_write_all (fh=0x2042890, offset=0, file_ptr_type=101, buf=0x2b872588d010, count=1, datatype=-872415232, myname=0x2b871e16a450 <myname> "MPI_FILE_WRITE_ALL", status=0x7fff21c0af50) at ../../../../src/mpi/romio/mpi-io/write_all.c:166
#10 0x00002b87194d12a7 in PMPI_File_write_all (fh=0x2042890, buf=0x2b872588d010, count=1, datatype=-872415232, status=0x7fff21c0af50) at ../../../../src/mpi/romio/mpi-io/write_all.c:69
#11 0x00000000004014d8 in main (argc=<optimized out>, argv=<optimized out>) at /home/robl/src/mpich/test/mpi/io/large_count.c:143```

roblatham00 · 2024-03-19T21:07:26Z

still in progress but pushed some more changes to the branch

roblatham00 · 2024-03-20T13:55:29Z

with a debug, no optimizaiton build:

Abort(1007866643) on node 0: Fatal error in internal_Waitall: Invalid MPI_Request, error stack:
internal_Waitall(125): MPI_Waitall(count=4, array_of_requests=0x27673d0, array_of_statuses=0x1) failed
internal_Waitall(49).: The supplied request in array element 2 was invalid (kind=0)

roblatham00 · 2024-03-20T14:56:10Z

hm. -fsanitize=undefined -fsanitize=address doesn't uncover anything

roblatham00 · 2024-03-22T16:30:46Z

pushed another round of commits that seems to make the big-count IOR workload happy. Still investigating Wei-keng's test case

roblatham00 · 2024-03-22T16:57:45Z

going back to "invalid request"... I've added return-type checking to every isend and irecv in that path and still get The supplied request in array element 2 was invalid (kind=15) .

other than checking return value of MPI_Isend and MPI_Irecv, how can I confirm the returned request is valid? I'd like to assert/error/dump-core then, not at waitall time.

I could change to MPI_Send and MPI_recv but I am pretty sure that will deadlock.

test/mpi/io/large_count.c

raffenet · 2024-05-13T15:35:04Z

test:mpich/warnings

hzhou · 2024-05-13T21:16:46Z

test:mpich/ch3/most
test:mpich/ch4/most

hzhou · 2024-05-13T21:17:34Z

test/mpi/io/Makefile.am

@@ -30,6 +30,8 @@ noinst_PROGRAMS = \
    external32_derived_dtype \
    tst_fileview \
    zero_count \
+    large_count \
+    large_dtype \


How big (in terms of file space and memory it needs) are these two tests?

I don't know the memory footprint, but the output file size is 18 GB when running on 2 MPI processes.

Just realized that both tests are not added to the testlist so they won't be run during CI testing. I guess that is fine for now.

/usr/bin/time says large_count needs ~26 MiB (26276 KiB) of memory (max RSS) -- that seems low.

Makes sense. That is how the chunking algorithm is supposed to work, right?

Wait, it allocates a local buffer of size ~4GB, so ~26MB does seem low

hzhou · 2024-05-14T13:28:53Z

The rd_end failure is a regression from this PR:

not ok 2487 - ./io/rd_end 2
  ---
  Directory: ./io
  File: rd_end
  Num-procs: 2
  Timeout: 180
  Date: "Mon May 13 17:33:52 2024"
  ...
## Test output (expected 'No Errors'):
## 0: count was 0; expected 5
##  Found 1 errors

hzhou · 2024-05-14T16:11:05Z

test/mpi/io/large_count.c

+    ERROR("MPI_Type_free");
+
+    /* allocate a local buffer */
+    buf_len = (size_t) NVARS *LEN * LEN;


So, 1100 * 2048 * 2048 -> ~ 4GB?

wkliao · 2024-05-14T16:16:52Z

User buffer malloc total size per process is 8 GB in large_dtype.c.
Googling keyword memory footprint suggested the following commands.

/usr/bin/time -v command

and

/usr/bin/time -l command

The former is for bash and latter mac.

The value shown in "maximum resident set size" is the max footprint.

wkliao · 2024-05-14T17:01:08Z

/usr/bin/time -v mpiexec -n 2 large_dtype -f dummy

	Elapsed (wall clock) time (h:mm:ss or m:ss): 35:33.03
	Maximum resident set size (kbytes): 9285480

roblatham00 · 2024-05-14T19:00:19Z

test:mpich/ch3/most
test:mpich/ch4/most

It is potentially possible to have more than INT_MAX number of noncontig segments, thus contig_access_count need be typed as MPI_Count in order to prevent overflow.

While the collective buffering routine will chunk up large requests, there is still a preliminary "what is everyone doing" step. Those exchanges might require large counts.

Derived from a pnetcdf-generated workload

These routines stored length values in int and unsigned variable types in some places. Update them to use MPI_Aint, which matches the eventual API call.

Huge patch set touching almost all of romio, but should be much fewer places where we store potentially large values in an int. Passes '-fsanitize=undefined' and also reduces '-Wshorten-64-to-32' warnings.

roblatham00 · 2024-05-14T20:20:18Z

test:mpich/ch3/most
test:mpich/ch4/most

roblatham00 · 2024-05-15T14:14:03Z

the failing tests are mostly OS/X failing to find the right autoconf version. However, I don't know how to resolve https://jenkins-pmrs.cels.anl.gov/job/mpich-review-ch4-ucx/3351/jenkins_configure=debug,label=ubuntu22.04_review/testReport/junit/(root)/io/02482_____io_i_noncontig_coll2_4__/ -- that's a segfault in i_noncontig2 but I cannot reproduce that on Imrpov's UCX stack (clang-16 with sanitizers) and I cannot rule out a bug in UCX

hzhou · 2024-05-15T14:33:05Z

the failing tests are mostly OS/X failing to find the right autoconf version. However, I don't know how to resolve https://jenkins-pmrs.cels.anl.gov/job/mpich-review-ch4-ucx/3351/jenkins_configure=debug,label=ubuntu22.04_review/testReport/junit/(root)/io/02482_____io_i_noncontig_coll2_4__/ -- that's a segfault in i_noncontig2 but I cannot reproduce that on Imrpov's UCX stack (clang-16 with sanitizers) and I cannot rule out a bug in UCX

Let's ignore the osx tests for now. Most of the ch4 failures are expected. OFI failures are due to known performance issues in the sockets provider. The io/i_noncontig2 failures seems to be related to ucx shared memory issue. When the server is overloaded, ucx and some ofi providers are known not to release sharedmemory promptly and resulting in out-of-space errors.

I think this PR is good to merge now.

hzhou requested a review from roblatham00 February 27, 2024 23:48

roblatham00 force-pushed the 2402_romio_count branch from 9c0ae76 to b24c4ae Compare March 1, 2024 22:16

hzhou commented Mar 1, 2024

View reviewed changes

src/mpi/romio/adio/ad_daos/ad_daos_io_str.c Outdated Show resolved Hide resolved

src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c Outdated Show resolved Hide resolved

src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c Show resolved Hide resolved

roblatham00 force-pushed the 2402_romio_count branch from b24c4ae to 3b653c5 Compare March 8, 2024 17:26

roblatham00 force-pushed the 2402_romio_count branch from 0f0d6f2 to 58fae4c Compare March 19, 2024 21:07

roblatham00 force-pushed the 2402_romio_count branch from 58fae4c to ebe1ee3 Compare March 22, 2024 16:29

wkliao reviewed Mar 22, 2024

View reviewed changes

test/mpi/io/large_count.c Show resolved Hide resolved

roblatham00 force-pushed the 2402_romio_count branch 2 times, most recently from da803b6 to 0b59b09 Compare May 13, 2024 20:51

hzhou commented May 13, 2024

View reviewed changes

hzhou commented May 14, 2024

View reviewed changes

roblatham00 force-pushed the 2402_romio_count branch from 0b59b09 to c0d63f4 Compare May 14, 2024 18:58

roblatham00 and others added 12 commits May 14, 2024 15:08

ROMIO: large count fixups

9e7f1cb

romio: make contig_access_count MPI_Count instead of int

b3e3228

It is potentially possible to have more than INT_MAX number of noncontig segments, thus contig_access_count need be typed as MPI_Count in order to prevent overflow.

ROMIO: fix large count overflows

7e3f8d4

ROMIO: use large counts routines

995bd31

While the collective buffering routine will chunk up large requests, there is still a preliminary "what is everyone doing" step. Those exchanges might require large counts.

test case for large count / large size I/O

5fd5863

Derived from a pnetcdf-generated workload

romio: Fix truncated length values in ad_read|write_str.c

7d207fd

These routines stored length values in int and unsigned variable types in some places. Update them to use MPI_Aint, which matches the eventual API call.

romio: "large count" improvements

38471c0

Huge patch set touching almost all of romio, but should be much fewer places where we store potentially large values in an int. Passes '-fsanitize=undefined' and also reduces '-Wshorten-64-to-32' warnings.

romio: move heap merge to common utliites

b28bb35

romio: make heap_merge and callers 64-bit clean

709543c

romio: warning fixes and cleanups

b325192

remove more asserts

031e15a

romio: daos: 64-to-32 warning fixes

dec5640

roblatham00 force-pushed the 2402_romio_count branch from c0d63f4 to dec5640 Compare May 14, 2024 20:08

hzhou merged commit 5861d90 into pmodels:main May 15, 2024
4 of 8 checks passed

hzhou deleted the 2402_romio_count branch May 15, 2024 14:34

raffenet mentioned this pull request May 16, 2024

[4.2.x] backport of ROMIO large count fixes #7006

Merged

4 tasks

romio: make contig_access_count MPI_Count instead of int #6928

romio: make contig_access_count MPI_Count instead of int #6928

Conversation

hzhou commented Feb 27, 2024 • edited by raffenet Loading

Pull Request Description

Author Checklist

hzhou commented Feb 27, 2024

wkliao commented Mar 1, 2024

hzhou commented Mar 1, 2024

roblatham00 commented Mar 1, 2024

wkliao commented Mar 1, 2024

roblatham00 commented Mar 1, 2024

wkliao commented Mar 2, 2024

roblatham00 commented Mar 8, 2024

wkliao commented Mar 8, 2024

wkliao commented Mar 8, 2024 • edited Loading

raffenet commented Mar 15, 2024

raffenet commented Mar 15, 2024

wkliao commented Mar 16, 2024

wkliao commented Mar 17, 2024

raffenet commented Mar 18, 2024

roblatham00 commented Mar 18, 2024 via email

roblatham00 commented Mar 19, 2024

roblatham00 commented Mar 19, 2024

roblatham00 commented Mar 19, 2024

roblatham00 commented Mar 20, 2024

roblatham00 commented Mar 20, 2024

roblatham00 commented Mar 22, 2024

roblatham00 commented Mar 22, 2024

raffenet commented May 13, 2024

hzhou commented May 13, 2024

hzhou May 13, 2024

Choose a reason for hiding this comment

wkliao May 13, 2024

Choose a reason for hiding this comment

hzhou May 14, 2024

Choose a reason for hiding this comment

roblatham00 May 14, 2024

Choose a reason for hiding this comment

hzhou May 14, 2024

Choose a reason for hiding this comment

hzhou May 14, 2024

Choose a reason for hiding this comment

hzhou commented May 14, 2024

hzhou May 14, 2024

Choose a reason for hiding this comment

wkliao commented May 14, 2024

wkliao commented May 14, 2024

roblatham00 commented May 14, 2024

roblatham00 commented May 14, 2024

roblatham00 commented May 15, 2024 • edited Loading

hzhou commented May 15, 2024

hzhou commented Feb 27, 2024 •

edited by raffenet

Loading

wkliao commented Mar 8, 2024 •

edited

Loading

roblatham00 commented May 15, 2024 •

edited

Loading