Advice regarding PIO async in multi-component model

[nichannah]

Hi,

I'd like to get some advice re implementing asynchronous IO for a single model in a multi-component model. I think my main query is where I should call pio_init() wrt. the existing call to MPI_Split that creates the individual communicators for the component models. The options I can think of:

1. Call pio_init() on all PEs before my existing MPI_Split. This will leave me with a new computation communicator which will become the new COMM_WORLD and an IO communicator. The new COMM_WORLD will then be split into component models.

2. Call pio_init() after the existing MPI_Split with only the PEs that are a part of the component model which uses the async IO. For various reasons I'm not sure how this would work, e.g. any subsequent use of MPI_COMM_WORLD for collectives in the code would break because the IO PEs are not going to join in.

3. Somehow replace the existing MPI_Split with the one that (presumably) is being called within pio_init()

Any thoughts or examples of how this is done elsewhere would be much appreciated.

[edwardhartnett]

You do not do the MPI_split.

You call PIO_Init() on all processors. Those that are I/O will not return from PIO_Init() until all computational units call PIO_Finalize().

[edwardhartnett]

Also, make sure you use the current master. It has some very important async fixes. I am hoping to do a release this week with those fixes and improvements.

[jedwards4b]

Hi Nic, Any use of MPI_COMM_WORLD for collectives will not work with async IO since the IO processors will not participate in the call. All uses of MPI_COMM_WORLD in the code need to be replaced with an explicit derived communicator. There are two interfaces to PIO_init_async in the first <https://github.com/NCAR/ParallelIO/blob/master/src/clib/pioc.c#L1587> you pass in COMM_WORLD and a COMP_COMM is returned. In the second <https://github.com/NCAR/ParallelIO/blob/master/src/clib/pioc.c#L2046> you pass in existing COMMs that you have created and IO_COMM members do not return. They both do the same thing, choose the one that best fits your code. You can use either all of comm_world or just the comm of the component which will use PIO but if you do the later you need to keep in mind that communications with your component from the other components will be with the computational split and not the comm you passed to PIO_INIT and you will need to tell the other models that somehow. Jim

…

On Mon, Feb 8, 2021 at 4:38 AM Edward Hartnett ***@***.***> wrote: Also, make sure you use the current master. It has some very important async fixes. I am hoping to do a release this week with those fixes and improvements. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1853 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOXUGFQJ7YVTGK7STY2GPTS57EL7ANCNFSM4XIJPGAA> .

-- Jim Edwards CESM Software Engineer National Center for Atmospheric Research Boulder, CO

[nichannah]

Thank you very much @edwardhartnett and @jedwards4b - this is what I was after.