Don't exceed maximum MPI tag value

[jhkennedy]

The MPI standard specifies that the MPI tag upper bound (MPI_TAG_UB) must
be at least 32767 and many implementations greatly exceed this number
(eg. cray-mpich/7.4.4 has MPI_TAG_UB == 2^21-1). However, in
multi-instance runs, this tag is quickly exceed (~30 instances).

This PR adds a check for tags larger than MPI_TAG_UB and reduces
msgtag so that it will take ~300 instances to exceed MPI_TAG_UB on cray
machines while still protecting against tag collisions out to ~100
instances (see #2397 and #2363).

This should be sufficient for any multi-instance runs not using
multi-driver (where this isn't a problem), and runs larger should be
using multi-driver.

Test suite: scripts_regression_tests.py on edison; MVK SystemTest on Titan (E3SM)
Test baseline: N/A
Test namelist changes: none
Test status: [bit for bit]

Fixes E3SM-Project/E3SM#2487

User interface changes?: N

Update gh-pages html (Y/N)?: N

Code review:

[amametjanov]

+1 on the upper bound check. My vote is to keep the tags unchanged in single-instance runs and use old (or this PR's values) tag values in multi-instance runs. Tracking down the hang in high-res runs took a while and reducing tags by a factor of 10 may bring that issue back.

[jedwards4b]

Are you sure that this is the correct type for this variable - from the mpich documentation:
The attribute_val in Fortran is a pointer to a Fortran integer, not a pointer to a void *.
Which I think implies that what is in max_mpi_tag is the address of the value and not the value itself?

[jhkennedy]

No I'm not a 100% sure. From my understanding though, this is returning the actual value not an address of the value.

This mpi-forum page explains it and provides some examples, but here is the relevant text:

MPI supports two types of attributes: address-valued (pointer) attributes, and integer valued attributes. C and C++ attribute functions put and get address valued attributes. Fortran attribute functions put and get integer valued attributes. When an integer valued attribute is accessed from C or C++, then MPI_xxx_get_attr will return the address of (a pointer to) the integer valued attribute, which is a pointer to MPI_Aint if the attribute was stored with Fortran MPI_xxx_SET_ATTR, and a pointer to int if it was stored with the deprecated Fortran MPI_ATTR_PUT. When an address valued attribute is accessed from Fortran, then MPI_xxx_GET_ATTR will convert the address into an integer and return the result of this conversion. This conversion is lossless if new style attribute functions are used, and an integer of kind MPI_ADDRESS_KIND is returned. The conversion may cause truncation if deprecated attribute functions are used. In C, the deprecated routines MPI_Attr_put and MPI_Attr_get behave identical to MPI_Comm_set_attr and MPI_Comm_get_attr.

And that same behavior is implied here as well:

https://www.open-mpi.org/doc/v3.0/man3/MPI_Comm_set_attr.3.php

[jedwards4b]

thanks

[jhkennedy]

@amametjanov I think you're right, it's probably best to keep the message tags unchanged (Note: this will limit multi-instance to ~20 instances without using multi-driver). I'm in the process of converting some of my multi-instance runs to use multi-driver. Once those pass I'll bump it back up and update the multi-instance docs to discuss this limitation.

[jedwards4b]

@jhkennedy is this ready to merge?

[jhkennedy]

@jedwards4b I think we need to rethink these tags first.

To recap, #2397 and #2363 caused the MPI tag values in the mct driver to change from:

 mpi_tag = comp(eci)%cplcompid*100+eci*10+N

to

 mpi_tag = comp(eci)%cplcompid*10000+eci*10+N

where N is in the range of [1,9], because there were some message tag collisions causing hangups on single instance high rez runs.

This change causes multi-instance runs to exceed the maximum MPI tag value at ~20 instances on titan (see E3SM-Project/E3SM#2487) and at ~100 instances on edison (see ESCOMP/CTSM#515 ; pinging @ckoven).

Multi-driver, should alleviate this issue, but I've run into #2295 when trying to change my multi-instance runs over to multi-driver (seems to be titan specific? trying at NERSC), and multi-driver isn't useful for ESCOMP/CTSM#515 because of how long it takes to build the DATM namelists in their case.

So the proposed solution was to leave a (MPI implementation dependent) limit on multi-instance runs and encourage multi-driver use, while also picking a middle ground in the MPI tag size (mpi_tag = comp(eci)%cplcompid*1000+eci*10+5; 1000 instead of 100 (old) or 10000 (new)).

But, due to the aforementioned multi-driver issues and @amametjanov feeling that 1000 still leaves too much potential for tag collision, I don't think this solution is the right one.

So, that leaves us with is there a different and better way to assign the tags?

For example, it seems the comp(eci)%cplcompid part of the tag varies the greatest so could we reorder the tag to something like:

 mpi_tag = eci*10000+comp(eci)%cplcompid*10+5

?

However, I'm not sure I understand how the collisions are occurring in single instance runs -- @amametjanov what are these tags colliding with?

[amametjanov]

@jhkennedy how about this patch to use old tags for multi-instance runs: https://gist.github.com/amametjanov/790661841af3a49977b1b584fd509668

[ckoven]

Hi All,
Just wanted to add that the --multi-dirver method doesn't seem to be working for my use-case described in ESCOMP/CTSM#515, I get DATM errors as described in ESCOMP/CTSM#515 (comment). So I guess I would just like to advocate maintaining/strengthening the ability to do single-instance DATM, multi-instance land simulations with large numbers of land instances if at all possible. Thanks!

[rljacob]

What is the status of this PR?

[jhkennedy]

@rljacob I'm testing @amametjanov patch -- so far looks good; I'll know more tomorrow.

[jhkennedy]

@jedwards4b this is ready to merge.

@ckoven this should allow you to run your multi-instance runs.

[jedwards4b]

It's not clear to me why we need a seperate case for ninst > 1.

[jedwards4b]

Why two cases here? Why not just use line 393 regardless of size(comp)?

[jhkennedy]

To summarize the thread, that was the suggestion by @amametjanov to multi-instance runs with >20 instance to run without exceeding the MPI tag limit, but still prevent tag collision on single instance high res runs. Multi-driver is a possible alternative, but there were challenges around setting that up for people in this thread and others I've talked to about this issue. It seems if multi-instance is going to be an option it should be able to handle at least a normal ensemble size.

I'd be welcome to a suggestion on how to assign these tag values entirely differently, but I don't know them, or with what they are colliding, well enough. I suggested re-ordering the way the tags were written, but that didn't get any traction from anyone in this thread. So this at least fixes the immediate issue.