MPI_Send_init error with AMR (using default pgen)

[hwhitehead]

I have encountered an error when using too many refinement levels in Athena++. This problem is reproducible on my system when following the AMR tutorial using the blast.cpp pgen. While the problem will run with num_levels = 3 as suggested in the tutorial, when running with num_levels = 7, the code returns the following error

Abort(1007304196) on node 3 (rank 3 in comm 0): Fatal error in internal_Send_init: Unknown error class, error stack:
internal_Send_init(30828): MPI_Send_init(buf=0x27101c0, count=160, MPI_DOUBLE, 6, 1053889, MPI_COMM_WORLD, request=0x26491f8) failed
internal_Send_init(30802): Invalid tag, value is 1053889
Abort(134888964) on node 1 (rank 1 in comm 0): Fatal error in internal_Send_init: Unknown error class, error stack:
internal_Send_init(30828): MPI_Send_init(buf=0x1d221c0, count=160, MPI_DOUBLE, 4, 1053889, MPI_COMM_WORLD, request=0x1c5b1f8) failed
internal_Send_init(30802): Invalid tag, value is 1053889
Abort(604651012) on node 2 (rank 2 in comm 0): Fatal error in internal_Send_init: Unknown error class, error stack:
internal_Send_init(30828): MPI_Send_init(buf=0x15b31c0, count=160, MPI_DOUBLE, 7, 1053889, MPI_COMM_WORLD, request=0x14ec1f8) failed
internal_Send_init(30802): Invalid tag, value is 1053889

I have previously run Athena++ (with a custom pgen) using 7 levels of refinement, so I wonder if this is specific to the machine I am running on. This page makes reference to the maximum tag being MPI implementation specific. The code which crashed was compiled with icpc, I can provide other details if that is illuminating.

Has anyone encountered this error before? Is there a known limit to the number of AMR levels? Can this limit be readily bypassed?

[tomidakn]

It seems to be indeed the tag limit. Please check your MPI_TAG_UB.
https://www.intel.com/content/www/us/en/developer/articles/technical/large-mpi-tags-with-the-intel-mpi.html

How many processes are you using? As the tag contains the local ID on the target node, it probably will run fine with more processes.

[hwhitehead]

Thanks for the fix, the problem can be avoided on my system by running the initial .rst file generation step on more cores.

My understanding is that the .rst generation step cannot run on more cores than there are meshblocks in the root grid, so in principle this may still cause issues if the number of meshblocks in the root and refined grids are VERY different. This is not the case in my system however, marking as solved!

[tomidakn]

Ah, I see. Maybe we need to modify the code to allow some processes have zero MeshBlock.