Optimize Utilities::MPI::broadcast()

[bangerth]

#11940 introduces an MPI broadcast function, which packs up the argument, determines the size, sends around the size, on all processors allocates a sufficiently large buffer, and then in a second MPI_Bcast sends the actual data.

This could be optimized: If the argument is a built-in type or fixed-size array thereof, we don't need to call pack() but can just send the data and the receiving end will know what the buffer size needs to be because it can just stack-allocate the output object.

@bergbauer -- FYI.

[fandreuz]

Hi @bangerth, I would like to work on this starter project. What do you recommend to use in order to detect if the template type T is built-in? I found is_fundamental (reference) but I do not know whether it is the optimal choice.

Also, when you say that we do not need to call pack() and just send the data, is there a preferred way to obtain the proper MPI_Datatype to be used in the call to MPI_Bcast?

[masterleinad]

I think it makes sense to add the optimization for MPI data types. For everything else, you would likely need to serialize or create a custom MPI data type to get the padding right.

In

dealii/include/deal.II/base/mpi.templates.h

Lines 46 to 174 in 9d8c8e9

	inline MPI_Datatype
	mpi_type_id(const bool *)
	{
	# if DEAL_II_MPI_VERSION_GTE(2, 2)
	return MPI_CXX_BOOL;
	# else
	return MPI_C_BOOL;
	# endif
	}
	inline MPI_Datatype
	mpi_type_id(const char *)
	{
	return MPI_CHAR;
	}
	inline MPI_Datatype
	mpi_type_id(const signed char *)
	{
	return MPI_SIGNED_CHAR;
	}
	inline MPI_Datatype
	mpi_type_id(const short *)
	{
	return MPI_SHORT;
	}
	inline MPI_Datatype
	mpi_type_id(const int *)
	{
	return MPI_INT;
	}
	inline MPI_Datatype
	mpi_type_id(const long int *)
	{
	return MPI_LONG;
	}
	inline MPI_Datatype
	mpi_type_id(const unsigned char *)
	{
	return MPI_UNSIGNED_CHAR;
	}
	inline MPI_Datatype
	mpi_type_id(const unsigned short *)
	{
	return MPI_UNSIGNED_SHORT;
	}
	inline MPI_Datatype
	mpi_type_id(const unsigned int *)
	{
	return MPI_UNSIGNED;
	}
	inline MPI_Datatype
	mpi_type_id(const unsigned long int *)
	{
	return MPI_UNSIGNED_LONG;
	}
	inline MPI_Datatype
	mpi_type_id(const unsigned long long int *)
	{
	return MPI_UNSIGNED_LONG_LONG;
	}
	inline MPI_Datatype
	mpi_type_id(const float *)
	{
	return MPI_FLOAT;
	}
	inline MPI_Datatype
	mpi_type_id(const double *)
	{
	return MPI_DOUBLE;
	}
	inline MPI_Datatype
	mpi_type_id(const long double *)
	{
	return MPI_LONG_DOUBLE;
	}
	inline MPI_Datatype
	mpi_type_id(const std::complex<float> *)
	{
	return MPI_COMPLEX;
	}
	inline MPI_Datatype
	mpi_type_id(const std::complex<double> *)
	{
	return MPI_DOUBLE_COMPLEX;
	}

, you can see that we already have functionality to convert C++ data types to the corresponding MPI data types.

[fandreuz]

Thank you @masterleinad, this is exactly what I was looking for. Am I allowed to use concept from C++20?

I would like to define a concept like the following:

template<typename T>
      concept BuiltIn_MPI_DataType = requires(T a) {
        { mpi_type_id(a) };
      };

and then provide two versions of broadcast, where the one meant to be used with built-in types is defined in the following way:

template <BuiltIn_MPI_DataType T>
    T
    broadcast(const MPI_Comm    &comm,
              const T           &object_to_send,
              const unsigned int root_process)
    {

[peterrum]

Am I allowed to use concept from C++20?

Unfortunately not. We are still at C++14.

[bangerth]

Ah, that would be nice if we could already use C++20 :-) Fortunately, MPI's list of data types is fixed, and you could use something like this to find out whether a type in question is in a specific list (encoded in the form of a std::tuple<...list of MPI data types...>): https://stackoverflow.com/questions/25958259/how-do-i-find-out-if-a-tuple-contains-a-type

A small amount of std::enable_if will then allow you to write the specialization you are looking for.