Passing --help to a deal.II application prints the kokkos help

[peterrum]

As pointed out by @nfehn , a simple program like:

#include <deal.II/base/mpi.h>

using namespace dealii;

int main(int argc, char *argv[])
{
  Utilities::MPI::MPI_InitFinalize mpi_initialization(argc, argv, 1);
}

give the following output if run with ./main --help:

--------------------------------------------------------------------------------
-------------Kokkos command line arguments--------------------------------------
--------------------------------------------------------------------------------
This program is using Kokkos.  You can use the following command line flags to
control its behavior:

Kokkos Core Options:
  --kokkos-help                  : print this message
  --kokkos-disable-warnings      : disable kokkos warning messages
  --kokkos-print-configuration   : print configuration
  --kokkos-tune-internals        : allow Kokkos to autotune policies and declare
                                   tuning features through the tuning system. If
                                   left off, Kokkos uses heuristics
  --kokkos-num-threads=INT       : specify total number of threads to use for
                                   parallel regions on the host.
  --kokkos-device-id=INT         : specify device id to be used by Kokkos.
  --kokkos-map-device-id-by=(random|mpi_rank)
                                 : strategy to select device-id automatically from
                                   available devices.
                                   - random:   choose a random device from available.
                                   - mpi_rank: choose device-id based on a round robin
                                               assignment of local MPI ranks.
                                               Works with OpenMPI, MVAPICH, SLURM, and
                                               derived implementations.

Kokkos Tools Options:
  --kokkos-tools-libs=STR        : Specify which of the tools to use. Must either
                                   be full path to library or name of library if the
                                   path is present in the runtime library search path
                                   (e.g. LD_LIBRARY_PATH)
  --kokkos-tools-help            : Query the (loaded) kokkos-tool for its command-line
                                   option support (which should then be passed via
                                   --kokkos-tools-args="...")
  --kokkos-tools-args=STR        : A single (quoted) string of options which will be
                                   whitespace delimited and passed to the loaded
                                   kokkos-tool as command-line arguments. E.g.
                                   `<EXE> --kokkos-tools-args="-c input.txt"` will
                                   pass `<EXE> -c input.txt` as argc/argv to tool

Except for --kokkos[-tools]-help, you can alternatively set the corresponding
environment variable of a flag (all letters in upper-case and underscores
instead of hyphens). For example, to disable warning messages, you can either
specify --kokkos-disable-warnings or set the KOKKOS_DISABLE_WARNINGS
environment variable to yes.

Join us on Slack, visit https://kokkosteam.slack.com
Report bugs to https://github.com/kokkos/kokkos/issues
--------------------------------------------------------------------------------

This is an unwanted behavior in user code if the output should be the help of the application (as in the case of ExaDG).

[bangerth]

That's a nuisance, but not something I would hold the release up for.

[tjhei]

It has been the same with PETSc since forever. I would argue that user applications need to pre-process the command args before calling MPI_InitFinalize and remove arguments understood by the application.
We do this in ASPECT for example:
https://github.com/geodynamics/aspect/blob/df9a873630d62bbb85a61b30da9ab604a97d1de6/source/main.cc#L687-L699

It is of course annoying that this is a breaking change.

[kronbichler]

It is annoying in my opinion, but I can understand it from a Kokkos perspective. In any case, the question is if deal.II wants to pass --help on to Kokkos and pretend that Kokkos were the main ingredient here. I feel it is just one of many wheels on the vehicle, so it should behave as such and remain quiet.

@tjhei Does PETSc hijack the --help command? I used PETSc before and did not observe it for that command, albeit for others. I know it does many other options, including when one crashes.

[tjhei]

@tjhei Does PETSc hijack the --help command?

I think PETSc only detects -help.

[bangerth]

As I mentioned on slack, I believe this is a bug in ExaDG: If you don't want the --help flag passed on, you need to filter it out. But I don't think that should be the job of deal.II.

[nfehn]

I think @kronbichler has a point here. From a software design perspective, the problem that I see here is that as a user, I do not see that dealii::Utilities::MPI::MPI_InitFinalize involves kokkos. It is - in my understanding - deal.II utility that one is using. From the deal.II documentation https://www.dealii.org/developer/doxygen/deal.II/classUtilities_1_1MPI_1_1MPI__InitFinalize.html, I do not get an idea either.

So if this is considered a "bug" in application code, which may not pass this piece of information to dealii, one may argue with the same line of reasoning that deal.II may not pass it on further to other TPLs ... TPLs which are not represented syntactically in the code that I write, or in the documentation provided by deal.II. Let my point out that the interface of dealii::Utilities::MPI::MPI_InitFinalize::MPI_InitFinalize(int argc, char ** argv) is kind of the same as that of the function main(int argc, char ** argv) and that this is actually the problem. From that perspective, it is - in my opinion - pretty clear that deal.II should not simply shift responsibility to user code. The problem seems to be the function interface, and this affects deal.II just as it affects user code with its main function.

I do not have irons in the fire here. I want to provide feedback, though, that deal.II seems to announce officially (via papers) that breaking changes are not introduced (and if so, they are minimal), but that users observe such incompatible changes actually year by year (which is acceptable if things change for the good of course). I consider Utilities::MPI something fundamental. My expectation/wish is that deal.II documents which types of arguments are allowed to be passed to dealii::Utilities::MPI::MPI_InitFinalize (maybe with a reference to documentation of third party libraries), and explicitly catches arguments such as --help. Otherwise, the behavior of dealii::Utilities::MPI::MPI_InitFinalize actually seems to be undefined implementation-wise and documentation-wise.

[bangerth]

That's a reasonable argument. Perhaps deal.II should simply not pass --help on to Kokkos or PETSc or MPI, but not actually remove these flags from argc and argv so that whatever is done after MPI_InitFinalize can deal with it?