#include <hpx/hpx_main.hpp> does not work

[eschnett]

I tried to used #include <hpx/hpx_main.hpp> to wrap my main function. My code looked approximately like this:

#include <hpx/hpx_main.hpp>
int main(int argc, char **argv)
{
  ...
  return 0;
}

This builds fine, but aborts at run time with an error stating that no hpx_main or hpx_main_user is defined:

{function}: The console locality does not implement any main entry point usable as the main HPX thread (e.g. no hpx_main, hpx::user_main, etc.)
{file}: /xfs1/eschnetter/compute/Cbeta/arrangements/ExternalLibraries/HPX/dist/hpx/src/hpx_user_main.cpp

This is incorrect:

$ nm exe/cactus_hpx-debug | grep ' T ' | grep main | c++filt
0000000000411a70 T main
0000000000413440 T hpx_main(int, char**)
0000000000412670 T hpx_main(boost::program_options::variables_map&)
00000000004126d0 T hpx_main()
00000000004134c0 T hpx::user_main(int, char**)
0000000000413750 T hpx::user_main()

I assume the problem is that there are two functions hpx_main, one defined in my executable, the other defined in an HPX shared library. At run time, the version in the shared library is used, leading to this error.

This work-around works for me:

#include <hpx/hpx_init.hpp>

int cactus_main(int argc, char **argv);

int hpx_main(int argc, char **argv)
{
  cactus_main(argc, argv);
  return hpx::finalize();
}

int main(int argc, char **argv)
{
  return hpx::init(argc, argv);
}
#define main cactus_main

[define my original main function here]

If the problem is indeed as I described above, then you could implement my work-around in HPX.

[hkaiser]

The correct prototype is:

int main(int argc, char* argv[]);

if you use that, it works.

[hkaiser]

Ok, I just checked the Standard. You're right, int main(int argc, char ** argv) is the correct signature. I'll fix it in HPX.

[K-ballo]

Those two function signatures are actually the same. There might be something else going on here.

[hkaiser]

Eric, if this does not fix the issue (as it might not), please give us the preprocessed output from your file containing main.

[eschnett]

This did not fix the issue.

Also, could you tell me how to re-open an issue?

[hkaiser]

Re-openeing as it's not fixed.

This should definitely work:

#include <hpx/hpx_main.hpp>
int main(int argc, char **argv)
{
    ...
    return 0;
}

and indeed it does work in many places. So something else is going on.

Let me explain how it's supposed to work. <hpx/hpx_main.hpp> defines main as:

#define main hpx::user_main

Further, the static library hpx_init.a implements the real int main(int argc, char** argv) which essentially calls hpx::user_main(). However, this library also implements it's own hpx::user_main() which is the one executed in your executable and which reports the error that no user defined HPX-main is available.

What could be wrong? Some ideas:

• the linking order makes the linker select hpx::user_main in hpx_init.a instead of in your executable, or
• the linker is not extracting single translation units from hpx_init.a (each of those functions described above are compiled separately) but extracts all modules at once

What else could be wrong?

[eschnett]

This is my link command:

ccache icpc -o "/xfs1/eschnetter/compute/Cbeta/exe/cactus_hpx-debug" -DOMPI_SKIP_MPICXX -D_FORTIFY_SOURCE=2 -DCARPET_DEBUG -fopenmp  -fPIC -xHost -g -debug all -traceback -std=gnu++11 -ansi_alias -D_intel_fast_memcpy=memcpy -D_intel_fast_memset=memset -U__STRICT_ANSI__  -O0 -fopenmp  -rdynamic  "/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/datestamp.o"  -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/lib -lthorn_Cactus -lthorn_CactusBindings -lthorn_Cactus -lthorn_HPX -lthorn_hwloc -lthorn_MPI -lthorn_zlib -lthorn_Boost -lthorn_Cactus -lthorn_CactusBindings  -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/HPX/lib/hpx -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/Boost/lib -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/hwloc/lib -Wl,-rpath,/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/HPX/lib/hpx -Wl,-rpath,/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/Boost/lib -Wl,-rpath,/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/hwloc/lib -Wl,-rpath,:/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/HPX/lib/hpx:/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/Boost/lib:/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/hwloc/lib -fPIC -xHost -g -debug -Lall -traceback -std=gnu++11 -ansi_alias -D_intel_fast_memcpy=memcpy -D_intel_fast_memset=memset -U__STRICT_ANSI__ -O0 -fopenmp -O3 -rdynamic -std=c++0x -Wall -Wno-strict-aliasing -Wsign-promo -Werror=vla -Werror=return-type -Wno-unused-but-set-parameter -Wno-unused-but-set-variable -Werror=uninitialized -wd1170 -wd858 -wd1098 -wd488 -wd2203 -pthread -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/hwloc/lib -L/usr/lib/openmpi/lib -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/Boost/lib -L/usr/local/cuda/lib64 -L/opt/intel/composerxe/bin/lib -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/HPX/lib/hpx -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/Boost/lib -L/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/hwloc/lib -Wl,-rpath,/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/HPX/lib/hpx -Wl,-rpath,/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/Boost/lib -Wl,-rpath,/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/hwloc/lib -Wl,-rpath,:/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/HPX/lib/hpx:/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/Boost/lib:/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/hwloc/lib -fPIC -xHost -g -debug -Wl,-rpath,all -traceback -std=gnu++11 -ansi_alias -D_intel_fast_memcpy=memcpy -D_intel_fast_memset=memset -U__STRICT_ANSI__ -O0 -fopenmp -O3 -rdynamic -std=c++0x -Wall -Wno-strict-aliasing -Wsign-promo -Werror=vla -Werror=return-type -Wno-unused-but-set-parameter -Wno-unused-but-set-variable -Werror=uninitialized -wd1170 -wd858 -wd1098 -wd488 -wd2203 -pthread -Wl,-rpath,/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/hwloc/lib -Wl,-rpath,/usr/lib/openmpi/lib -Wl,-rpath,/xfs1/eschnetter/compute/Cbeta/configs/hpx-debug/scratch/external/Boost/lib -Wl,-rpath,/usr/local/cuda/lib64 -Wl,-rpath,/opt/intel/composerxe/bin/lib -lpthread -lhpx -lhpx_init -lhpx_serialization -lboost_date_time -lboost_filesystem -lboost_program_options -lboost_regex -lboost_serialization -lboost_system -lboost_thread -lboost_atomic -lboost_chrono -lmpi_cxx -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lm -ldl -lhwloc -lm -lcudart -lpthread -lmpi -lmpi_cxx -lz -lboost_filesystem -lboost_system -lm -lm -lm -lm -lm -lm -lm -lm -lcrypt -lcudart -lifcore -limf -lsvml

These are the options used by Cactus, augmented from the HPX options gathered via pkg-config.

This is an Ubuntu system.

[hkaiser]

Which module referenced from the linker command above has the main() entry point?

[eschnett]

Thanks for the explanation.

In Cactus, all thorns are linked into static libraries, which are then passed to the linker. This includes the main routine. In other words, the Cactus main routine (which is now hpx::user_main) is not linked unconditionally, but is searched in the static libraries passed to the linker. I assume that hpx_init.a is a random-access static library, so that its hpx::user_main will be preferred over the one defined in libthorn_Cactus.a.

I don't know how to correct this with link order. Since I have a work-around and we understand the problem, you could just document this and close this issue.

[hkaiser]

Ahh, so the problem is that the correct hpx::user_main lives in a static library (different from hpx_init.a) which makes it almost impossible to force the linker to select the one we need. By coincidence the linker selected the one in hpx_init.a, which is the 'wrong' one... The scheme we use does work only if hpx::user_main is in an object file which is unconditionally linked. That's an interesting and unexpected problem. I'll try to add a note to the documentation about this.