Event-based communication

[soumyadipghosh]

For doing event-based communication, we need a variable called boundary_solution in addition to local_solution as defined in schwarz_solver.hpp. I am wondering what would be the datatype of that variable. For a 2D problem, each PE has 4 boundaries, each of which is a vector - so making it a gko::matrix::Dense is okay ?

[pratikvn]

Yes, gko::matrix::Dense<ValueType> as it is for the local_solution should be fine.

[soumyadipghosh]

I added the following line in schwarz_solver.hpp for declaration:
std::shared_ptr<gko::matrix::Dense<ValueType>> boundary_solution;

Then, in initialize.cpp, I do something like this:
boundary_solution = vec::create(settings.executor, gko::dim<2>(local_size * overlap_size * 4, 1));. Correct me if I am wrong.

[pratikvn]

Why do you need the size to be local_size * overlap_size * 4 ?

[soumyadipghosh]

size of domain across one dimension * overlap at each boundary * 4 boundary PEs (at max) for every PE.

[soumyadipghosh]

I was going through exchange_boundary_onesided in schwarz_solver.cpp. The MPI_Put routines don't have lock or unlock before and after them. How do you do that ? Is it the MPI_Win_lock_all in setup_windows ?

[pratikvn]

Yes, the setup_windows just does the MPI_Win_lock_all and unlocks it later in the clear from the Communicate class. I did initially have the single locks just before and after puts, but it seemed to be slower so I removed them. But if you want to add them for your case feel free to test them out and see if it is slower or faster in your case.

[pratikvn]

Another small note: I manage the formatting by using clang-format. I think it does a good job of keeping a uniform formatting throughout the whole code. The configuration file that I use is also in the root of this git repo. It would be nice if you too could you use it before you commit the code. Thanks.

[soumyadipghosh]

I made some changes to schwarz_solver.cpp. I also did a git merge develop to merge all changes you made to develop. Now when I go to compile the event-based branch, I get the following errors:

Output

[ 10%] Building CXX object CMakeFiles/schwarz.dir/source/initialization.cpp.o
In file included from /afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/source/initialization.cpp:46:
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp: In constructor ‘SchwarzWrappers::device_guard::device_guard(int)’:
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:66:39: error: ‘cudaGetDevice’ was not declared in this scope
         SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaGetDevice(&original_device_id));
                                       ^~~~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:92:25: note: in definition of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
         auto _errcode = _cuda_call;                                    \
                         ^~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:93:25: error: ‘cudaSuccess’ was not declared in this scope
         if (_errcode != cudaSuccess) {                                 \
                         ^~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:66:9: note: in expansion of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
         SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaGetDevice(&original_device_id));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:93:25: note: suggested alternative: ‘euidaccess’
         if (_errcode != cudaSuccess) {                                 \
                         ^~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:66:9: note: in expansion of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
         SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaGetDevice(&original_device_id));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:67:39: error: ‘cudaSetDevice’ was not declared in this scope
         SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaSetDevice(device_id));
                                       ^~~~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:92:25: note: in definition of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
         auto _errcode = _cuda_call;                                    \
                         ^~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:93:25: error: ‘cudaSuccess’ was not declared in this scope
         if (_errcode != cudaSuccess) {                                 \
                         ^~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:67:9: note: in expansion of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
         SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaSetDevice(device_id));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:93:25: note: suggested alternative: ‘euidaccess’
         if (_errcode != cudaSuccess) {                                 \
                         ^~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:67:9: note: in expansion of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
         SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaSetDevice(device_id));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/settings.hpp:51,
                 from /afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/partition_tools.hpp:52,
                 from /afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/initialization.hpp:53,
                 from /afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/source/initialization.cpp:47:
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp: In destructor ‘SchwarzWrappers::device_guard::~device_guard()’:
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:82:13: error: ‘cudaSetDevice’ was not declared in this scope
             cudaSetDevice(original_device_id);
             ^~~~~~~~~~~~~
In file included from /afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/source/initialization.cpp:46:
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:84:43: error: ‘cudaSetDevice’ was not declared in this scope
             SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaSetDevice(original_device_id));
                                           ^~~~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:92:25: note: in definition of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
         auto _errcode = _cuda_call;                                    \
                         ^~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:93:25: error: ‘cudaSuccess’ was not declared in this scope
         if (_errcode != cudaSuccess) {                                 \
                         ^~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:84:13: note: in expansion of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
             SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaSetDevice(original_device_id));
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/exception_helpers.hpp:93:25: note: suggested alternative: ‘euidaccess’
         if (_errcode != cudaSuccess) {                                 \
                         ^~~~~~~~~~~
/afs/crc.nd.edu/user/s/sghosh2/Public/schwarz-lib/include/device_guard.hpp:84:13: note: in expansion of macro ‘SCHWARZ_ASSERT_NO_CUDA_ERRORS’
             SCHWARZ_ASSERT_NO_CUDA_ERRORS(cudaSetDevice(original_device_id));
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/schwarz.dir/source/initialization.cpp.o] Error 1
make[1]: *** [CMakeFiles/schwarz.dir/all] Error 2
make: *** [all] Error 2

[pratikvn]

develop should work now.

[soumyadipghosh]

I get the following error now:

schwarz-lib/source/schwarz_solver.cpp:66:19: error: ‘class gko::OmpExecutor’ has no member named ‘get_exec_info’ ->get_exec_info();

Is it a problem with the Ginkgo version I have (1.0.0) ?

[tcojean]

I think you actually need the branch expt-develop from Ginkgo right now.

[soumyadipghosh]

So the expt-develop is for version 1.1.0? Is get_exec_info a method in 1.1.0 ?

[tcojean]

No, expt-develop has nothing to do with the Ginkgo release versions. Ginkgo 1.1.0 can be found in the most up to date master branch of Ginkgo. expt-develop on the other end has experimental features (machine topology information and more) which this project uses.

[pratikvn]

Yes, @tcojean is correct. You will need to checkout the expt-develop branch now as that contains the get_exec_info method.

[soumyadipghosh]

Ok, develop compiles now, thanks @tcojean.

[soumyadipghosh]

How do I initialize all values of a gko::matrix::Dense type matrix to 0, e.g, in line 797 in develop branch of schwarz_solver.cpp?

Edit: I can initialize the matrix using get_values() after declaring it but is there a built-in function for that?