OpenACC: add atomics support #6446

dalg24 · 2023-09-14T03:15:10Z

Supersedes #5955

… on a sequential host

- Remove const_cast() - Change Kokkos::abort() with printf() - Add FIXME_OPENACC comment.

(OpenACC C/C++ does not support atomic max/min/mod operations) Disable TestOpenACC_BitManipulationBuiltins for OpenACC due to errors.

…older).

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

…or older)

PR.

…r the OpenACC backend.

backend.

suggested by the code review.

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

functions.

code review.

Re-enable atomic max/min tests for OpenACC.

dalg24 · 2023-10-12T14:19:27Z

Retest this please

…fail due to compiler bugs, which are reported to NVIDIA. - Change the values of start and end variables in TestAtomicOperations_double.hpp and TestAtomicOperations_float.hpp so that atomic-division tests calculate trivial divisions. (In the original tests, NVHPC compiler failed since device results are slightly different from the host results due to precision mismatch.)

host and device atomic operations using a relative error.

dalg24 · 2023-11-03T03:07:45Z

It looks good to me. Let's re trigger the CI.
Retest this please.

This reverts commit 18132bf.

dalg24 · 2023-11-06T05:46:53Z

This is ready.
It will conflict with #6534 but these are trivial to resolve.

tpls/desul/include/desul/atomics/Compare_Exchange_OpenACC.hpp

tpls/desul/include/desul/atomics/Lock_Based_Fetch_Op_OpenACC.hpp

core/unit_test/TestAtomicOperations.hpp

masterleinad · 2023-11-07T19:57:33Z

core/unit_test/TestAtomicOperations.hpp

+#if defined(KOKKOS_ENABLE_OPENACC) && defined(KOKKOS_COMPILER_NVHPC)
+    // NVHPC may use different internal precisions for the device and host
+    // atomic operations. Therefore, relative errors are used to compare the
+    // host results and device results.
+    case 5:
+      return update != 0 ? atomic_op_test_rel<DivAtomicTest, T, ExecSpace>(
+                               old_val, update)
+                         : true;
+#else


Note that we have very similar "problems" for 32-bit support.

tpls/desul/include/desul/atomics/Compare_Exchange_OpenACC.hpp

tpls/desul/include/desul/atomics/Lock_Based_Fetch_Op_OpenACC.hpp

dalg24 · 2023-11-08T18:53:23Z

(I applied Bruno's suggestion (proposed in desul/desul#118) and merged develop to make sure there is no surprise after the SYCL updates)

dalg24 · 2023-11-08T21:12:10Z

Retest this please

Rombur · 2023-11-13T15:08:28Z

This PR outputs an insane number of messages
1: DESUL error in device_atomic_compare_exchange(): Not supported atomic operation in the OpenACC backend
see https://cloud.cees.ornl.gov/jenkins-ci/job/Kokkos/14840/consoleFull

I think it's filling up the server with the log.

dalg24 · 2023-11-13T18:01:06Z

Are you able to fix?

Rombur · 2023-11-13T18:14:26Z

Still waiting to get access to the machine to see if that's the issue.

seyonglee · 2023-11-14T05:00:08Z

Are you able to fix?

Those error messages are printed because some of the internal Kokkos host codes call atomic_compare_exchange() API with non-arithmetic data types. One way to fix this issue would be to silence this message when atomic_compare_exchange() API with a non-arithmetic type is called on the host.

seyonglee and others added 30 commits July 20, 2023 12:29

Initial OpenACC atomic construct implementation.

4c1738b

Partially fixed bugs in the OpenACC atomic implementations.

354ae54

First working version, where general atomic implementations work only…

6cf197e

… on a sequential host

Update Unit Test CMake

2366a86

Update as suggested by code review:

4291920

- Remove const_cast() - Change Kokkos::abort() with printf() - Add FIXME_OPENACC comment.

Disable unsupporte OpenACC atomic tests.

d79b45c

(OpenACC C/C++ does not support atomic max/min/mod operations) Disable TestOpenACC_BitManipulationBuiltins for OpenACC due to errors.

Apply ClangFormat

94da50f

Disable unsupported unit tests when by old NVHPC compilers (V22.5 or …

a8f0d88

…older).

Apply ClangFormat

a85e293

Update tpls/desul/include/desul/atomics/Fetch_Op_OpenACC.hpp

b7a6c33

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

Restore unit tests that were disabled for old NVHPC compilers (V22.5 …

0d8b2b3

…or older)

Update unit test CMakeLists.txt to include unit tests enabled by this

968b097

PR.

Change the minimum version of the NVHPC compiler from 22.3 to 22.9 fo…

01ee8b9

…r the OpenACC backend.

Changed the way to guard unsupported atomic tests for the OpenACC

1d756c0

backend.

Remove unnecessary guarding on unsupported atomic tests for the OpenACC

ead41ff

backend.

Minor updates according to the code review.

1908f4c

Changed the supported-type-checking code from macro to alias template as

b1bc407

suggested by the code review.

Undo changing the minimim required version of NVHPC.

5ffa9c6

Apply suggestions from code review

628abd9

Co-authored-by: Damien L-G <dalg24+github@gmail.com>

Change the KOKKOS_COMPILER_NVHPC macro to __NVCOMPILER

3b73f61

Rename a variable's name from tmp to `old in atomic_fetch_*()

98c90a8

functions.

Change ptr[0] to *ptr as suggested by the code review.

c8b613a

Add FIXME comments in device_atomic_thread_fence().

5d4e66e

Moved definitions into the desul::Impl namespace as suggested by the

e43656e

code review.

Clean up the OpenACC atomic implementations.

f63a6c4

Re-enable atomic max/min tests for OpenACC.

Fix a typo (sid => std)

72ddf87

Minor bug fix on OpenACC

0692166

Merge branch 'develop' into openacc_atomics

7d8a0b6

Update .jenkins to the latest.

75f91b0

Test this please

f4370be

dalg24 added 3 commits October 11, 2023 10:49

Disable complex float atomic unit tests for OpenACC backend

372ed18

Sync with PR opened on the desul side

f0a72ad

DO NOT MERGE disable all CI but OpenACC

18132bf

dalg24 force-pushed the openacc_atomics branch from 2536a2d to 18132bf Compare October 11, 2023 17:57

seyonglee added 2 commits October 27, 2023 23:07

Add atomic_op_test)rel() to TestAtomicOperations.hpp, which compares the

cfa7360

host and device atomic operations using a relative error.

Revert "DO NOT MERGE disable all CI but OpenACC"

585f7dc

This reverts commit 18132bf.

dalg24 changed the title ~~Nothing to see move along~~ OpenACC: add atomics support Nov 3, 2023

dalg24 mentioned this pull request Nov 3, 2023

Add support for OpenACC atomics desul/desul#108

Merged

dalg24 marked this pull request as ready for review November 6, 2023 05:45

Rombur reviewed Nov 6, 2023

View reviewed changes

tpls/desul/include/desul/atomics/Compare_Exchange_OpenACC.hpp Show resolved Hide resolved

tpls/desul/include/desul/atomics/Compare_Exchange_OpenACC.hpp Show resolved Hide resolved

tpls/desul/include/desul/atomics/Lock_Based_Fetch_Op_OpenACC.hpp Outdated Show resolved Hide resolved

masterleinad reviewed Nov 7, 2023

View reviewed changes

dalg24 mentioned this pull request Nov 8, 2023

[atomics] Fixup Kokkos -> DESUL error message with OpenACC desul/desul#118

Merged

dalg24 added 2 commits November 8, 2023 08:45

Merge remote-tracking branch 'upstream/develop' into openacc_atomics

6202616

[desul_atomics] Fixup Kokkos -> DESUL in error message with OpenACC

cc92c5c

masterleinad approved these changes Nov 9, 2023

View reviewed changes

seyonglee approved these changes Nov 9, 2023

View reviewed changes

dalg24 merged commit 97a90d5 into kokkos:develop Nov 9, 2023
27 of 28 checks passed

dalg24 deleted the openacc_atomics branch November 9, 2023 23:15

dalg24 mentioned this pull request Nov 9, 2023

CHANGELOG: 4.3.0 #6519

Closed

masterleinad mentioned this pull request Nov 10, 2023

Clean up atomic_op_test[_rel] #6591

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenACC: add atomics support #6446

OpenACC: add atomics support #6446

dalg24 commented Sep 14, 2023 •

edited

dalg24 commented Oct 12, 2023

dalg24 commented Nov 3, 2023

dalg24 commented Nov 6, 2023

masterleinad Nov 7, 2023

dalg24 commented Nov 8, 2023

dalg24 commented Nov 8, 2023

Rombur commented Nov 13, 2023

dalg24 commented Nov 13, 2023

Rombur commented Nov 13, 2023

seyonglee commented Nov 14, 2023

OpenACC: add atomics support #6446

OpenACC: add atomics support #6446

Conversation

dalg24 commented Sep 14, 2023 • edited

dalg24 commented Oct 12, 2023

dalg24 commented Nov 3, 2023

dalg24 commented Nov 6, 2023

masterleinad Nov 7, 2023

Choose a reason for hiding this comment

dalg24 commented Nov 8, 2023

dalg24 commented Nov 8, 2023

Rombur commented Nov 13, 2023

dalg24 commented Nov 13, 2023

Rombur commented Nov 13, 2023

seyonglee commented Nov 14, 2023

dalg24 commented Sep 14, 2023 •

edited