Add CUDA/HIP implementations of reduction operators #12569

devreal · 2024-05-23T01:03:40Z

This is the second part #12318, which provides the device-side reduction operators and adds stream semantics to ompi_op_reduce.

As usual, the operators are generated from macros. Function pointers to kernel launch functions are stored inside the ompi_op_t as a pointer to a struct that is filled if accelerator support is available.

There are two pieces to the cuda/hip implementation:

A small wrapper that takes care of local configuration specificities (type sizes in Fortran, for example) and that handles OMPI/OPAL types.
An implementation that is agnostic of OMPI/OPAL headers. We found that some vendor compilers didn't like pulling in OMPI/OPAL headers so we had to split.

Currently not supported are short float and long double since they are either not supported everywhere or not standardized. I hope I caught all other types, including pair types for loc functions. Since the implementations are agnostic of OMPI/OPAL headers, the code has to map the fortran types to C types in the implementation.

The device_op_pre and device_op_post functions are there to setup the environment for the kernel, including allocating memory on the device if one of the inputs is not on the chosen device. Operators cannot return an error so whatever the caller feeds us we have to eat. Not pretty, but hopefully better than aborting.

This branch requires #12356. I will rebase once that is merged.

Questions:

Should the rocm component be renamed hip? (I'm afraid it should, @edgargabriel :D)
How do people feel about generating the hip component from the cuda component using hipify+sed scripts? We'd alway require hipify to be available and I could use some help integrating that into the build system but there really isn't much difference between the two.

ompi/mca/op/rocm/Makefile.am

ompi/mca/op/rocm/op_rocm.h

ompi/mca/op/rocm/op_rocm_component.c

ompi/mca/op/rocm/op_rocm_functions.c

edgargabriel · 2024-06-04T13:38:26Z

ompi/mca/op/rocm/op_rocm_functions.c

+    }
+
+    if (MCA_ACCELERATOR_NO_DEVICE_ID == target_device) {
+        opal_accelerator.mem_release_stream(device, target, stream);


just as a thought for a subsequent PR, we could get rid of the mem_alloc and mem_release functions in the accelerator framework interfaces and have only the stream based version, with the default stream being used if no stream argument has been provided by the user. This would reduce the API functions a bit and avoid nearly identical code.

edgargabriel · 2024-06-04T13:41:08Z

ompi/mca/op/rocm/Makefile.am

+sources = op_rocm_component.c op_rocm.h op_rocm_functions.c op_rocm_impl.h
+rocm_sources = op_rocm_impl.hip
+
+HIPCC = hipcc


we might have to change that in the near future. hipcc is going away, we should be using amdclang with --offload-arch arguments. Its ok to leave it for now as is.

bosilca · 2024-05-23T03:32:20Z

ompi/mca/op/cuda/op_cuda.h

+#define xstr(x) #x
+#define str(x) xstr(x)
+
+#define CHECK(fn, args)                                       \


We don't abort inside the software stack.

bosilca · 2024-05-23T03:33:48Z

ompi/mca/op/base/op_base_op_select.c

@@ -152,22 +152,50 @@ int ompi_op_base_op_select(ompi_op_t *op)
        }

        /* Copy over the non-NULL pointers */
-        for (i = 0; i < OMPI_OP_BASE_TYPE_MAX; ++i) {


Too many unnecessary retain/release. We can instead manipulate the avail->ao_module module when needed, starting from the highest to the lowest priority and calling the corresponding op until one returns success.

bosilca · 2024-05-23T03:34:43Z

ompi/mca/op/cuda/Makefile.am

+#sources_extended = op_cuda_functions.cu
+cu_sources = op_cuda_impl.cu
+
+NVCC = nvcc -g


Not good, we need to discover the CUDA compiler via the official mechanisms instead of hardcoding it here. At best you should use the one discovered by the m4 script nvcc_bin but even that is a stretch as we will not know what flags to pass.

It's easy to replace $nvcc_bin but I will need some help with integrating configure flags for the nvcc flags.

ompi/mca/op/cuda/op_cuda_functions.c

bosilca · 2024-06-16T15:43:49Z

ompi/mca/op/cuda/Makefile.am

+
+# -o $($@.o:.lo)
+
+# Open MPI components can be compiled two ways:


This is especially not true for this component, it can only be built dynamically.

The operator support should only be built dynamically? @edgargabriel suggested that they should be made dynamic by default but should we disallow static building entirely?

If I correctly understand, allowing static builds forces libompi.so to have a dependency on CUDA. This will break the build on non-CUDA machines.

The accelerator components are dynamic-by-default (#12055) but I couldn't find a similar mechanism for OMPI. We should still allow building the ops statically, for those who know what they are doing.

As soon as a component calls into libcuda (or more precisely in this case libcudart) it never be built statically.

I'm not sure why that is. The OMPI library would have to be linked against libcudart but that's possible if you build for a CUDA environment specifically.

I marked the two op modules as dso-by-default now.

Just for the sake of it, I build ompi/main with CUDA from scratch, and now the dependency on libcudart exists everywhere, including ompi_info.

Yes, this is broken on main, this branch doesn't change that.

ompi/mca/op/cuda/op_cuda_impl.cu

bosilca · 2024-06-17T13:48:06Z

ompi/mca/op/cuda/op_cuda_impl.cu

+        const int stride = blockDim.x * gridDim.x;                                                  \
+        for (int i = index; i < n/vlen; i += stride) {                                              \
+            vtype vin = ((vtype*)in)[i];                                                            \
+            vtype vinout = ((vtype*)inout)[i];                                                      \


Why don't you use the templated op defined earlier in the file ? Or if you don't need it you should remove it.

I am reworking the vectorization to make it more flexible and avoid some of the stuff I had to do to map the fixed size integers onto vectors of variable size integers.

I reworked the vectorization with a custom type and some template work. The goal now is to consistently have 128bit loads and stores.

ompi/mca/op/cuda/op_cuda_impl.cu

bosilca · 2024-06-17T13:54:41Z

ompi/mca/op/op.h

    /** Function pointers for all the different datatypes to be used
        with the MPI_Op that this module is used with */
-    ompi_op_base_handler_fn_1_0_0_t opm_fns[OMPI_OP_BASE_TYPE_MAX];
-    ompi_op_base_3buff_handler_fn_1_0_0_t opm_3buff_fns[OMPI_OP_BASE_TYPE_MAX];
+    union {


Overly complicated, but I can't think of anything significantly better right now.

ompi/op/op.h

devreal · 2024-06-19T15:38:51Z

ompi/mca/op/rocm/op_rocm_functions.c

+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wgnu-zero-variadic-macro-arguments"
+
+static inline void device_op_pre(const void *orig_source1,


@bosilca @edgargabriel
If the device_op_pre and device_op_post use the accelerator framework they are pretty much independent of the the model (minus the last two lines). I wonder whether they should be moved to a header in base/ and shared between the two implementations.

The last two lines can be taken out and put into the op macro from where they are called.

ompi/mca/op/cuda/op_cuda_component.c

bosilca · 2024-06-21T00:30:02Z

Let me add some generic comment here, mostly as a reminder to self. The reason is that I don't think this is how we should use these op modules, especially not with accelerators. In my vision we decide once and for all, for each operation (or collective) what MPI_Op we will use, and we will stay with it for the entire duration. First, because there is no reason to execute half of the MPI_Op on the host and the other half on the device, it is all or none. Second, because we definitely don't want to start each kernel independently, the overhead will be just too costly, annihilating most of the benefits. Instead, once we start a collective, we would start a "service" bound to a specific context (GPU or CPU) and this service will remain active for as long as we are in a collective that needs GPU op, removing all costs related to the kernel submission. Instead, the GPU threads will poll into a well defined memory location for work updates, and the CPU will post new ops in this queue.

The only drawback I can see is that the service will take resources from the application, but this loss is very small, as a single (or two SM) are more than enough to saturate the network bandwidth. Once, we are outside collectives requiring GPU op, we can release these resources back to the application.

The operators are generated from macros. Function pointers to kernel launch functions are stored inside the ompi_op_t as a pointer to a struct that is filled if accelerator support is available. The ompi_op* API is extended to include versions taking streams and device IDs to allow enqueuing operators on streams. The old functions map to the stream versions with a NULL stream. Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

CUDA provides only limited vector widths and only for variable width integer types. We use our own vector type and some C++ templates to get more flexible vectors. We aim to get 128bit loads by adjusting the width based on the type size. Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

devreal · 2024-07-02T02:53:02Z

The CUDA test fails because we detect CUDA but NVCC is not available (at least in PATH). We'll need to check for NVCC to be available and bail out if not. Ideally, we can make NVCC available in CI as well.

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

bosilca · 2024-07-08T19:18:41Z

config/opal_check_nvcc.m4

+          # no path specified, try to find nvcc
+          [AC_PATH_PROG([OPAL_NVCC], [nvcc], [])])
+
+    # If the user requested to disable sphinx, then pretend we didn't


nvcc not sphinx I assume

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

devreal · 2024-07-09T22:00:58Z

I updated the PR to have precious variables NVCC, HIPCC, NVCCFLAGS, and HIPCCFLAGS so that they can be specified on the command line and show up at the bottom ./configure --help. I will squash all changes and rebase once approved.

devreal added the Target: main label May 23, 2024

devreal requested review from bosilca, Akshay-Venkatesh and edgargabriel May 23, 2024 01:03

edgargabriel requested changes Jun 4, 2024

View reviewed changes

This was referenced Jun 10, 2024

Add accelerator-awareness to most allreduce implementations #12611

Open

Reduce_local Segmentation fault when Running with IMB-MPI1 built for GPU #12620

Closed

bosilca reviewed Jun 17, 2024

View reviewed changes

devreal force-pushed the op-accel-kernels branch 2 times, most recently from 3ab3371 to 3e4425d Compare June 19, 2024 12:48

devreal commented Jun 19, 2024

View reviewed changes

devreal force-pushed the op-accel-kernels branch from 3d1ef9c to 6a85957 Compare June 20, 2024 01:00

bosilca reviewed Jun 20, 2024

View reviewed changes

ompi/mca/op/cuda/op_cuda_component.c Outdated Show resolved Hide resolved

devreal force-pushed the op-accel-kernels branch 2 times, most recently from f689d6d to 25c24c9 Compare June 20, 2024 16:07

devreal and others added 7 commits July 1, 2024 13:46

Build op/cuda and op/rocm as dso by default

bc0f531

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

Remove DECLSPEC from internal functions

6a4c65d

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

op/cuda: Lazily initialize the CUDA information

6aaa666

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

op/cuda: cleanup and remove short float remnants

ea5a6df

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

Add LDFLAGS to op/rocm linker command

b56ab0a

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

devreal force-pushed the op-accel-kernels branch from 1ed9579 to b56ab0a Compare July 1, 2024 17:46

First attempt to check for NVCC

5fbc2d4

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

devreal force-pushed the op-accel-kernels branch from e9d9a16 to 5fbc2d4 Compare July 8, 2024 13:48

Add check for hipcc

a4a84f5

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

devreal force-pushed the op-accel-kernels branch from 4d73198 to a4a84f5 Compare July 8, 2024 18:59

bosilca reviewed Jul 8, 2024

View reviewed changes

devreal added 3 commits July 8, 2024 18:12

Mark NVCC, NVCCFLAGS, HIPCC, and HIPCCFLAGS as precious

13487fd

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

Point CI workflows to nvcc/hipcc

717b3db

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

Merge branch 'main' into op-accel-kernels

82d0090

Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CUDA/HIP implementations of reduction operators #12569

Add CUDA/HIP implementations of reduction operators #12569

devreal commented May 23, 2024

edgargabriel Jun 4, 2024

edgargabriel Jun 4, 2024

bosilca May 23, 2024

bosilca May 23, 2024

bosilca May 23, 2024

devreal Jun 19, 2024

bosilca Jun 16, 2024

devreal Jun 19, 2024

bosilca Jun 19, 2024

devreal Jun 19, 2024

bosilca Jun 19, 2024 •

edited

Loading

devreal Jun 19, 2024

bosilca Jun 20, 2024

devreal Jun 20, 2024

bosilca Jun 17, 2024

devreal Jun 20, 2024

devreal Jun 20, 2024

bosilca Jun 17, 2024

devreal Jun 19, 2024

bosilca commented Jun 21, 2024

devreal commented Jul 2, 2024

bosilca Jul 8, 2024

devreal commented Jul 9, 2024


		# -o $($@.o:.lo)

		# Open MPI components can be compiled two ways:

Add CUDA/HIP implementations of reduction operators #12569

Are you sure you want to change the base?

Add CUDA/HIP implementations of reduction operators #12569

Conversation

devreal commented May 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bosilca Jun 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bosilca commented Jun 21, 2024

devreal commented Jul 2, 2024

Choose a reason for hiding this comment

devreal commented Jul 9, 2024

bosilca Jun 19, 2024 •

edited

Loading