Content

• How To
• Configuration
• Time Measurement
• Tasks
• Results

How To

1. Create github account (if not exists);
2. Make sure SSH clone & commit is working (Connecting to GitHub with SSH);
3. Fork this repo (just click Fork button on the top of the page, detailed instructions here)
4. Clone your forked repo into your local machine, use your user instead of username:

git clone git@github.com:username/cuda-2026.git
cd cuda-2026

5. Go to your group folder, e.g.:

cd 3822B1FI1

6. Go to needed task folder, e.g.:

cd 1_gelu_omp

7. Create new folder with your surname and name (make sure it's the same for all tasks), e.g.:

mkdir petrov_ivan

8. Copy your task source/header files (including main program) into this folder (use copy instead of cp on Windows), e.g.:

cd petrov_ivan
cp /home/usr/lab/*.cpp .
cp /home/usr/lab/*.h .

8. Push your sources to github repo, e.g.:

cd ..
git add .
git commit -m "1_gelu_omp task"
git push

9. Go to your repo in browser, click Contribute button on the top of page, then Open pull request. Provide meaningfull request title and description, then Create pull request (see details here).
10. Go to Pull Requests page in course repo, find your pull request and check if there are no any merge conflicts occur. If merge conflicts happen - resolve it following the instruction provided by github.

Time Measurement

The following scheme is used to measure task execution time:

int main() {
    // ...

    // Warming-up
    Task(input, size);

    // Performance Measuring
    std::vector<double> time_list;
    for (int i = 0; i < 4; ++i) {
        auto start = std::chrono::high_resolution_clock::now();
        Task(input, size);
        auto end = std::chrono::high_resolution_clock::now();
        std::chrono::duration<double> duration = end - start;
        time_list.push_back(duration.count());
    }
    double time = *std::min_element(time_list.begin(), time_list.end());

    // ...
}

Configuration

• CPU: Intel Core i5 12600K (4 cores, 4 threads)
• RAM: 16 GB
• GPU: NVIDIA RTX 4060 (8 GB)
• OS: Ubuntu 22.04.3 LTS
• Host Compiler: GCC 11.4.0 (C++17)
• CUDA: 12.9

Tasks

Task #1: OpenMP GELU Implementation

The Gaussian Error Linear Unit (GELU) is an activation function frequently used in Deep Neural Networks (DNNs) and can be thought of as a smoother ReLU.

To approximate GELU function, use the following formula:

GELU(x) = $0.5x(1 + tanh(\sqrt{2 / \pi}(x + 0.044715 * x^3)))$

Implement the function with the following interface in C++:

std::vector<float> GeluOMP(const std::vector<float>& input);

Size of result vector should be the same as for input. Use OpenMP technology to make your function parallel & fast.

Two files are expected to be uploaded:

• gelu_omp.h

#ifndef __GELU_OMP_H
#define __GELU_OMP_H

#include <vector>

std::vector<float> GeluOMP(const std::vector<float>& input);

#endif // __GELU_OMP_H

• gelu_omp.cpp

#include "gelu_omp.h"

std::vector<float> GeluOMP(const std::vector<float>& input) {
    // Place your implementation here
}

Performance Hints:

• better formula to compute GELU, e.g. replace tanh() with exp();
• loop unrolling;
• loop vectorization;
• vector allocation and computations in different threads (Windows only).

Task #2: CUDA GELU Implementation

Implement the function with the following interface in CUDA C++ using the formula described above:

std::vector<float> GeluCUDA(const std::vector<float>& input);

Size of result vector should be the same as for input. Use CUDA technology to make your function work on NVIDIA GPU. Try to make it fast.

Two files are expected to be uploaded:

• gelu_cuda.h

#ifndef __GELU_CUDA_H
#define __GELU_CUDA_H

#include <vector>

std::vector<float> GeluCUDA(const std::vector<float>& input);

#endif // __GELU_CUDA_H

• gelu_cuda.cu

#include "gelu_cuda.h"

std::vector<float> GeluCUDA(const std::vector<float>& input) {
    // Place your implementation here
}

Performance Hints:

• overlap host memory allocation and CUDA computations;
• allocate and free device memory once;
• use better formula to compute GELU, e.g. replace tanh() with exp().

Task #3: Naive Matrix Multiplication using OpenMP

General matrix multiplication (GEMM) is a very basic and broadly used linear algebra operation applied in high performance computing (HPC), statistics, deep learning and other domains. There are a lot of GEMM algorithms with different mathematical complexity form $O(n^3)$ for naive and block approaches to $O(n^{2.371552})$ for the method descibed by Williams et al. in 2024 [1]. But despite a variety of algorithms with low complexity, block matrix multiplication remains the most used implementation in practice since it fits to modern HW better.

To start learning matrix multiplication smoother, let us start with naive approach here. To compute matrix multiplication result C for matricies A and B, where C = A * B and the size for all matricies are $n*n$, one should use the following formula for each element of C (will consider only square matricies for simplicity):

$c_{ij}=\sum_{k=1}^na_{ik}b_{kj}$

To complete the task one should implement a function that multiplies two square matricies using OpenMP with the following interface:

std::vector<float> NaiveGemmOMP(const std::vector<float>& a,
                                const std::vector<float>& b,
                                int n);

Each matrix must be stored in a linear array by rows, so that a.size()==n*n. Function takes two matricies and their size as inputs, and returns result matrix also stored by rows.

For simplicity, let's consider matrix size is always power of 2.

Two files are expected to be uploaded:

• naive_gemm_omp.h:

#ifndef __NAIVE_GEMM_OMP_H
#define __NAIVE_GEMM_OMP_H

#include <vector>

std::vector<float> NaiveGemmOMP(const std::vector<float>& a,
                                const std::vector<float>& b,
                                int n);

#endif // __NAIVE_GEMM_OMP_H

• naive_gemm_omp.cpp:

#include "naive_gemm_omp.h"

std::vector<float> NaiveGemmOMP(const std::vector<float>& a,
                                const std::vector<float>& b,
                                int n) {
    // Place your implementation here
}

Performance Hints:

• cache-friendly memory accesses;
• loop unrolling;
• loop vectorization.

Task #4: Naive Matrix Multiplication using CUDA

In this task one should implement naive approach for matrix multiplication in CUDA trying to make it fast enough (pay attention to global memory accesses in your code).

Each matrix must be stored in a linear array by rows, so that a.size()==n*n. Function takes two matricies and their size as inputs, and returns result matrix also stored by rows.

For simplicity, let's consider matrix size is always power of 2.

Two files are expected to be uploaded:

• naive_gemm_cuda.h:

#ifndef __NAIVE_GEMM_CUDA_H
#define __NAIVE_GEMM_CUDA_H

#include <vector>

std::vector<float> NaiveGemmCUDA(const std::vector<float>& a,
                                 const std::vector<float>& b,
                                 int n);

#endif // __NAIVE_GEMM_CUDA_H

• naive_gemm_cuda.cu:

#include "naive_gemm_cuda.h"

std::vector<float> NaiveGemmCUDA(const std::vector<float>& a,
                                 const std::vector<float>& b,
                                 int n) {
    // Place your implementation here
}

Performance Hints:

• warp-friendly memory accesses;
• multiple elements per warp processing;
• loop unrolling and memory load vectorization;
• block size selection;
• overlap host memory allocation and CUDA computations.

Task #5: Block Matrix Multiplication using OpenMP

In real applications block-based approach for matrix multiplication can get multiple times faster execution comparing with naive version due to cache friendly approach. To prove this in practice, implement such a version in C++ using OpenMP.

In block version algorithm could be divided into three stages:

1. Split matricies into blocks (block size normally affects performance significantly so choose it consciously);
2. Multiply two blocks to get partial result;
3. Replay step 2 for all row/column blocks accumulating values into a single result block.

From math perspective, block matrix multiplication could be described by the following formula, where $C_{IJ}$, $A_{IK}$ and $B_{KJ}$ are sub-matricies with the size $block_size*block_size$:

$C_{IJ}=\sum_{k=1}^{block_count}A_{IK}B_{KJ}$

Each matrix must be stored in a linear array by rows, so that a.size()==n*n. Function takes two matricies and their size as inputs, and returns result matrix also stored by rows.

For simplicity, let's consider matrix size is always power of 2.

Two files are expected to be uploaded:

• block_gemm_omp.h:

#ifndef __BLOCK_GEMM_OMP_H
#define __BLOCK_GEMM_OMP_H

#include <vector>

std::vector<float> BlockGemmOMP(const std::vector<float>& a,
                                const std::vector<float>& b,
                                int n);

#endif // __BLOCK_GEMM_OMP_H

• block_gemm_omp.cpp:

#include "block_gemm_omp.h"

std::vector<float> BlockGemmOMP(const std::vector<float>& a,
                                const std::vector<float>& b,
                                int n) {
    // Place your implementation here
}

As in previous task, let us consider all matricies are square.

Performance Hints:

• cache-friendly memory accesses;
• loop unrolling;
• loop vectorization.

Task #6: Block Matrix Multiplication using CUDA

In CUDA C++ block-based approach looks similar. But to get better performance one should use CUDA shared memory to store each particular block while computations. With this consideration, algorithm will be the following:

1. A single CUDA block should compute a single block of result matrix C, a single CUDA thread - a single matrix C element;
2. For each A block in a row and B block in a column:
   1. Load A block into shared memory;
   2. Load B block into shared memory;
   3. Synchronize over all threads in block;
   4. Compute BlockA * BlockB and accumulate into C block in shared memory;
   5. Synchronize over all threads in block;
3. Dump block C from shared to global memory.

Each matrix must be stored in a linear array by rows, so that a.size()==n*n. Function takes two matricies and their size as inputs, and returns result matrix also stored by rows.

For simplicity, let's consider matrix size is always power of 2.

Two files are expected to be uploaded:

• block_gemm_cuda.h:

#ifndef __BLOCK_GEMM_CUDA_H
#define __BLOCK_GEMM_CUDA_H

#include <vector>

std::vector<float> BlockGemmCUDA(const std::vector<float>& a,
                                 const std::vector<float>& b,
                                 int n);

#endif // __BLOCK_GEMM_CUDA_H

• block_gemm_cuda.cu:

#include "block_gemm_cuda.h"

std::vector<float> BlockGemmCUDA(const std::vector<float>& a,
                                 const std::vector<float>& b,
                                 int n) {
    // Place your implementation here
}

Performance Hints:

• shared memory usage to store matrix block;
• warp-friendly memory accesses;
• multiple elements per warp processing;
• loop unrolling and memory load vectorization;
• block size selection;
• overlap host memory allocation and CUDA computations.

Task #7: Matrix Multiplication using cuBLAS

The most performant way to multiply two matrices on particular hardware is to use vendor-provided library for this purpose. In CUDA it's cuBLAS. Try to use cuBLAS API to implement general matrix multiplication in most performant way.

Each matrix must be stored in a linear array by rows, so that a.size()==n*n. Function takes two matricies and their size as inputs, and returns result matrix also stored by rows.

For simplicity, let's consider matrix size is always power of 2.

Note, that in cuBLAS API matrix is expected to be stored by columns, so additional transpose may be required.

Two files are expected to be uploaded:

• gemm_cublas.h:

#ifndef __GEMM_CUBLAS_H
#define __GEMM_CUBLAS_H

#include <vector>

std::vector<float> GemmCUBLAS(const std::vector<float>& a,
                              const std::vector<float>& b,
                              int n);

#endif // __GEMM_CUBLAS_H

• gemm_cublas.cu:

#include "gemm_cublas.h"

std::vector<float> GemmCUBLAS(const std::vector<float>& a,
                              const std::vector<float>& b,
                              int n) {
    // Place your implementation here
}

Performance Hints:

• overlap host memory allocation and CUDA computations;
• avoid redundant device memory allocation.

Task #8: FFT (Fast Fourier Transform) using cuFFT

Another widely used operation in HPC & signal processing is discrete Fourier Transform. Naive approach (by definition) has $O(n^2)$ complexity and is not used in practice due to its slowness. Better way is Fast Fourier Transform (FFT) algorithm with $O(n*log(n))$ complexity.

Due to its frequent use, FFT algorithm implementation is normally a part of vendor-optimized solutions for various hardware chips. For NVIDIA GPUs one should take cuFFT library.

To pass the task one should implement a funtion that takes $batch$ signals of $n$ complex elements, and performs complex-to-complex forward and than inverse Fourier transform for them. For better performance use cuFFT API.

Required function should have the following prototype:

std::vector<float> FffCUFFT(const std::vector<float>& input, int batch);

Here $batch$ is a number of independent signals, $input$ contains complex values in the format of $(real, imaginary)$ pairs of floats storing pair by pair. So $input$ array size must be equal to $2 * n * batch$.

The function should perform the following actions:

1. Compute forward Fourier transform for $input$;
2. Compute inverse Fourier transform for the result of step 1;
3. Normalize result of step 2 by $n$.

Returned array must store result of step 3 in the same format of $(real, imaginary)$ pairs as $input$ and have the same size.

Note, that due to Fourier Transform math properties, result array will have the same values as input one. This specificity could be used for self-checking.

Two files are expected to be uploaded:

• fft_cufft.h:

#ifndef __FFT_CUFFT_H
#define __FFT_CUFFT_H

#include <vector>

std::vector<float> FffCUFFT(const std::vector<float>& input, int batch);

#endif // __FFT_CUFFT_H

• fft_cufft.cu:

#include "fft_cufft.h"

std::vector<float> FffCUFFT(const std::vector<float>& input, int batch) {
    // Place your implementation here
}

Performance Hints:

• make normalization on device;
• do not allocate redundant device memory;
• overlap host memory allocation and CUDA computations.

Task #9: OpenCL GELU Implementation

Implement GELU function with the following interface in OpenCL using the formula described in task #1:

std::vector<float> GeluOCL(const std::vector<float>& input, int platform);

Size of result vector should be the same as for input. Use OpenCL technology to make your function work on NVIDIA GPU. Try to make it fast.

Use CL_DEVICE_GPU flag to choose GPU device. Use platform platform and 0 device. Store your OpenCL kernel in a string constant.

Two files are expected to be uploaded:

• gelu_ocl.h

#ifndef __GELU_OCL_H
#define __GELU_OCL_H

#include <vector>

std::vector<float> GeluOCL(const std::vector<float>& input, int platform);

#endif // __GELU_OCL_H

• gelu_ocl.cpp

#include "gelu_ocl.h"

std::vector<float> GeluOCL(const std::vector<float>& input, int platform) {
    // Place your implementation here
}

Performance Hints:

• perform OpenCL boilerplate code once;
• use better formula to compute GELU, e.g. replace tanh() with exp();
• overlap host memory allocation and GPU computations.

Results

1_gelu_omp (134217728 elements)

Group	Name	Result	Rank
FAST	FAST	0.1083	-
3823B1FI3	kurpiakov_alexei	0.1458	13
3823B1FI3	kurpiakov_aleksei	0.1523	14
3823B1FI3	kichanova_ksenia	0.1554	12
3823B1FI3	kutergin_valentin	0.2395	7
3823B1FI3	romanov_artem	0.2437	4
3823B1FI3	baldin_andrew	0.2467	3
3823B1FI3	gutyansky_alexey	0.2485	5
3823B1FI3	pylaeva_svetlana	0.2500	11
3823B1FI3	levonychev_ivan	0.2506	8
3823B1FI3	papulina_yuliya	0.2558	10
3823B1FI3	lukin_ivan	0.2660	2
3823B1FI3	votincev_dmitri	0.2775	1
3823B1FI3	zavyalov_alexey	0.3588	9
REF	REF	0.7275	-
3823B1FI3	chacshin_vladimir	RUN FAILED	6

2_gelu_cuda (134217728 elements)

Group	Name	Result	Rank
3823B1FI3	zavyalov_alexey	0.0819	5
FAST	FAST	0.1455	-
3823B1FI3	romanov_artem	0.1519	7
3823B1FI3	kichanova_ksenia	0.1553	6
3823B1FI3	levonychev_ivan	0.1754	8
3823B1FI3	kutergin_valentin	0.1806	3
3823B1FI3	lukin_ivan	0.1868	4
3823B1FI3	votincev_dmitri	0.2079	2
3823B1FI3	baldin_andrew	0.2152	1
REF	REF	0.2167	-

3_naive_gemm_omp (1024 elements)

Group	Name	Result	Rank
3823B1FI3	kurpiakov_aleksei	0.0179	14
3823B1FI3	kurpiakov_alexei	0.0189	11
3823B1FI3	levonychev_ivan	0.0195	13
3823B1FI3	kutergin_valentin	0.0214	9
3823B1FI3	kichanova_ksenia	0.0214	12
3823B1FI3	papulina_yuliya	0.0220	8
3823B1FI3	baldin_andrew	0.0253	4
FAST	FAST	0.0254	-
3823B1FI3	votincev_dmitri	0.0255	2
3823B1FI3	chacshin_vladimir	0.0257	3
3823B1FI3	zavyalov_alexey	0.0264	6
3823B1FI3	gutyansky_alexey	0.0268	7
3823B1FI3	pylaeva_svetlana	0.0270	10
3823B1FI3	lukin_ivan	0.0277	5
3823B1FI3	romanov_artem	0.0325	1
REF	REF	0.7279	-

4_naive_gemm_cuda (4096 elements)

Group	Name	Result	Rank
3823B1FI3	kutergin_valentin	0.0933	3
3823B1FI3	kichanova_ksenia	0.1231	4
3823B1FI3	votincev_dmitri	0.1465	7
FAST	FAST	0.1591	-
3823B1FI3	levonychev_ivan	0.1599	5
3823B1FI3	romanov_artem	0.1718	6
3823B1FI3	baldin_andrew	0.2100	1
3823B1FI3	zavyalov_alexey	0.2321	2
REF	REF	0.5797	-

5_block_gemm_omp (1024 elements)

Group	Name	Result	Rank
3823B1FI3	kurpiakov_aleksei	0.0159	10
3823B1FI3	levonychev_ivan	0.0204	12
3823B1FI3	kutergin_valentin	0.0220	11
FAST	FAST	0.0223	-
3823B1FI3	kichanova_ksenia	0.0235	9
3823B1FI3	chacshin_vladimir	0.0262	2
3823B1FI3	romanov_artem	0.0264	1
3823B1FI3	gutyansky_alexey	0.0283	7
3823B1FI3	lukin_ivan	0.0284	4
3823B1FI3	pylaeva_svetlana	0.0291	8
3823B1FI3	zavyalov_alexey	0.0294	5
3823B1FI3	baldin_andrew	0.0353	3
3823B1FI3	papulina_yuliya	0.0387	6
REF	REF	0.1666	-
3823B1FI3	votincev_dmitri	BUILD FAILED	-

6_block_gemm_cuda (4096 elements)

Group	Name	Result	Rank
3823B1FI3	kichanova_ksenia	0.1218	1
3823B1FI3	baldin_andrew	0.1260	2
3823B1FI3	kutergin_valentin	0.1268	3
3823B1FI3	romanov_artem	0.1384	5
3823B1FI3	votincev_dmitri	0.1445	4
FAST	FAST	0.1469	-
REF	REF	0.7454	-

7_gemm_cublas (4096 elements)

Group	Name	Result	Rank
3823B1FI3	levonychev_ivan	0.0368	1
3823B1FI3	kurpiakov_aleksei	0.0375	5
3823B1FI3	romanov_artem	0.0377	4
3823B1FI3	baldin_andrew	0.0383	2
3823B1FI3	votincev_dmitri	0.0386	3
FAST	FAST	0.0484	-
REF	REF	0.0534	-
3823B1FI3	kichanova_ksenia	TEST FAILED	-

8_fft_cufft (131072 elements)

Group	Name	Result	Rank
3823B1FI3	kichanova_ksenia	0.0729	1
3823B1FI3	baldin_andrew	0.0844	4
3823B1FI3	kurpiakov_aleksei	0.0867	3
3823B1FI3	votincev_dmitri	0.0892	5
FAST	FAST	0.0916	-
3823B1FI3	levonychev_ivan	0.0987	2
REF	REF	0.2027	-

9_gelu_ocl (134217728 elements)

Group	Name	Result	Rank
FAST	FAST	0.1449	-
3823B1FI3	levonychev_ivan	0.2502	1
3823B1FI3	baldin_andrew	0.2518	2
REF	REF	0.2986	-
3823B1FI3	kichanova_ksenia	TEST FAILED	-

Tasks Done

3823B1FI3

Group	Name	Passed	Score
3823B1FI3	baldin_andrew	9/9	524
3823B1FI3	chacshin_vladimir	3/9	140
3823B1FI3	gutyansky_alexey	3/9	154
3823B1FI3	kichanova_ksenia	7/9	398
3823B1FI3	kurpiakov_aleksei	5/9	275
3823B1FI3	kurpiakov_alexei	2/9	105
3823B1FI3	kutergin_valentin	6/9	340
3823B1FI3	levonychev_ivan	8/9	449
3823B1FI3	lukin_ivan	4/9	211
3823B1FI3	papulina_yuliya	3/9	146
3823B1FI3	pylaeva_svetlana	3/9	140
3823B1FI3	romanov_artem	7/9	395
3823B1FI3	votincev_dmitri	7/9	394
3823B1FI3	zavyalov_alexey	5/9	262

Passed: 1

Total Passed: 1

---

Maximum Score: 576 (64 per task)