OpenMP support in AST and ASR

[certik]

Steps:

• Let's create a set of representative tests that fully compile with gfortran + openmp
• Parse them to AST (probably we need to add new AST nodes for the openmp features) Initial OpenMP Pragma support #3864
• In ASR we should use DoConcurrent (our parallel loop in ASR) and possibly extend it with new fields as needed
• We need to add pass similar to the current Symbolics pass that translates DoConcurrent into specific OpenMP API calls feat: complete openmp ASR to ASR pass #4185

Let's do a depth-first approach, the simplest example we can take is probably https://github.com/certik/hfsolver/blob/b4c50c1979fb7e468b1852b144ba756f5a51788d/src/sparse.f90#L200, let's write a standalone test for it, and try to compile it. We can even simplify it further if needed.

[certik]

Let's start with using do concurrent and use shared and local specifiers. Then let's write the OpenMP backend to paralellize this. To get started, let's write C code that uses OpenMP directly and ensure it works. Then let's write an ASR->ASR pass that takes DoConcurrent and replaces it with direct calls to OpenMP C/API.

[Pranavchiku]

Here is a useful doc that contains examples with increasing complexity in all C / C++ / Fortran ( fixed and free form ) languages: https://www.openmp.org/wp-content/uploads/openmp-examples-4.5.0.pdf

OpenMp-C: https://github.com/ResearchComputing/Documentation/blob/main/docs/programming/OpenMP-C.md
OpenMp-Fortran: https://github.com/ResearchComputing/Documentation/blob/main/docs/programming/OpenMP-Fortran.md

Quick reference card: https://www.openmp.org/wp-content/uploads/OpenMP3.1-CCard.pdf
Quick Reference card fortran: https://www.openmp.org/wp-content/uploads/OpenMP3.1-FortranCard.pdf

[Pranavchiku]

I did spend time and now able to run openMp for c++ and fortran on my mac. I'll post down the instructions to setup and work.

Setup Instructions

brew install llvm libomp
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"

and then set number of threads

export OMP_NUM_THREADS=4

Note: I was unable to find suitable way of installing llvm and libomp using conda / mamba hence resorted to homebrew.

C++

Here is a simple code to initialise array, perform operation and calculate sum:

cpp

#include <stdio.h>
#include <omp.h>

void initialize_array( int n, float *a, float val ) {
	int i;
	#pragma omp parallel
	{
		#pragma omp for
		{
			for (i = 0; i < n; i++) {
				a[i] = val;
			}
		}
	}
}

void simple( int n, float *a, float *b ) {
	int i;
	#pragma omp parallel
	{
		#pragma omp for
		{
			for (i = 1; i < n; i++) { /* i is private by default */
				b[i] = ( a[i] + a[i-1]) / 2.0;
			}
		}
	}

}

void parallel_Sum( int n, float *a, float *b ) {
	int i;
	float partial_sum, total_sum;

	#pragma omp parallel private(partial_sum) shared(total_sum)
	{
		partial_sum = 0;
		total_sum = 0;

		#pragma omp for
		{
			for (i = 0; i < n; i++) {
				partial_sum += a[i];
			}
		}

		#pragma critical
		{
			total_sum += partial_sum;
		}
	}

	printf("Total sum: %f\n", total_sum);
}

int main() {
	float a[1000000];
	float b[1000000];

	initialize_array(1000000, a, 1.0);
	initialize_array(1000000, b, -1.0);

	simple(1000000, a, b);

	parallel_Sum(1000000, a, b);
}

% time clang++ -fopenmp e.cpp && ./a.out
clang++ -fopenmp e.cpp  0.07s user 0.08s system 37% cpu 0.420 total
Total sum: 1000000.000000

Fortran

Corresponding fortran code

Fortran

subroutine simple(n, a, b)
    integer :: i, n
    real :: a(n), b(n)

!$omp parallel private(i)
    !$omp do
    do i = 2, n
        b(i) = (a(i) + a(i-1)) / 2.0
    end do
    !$omp end do

!$omp end parallel
end subroutine

subroutine parallel_sum(n, a)
integer :: i, n
real :: a(n), partial_sum, total_sum

!$omp parallel private(partial_sum) shared(total_sum)
    partial_sum = 0
    total_sum = 0

    !$omp do
        do i = 1, n
            partial_sum = partial_sum + a(i)
        end do
    !$omp end do

    !$omp critical
        total_sum = total_sum + partial_sum
    !$omp end critical
!$omp end parallel

print *, total_sum
if (abs(total_sum - 999998.0) > 1e-10) error stop

end subroutine

program main
integer :: n
real :: a(1000000), b(1000000)

n = 1000000
a = 1.0
b = -1.0

call simple(n, a, b)

call parallel_sum(n, b)
end program

% time gfortran d.f90 -fopenmp && ./a.out
gfortran d.f90 -fopenmp  0.06s user 0.11s system 70% cpu 0.244 total
   999998.000    

I'll write the same code using do concurrent and get on working forward. This is pretty exciting!

[Pranavchiku]

I am trying to write the following fortran code using do concurrent loop and not exactly sure about how can we represent private and shared variable in our grammar

subroutine simple(n, a, b)
integer :: i, n
real :: a(n), b(n)

!$omp parallel private(i)
!$omp do
do i = 2, n
    b(i) = (a(i) + a(i-1)) / 2.0
end do
!$omp end do

!$omp end parallel

end subroutine

do concurrent has the following locality-spec, initially I thought we'll fine with local_init representing private ( openmp ) and shared representing shared ( openmp ). On following discussion at https://fortran-lang.discourse.group/t/clarification-on-do-concurrent/1647/27, I suspect we can't assume it this way.

is LOCAL ( variable-name-list )
or LOCAL_INIT ( variable-name-list )
or SHARED ( variable-name-list )
or DEFAULT ( NONE )

cc @certik

For now, we can move ahead with the following do concurrent loop I think

do concurrent (i = 2:n)
    b(i) = (a(i) + a(i-1)) / 2.0
end do

[Pranavchiku]

I think we can have list of expressions for shared and local representing shared and private respectively. That way we can identify those variables in ASR->ASR pass. If this design is fine, let me know, I'll open PR with required changes, till then I am exploring how to: replace it with direct calls to OpenMP C/API.

diff --git a/src/libasr/ASR.asdl b/src/libasr/ASR.asdl
index f61430d06..c2aaa493b 100644
--- a/src/libasr/ASR.asdl
+++ b/src/libasr/ASR.asdl
@@ -35,7 +35,7 @@ stmt
     | Cycle(identifier? stmt_name)
     | ExplicitDeallocate(expr* vars)
     | ImplicitDeallocate(expr* vars)
-    | DoConcurrentLoop(do_loop_head head, stmt* body)
+    | DoConcurrentLoop(do_loop_head head, expr* shared, expr* local, stmt* body)
     | DoLoop(identifier? name, do_loop_head head, stmt* body, stmt* orelse)

[Thirumalai-Shaktivel]

Yup, I think that's the way to go.
For ASR pass, we have to implement something similar to symbolics pass.
If there is any other doubts, do let me know.

[Thirumalai-Shaktivel]

@Pranavchiku Try working on the DoConcurrentLoop. I'm working on recognising !$omp parallel.. in the ASR. I will open up a new PR soon.

[Pranavchiku]

Perfect, yes. This will be a good way forward.

[Thirumalai-Shaktivel]

OpenMP Fortran syntax:

• https://www.openmp.org/wp-content/uploads/OpenMP3.0-FortranCard.pdf
• https://www.openmp.org/wp-content/uploads/OpenMP-4.5-1115-F-web.pdf

[Pranavchiku]

I'll wrap #3859 and then will start on this.