Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel qsort without OpenMP #7689

Closed
Gumix opened this issue Sep 15, 2022 · 5 comments · Fixed by #8562
Closed

Parallel qsort without OpenMP #7689

Gumix opened this issue Sep 15, 2022 · 5 comments · Fixed by #8562
Labels
3.0 Target is 3.0 and all newer release/master branches refactoring Code refactoring

Comments

@Gumix
Copy link
Contributor

Gumix commented Sep 15, 2022

Currently OpenMP is used by Tarantool only for parallel qsort:

tarantool/third_party/qsort_arg_mt.c

Lines 218 to 219 in c294a92

#pragma omp task
qsort_arg_mt_internal(a, d1 / es, es, cmp, arg);

However libgomp.a in some Linux distros (e.g. CentOS) doesn't work with position-independent code, which is required for security.

/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/libgomp.a(parallel.o): relocation R_X86_64_32 against hidden symbol `gomp_global_icv' can not be used when making a PIE object
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/libgomp.a(task.o): relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIC

#7587 (comment)

Let's get rid of OpenMP by rewriting qsort to pthread or something like that.
@Gumix Gumix added the teamC label Sep 15, 2022
@CuriousGeorgiy
Copy link
Member

How about using std::sort and setting a parallel execution policy?

@kyukhin kyukhin added refactoring Code refactoring and removed teamC labels Sep 16, 2022
@CuriousGeorgiy
Copy link
Member

CuriousGeorgiy commented Oct 2, 2022 •
edited

Good point of reference on parallel sorting algorithms: https://stackoverflow.com/a/3969847

@Gumix Gumix mentioned this issue Oct 12, 2022
Merged
@Gumix
Copy link
Contributor Author

Gumix commented Dec 27, 2022

How about using std::sort and setting a parallel execution policy?

It requires at least GCC 10 (or GCC 9 + TBB), while we still do support GCC 4.8, but most importantly, parallel STL is actually implemented in GCC via OpenMP.

@Gumix Gumix mentioned this issue Apr 2, 2023
Closed
@nshy nshy mentioned this issue Apr 12, 2023
Merged
nshy added a commit to nshy/tarantool that referenced this issue Jun 8, 2023
@nshy
box: introduce memtx_sort_threads config parameter
11a963d 
Closes tarantool#3389
Closes tarantool#7689
Closes tarantool#4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```
nshy added a commit to nshy/tarantool that referenced this issue Jun 8, 2023
@nshy
box: introduce memtx_sort_threads config parameter
f3c5427 
Closes tarantool#3389
Closes tarantool#7689
Closes tarantool#4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```
nshy added a commit to nshy/tarantool that referenced this issue Jun 9, 2023
@nshy
box: introduce memtx_sort_threads config parameter
f83a2cc 
Closes tarantool#3389
Closes tarantool#7689
Closes tarantool#4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```
nshy added a commit to nshy/tarantool that referenced this issue Jun 9, 2023
@nshy
box: introduce memtx_sort_threads config parameter
f3ed1ad 
Closes tarantool#3389
Closes tarantool#7689
Closes tarantool#4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```
nshy added a commit to nshy/tarantool that referenced this issue Jun 9, 2023
@nshy
box: introduce memtx_sort_threads config parameter
e8c81ef 
Closes tarantool#3389
Closes tarantool#7689
Closes tarantool#4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```
nshy added a commit to nshy/tarantool that referenced this issue Jun 9, 2023
@nshy
box: introduce memtx_sort_threads config parameter
e02039e 
Closes tarantool#3389
Closes tarantool#7689
Closes tarantool#4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```
alyapunov pushed a commit that referenced this issue Jun 9, 2023
@nshy @alyapunov
box: introduce memtx_sort_threads config parameter
4f617b7 
Closes #3389
Closes #7689
Closes #4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```
@CuriousGeorgiy
Copy link
Member

CuriousGeorgiy commented Jun 10, 2023 •
edited

@nshy you forgot to remove this workaround:

tarantool/cmake/hardening.cmake

Lines 5 to 6 in de0731b

# ligomp.a for AArch64 CentOS is compiled without PIC support.
if (ENABLE_FUZZER OR TARGET_OS_FREEBSD OR ${CMAKE_SYSTEM_PROCESSOR} MATCHES "aarch64")

Gumix added a commit to Gumix/tarantool that referenced this issue Jun 16, 2023
@Gumix
build: remove dependencies on libgomp
6a0be5c 
OpenMP is no longer used since commit 4f617b7 ("box: introduce
memtx_sort_threads config parameter"). All dependencies on libgomp
should be removed.

Follow-up tarantool#7689

NO_DOC=build
NO_TEST=build
@nshy
Copy link
Contributor

nshy commented Jun 19, 2023

Thanks for the care! The issue is addressed in PR #8784

locker pushed a commit that referenced this issue Jun 20, 2023
@Gumix @locker
build: remove dependencies on libgomp
bb7c162 
OpenMP is no longer used since commit 4f617b7 ("box: introduce
memtx_sort_threads config parameter"). All dependencies on libgomp
should be removed.

Follow-up #7689

NO_DOC=build
NO_TEST=build
@locker locker added the 3.0 Target is 3.0 and all newer release/master branches label Jun 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.0 Target is 3.0 and all newer release/master branches refactoring Code refactoring
Projects
None yet
Milestone
No milestone
5 participants
@locker @nshy @Gumix @kyukhin @CuriousGeorgiy