Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault when passing np.array pointer to OpenMP code compiled with latest g++ #3135

Open
david-cortes opened this issue Sep 7, 2019 · 0 comments

Comments

@david-cortes
Copy link

commented Sep 7, 2019

Compiling some code with g++ 9.2.1 will oftentimes segfault when there are openmp segments, sometimes throwing a message about double-free. Problem doesn't happen with earlier versions of g++, nor with clang or msvc.

Example code that oftentimes segfaults:

  • cppfile.cpp:
#include <vector>
#include <stddef.h>
#include <omp.h>
#include <unordered_set>

void check_more_two_values(double arr_num[], size_t nrows, size_t ncols, int nthreads, char too_few_values[])
{
    std::vector<std::unordered_set<double>> seen_values(ncols);

    #pragma omp parallel for schedule(dynamic) num_threads(std::min(nthreads, (int)ncols)) shared(arr_num, nrows, ncols, too_few_values, seen_values)
    for (size_t col = 0; col < ncols; col++) {
        for (size_t row = 0; row < nrows; row++) {
            if (!isnan(arr_num[row + col * nrows]))
                seen_values[col].insert(arr_num[row + col * nrows]);
            if (seen_values[col].size() > 2) break;
        }
        if (seen_values[col].size() <= 2) too_few_values[col] = true;
    }
}
  • cyfile.pyx:
import numpy as np
cimport numpy as np
import ctypes

cdef extern from "cppfile.cpp":
	void check_more_two_values(double *arr_num, size_t nrows, size_t ncols, int nthreads, char *too_few_values)

def call_check_more_two_values(np.ndarray[double, ndim = 2] arr, int nthreads):
	cdef size_t nrows = arr.shape[0]
	cdef size_t ncols = arr.shape[1]
	cdef np.ndarray[char, ndim = 1] outp = np.zeros(ncols, dtype = ctypes.c_char)

	if not arr.flags['F_CONTIGUOUS']:
		arr = np.asfortranarray(arr)
	
	check_more_two_values(&arr[0, 0], nrows, ncols, nthreads, &outp[0])
	return outp
  • setup.py:
try:
    from setuptools import setup
    from setuptools.extension import Extension
except:
    from distutils.core import setup
    from distutils.extension import Extension

import numpy as np
from Cython.Distutils import build_ext


setup(
    name  = "cyext",
    packages = ["cyext"],
    version = '1.0.6',
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension(
                                "cyext",
                                sources=["cyfile.pyx"],
                                include_dirs=[np.get_include()],
                                language="c++",
                                install_requires = ["numpy", "pandas>=0.24.0", "cython"],
                                extra_compile_args = ["-std=c++11", "-O2", "-march=native", "-fopenmp"],
                                extra_link_args = ["-fopenmp"]
                            )]
    )
  • pyfile.py:
import numpy as np, pandas as pd
import cyext

nrows = 2000
ncols = 9
nthreads = 9
np.random.seed(100)
arr = pd.DataFrame({
	"col1" : np.random.gamma(1, 1, size = nrows),
	"col2" : np.random.random(size = nrows),
	"col3" : np.random.normal(size = nrows),
	"col4" : np.random.beta(1, 1, size = nrows),
	"col5" : np.zeros(nrows),
	"col6" : np.ones(nrows),
	"col7" : np.random.randint(5, size = nrows),
	"col8" : np.random.randint(2, size = nrows),
	"col9" : np.random.random(size = nrows) ** 2
	}).astype("float64")

has_few_vals = cyext.call_check_more_two_values(arr.to_numpy(), nthreads)
print(has_few_vals)

arr *= 2
has_few_vals = cyext.call_check_more_two_values(arr.to_numpy(), nthreads)
print(has_few_vals)

Compile with python setup.py build_ext --inplace --force, then execute python pyfile.py. Removing -fopenmp from setup.py will not result in a segfault.

Setup: Cython 0.29.13, Python 3.7.3, running on debian bullseye (testing), kernel 5.2, CPU Ryzen 2700.

Attached is the whole older with cython-generated .cpp file from cyfile.pyx and the shared object.
cy_test.tar.gz

@david-cortes david-cortes changed the title Segfault when interfacing OpenMP C++ code Segfault when interfacing OpenMP C++ code compiled with latest g++ Sep 7, 2019

@david-cortes david-cortes changed the title Segfault when interfacing OpenMP C++ code compiled with latest g++ Segfault when passing np.array pointer to OpenMP code compiled with latest g++ Sep 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.