Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: release GIL during STRtree creation #395

Open
brendan-ward opened this issue Sep 23, 2021 · 1 comment
Open

ENH: release GIL during STRtree creation #395

brendan-ward opened this issue Sep 23, 2021 · 1 comment
Labels
enhancement New feature or request to-shapely This issue needs to be transferred to shapely

Comments

@brendan-ward
Copy link
Contributor

We aren't currently releasing the GIL during the creation of STRtree; this might be possible and could yield potential performance improvements in a multi-threaded environment.

In particular, the initial query against the tree after construction toward the end of STRtree_new should not need to hold the GIL.

@brendan-ward brendan-ward added the enhancement New feature or request label Sep 23, 2021
@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented Sep 23, 2021

In particular, the initial query against the tree after construction toward the end of STRtree_new should not need to hold the GIL.

Ah good point, it might be that the most expensive part is actually here? So releasing the GIL here might give most of the benefit.

The rest of the code in STRtree_New is inserting the geometries in the tree, and adding them to a vector, which needs an INCREF, which I think cannot be done without the GIL?

pygeos/src/strtree.c

Lines 151 to 200 in ea76ac9

GEOS_INIT;
tree = GEOSSTRtree_create_r(ctx, (size_t)node_capacity);
if (tree == NULL) {
errstate = PGERR_GEOS_EXCEPTION;
return NULL;
GEOS_FINISH;
}
n = PyArray_SIZE((PyArrayObject*)arr);
_geoms = (GeometryObject**)malloc(n * sizeof(GeometryObject*));
for (i = 0; i < n; i++) {
/* get the geometry */
ptr = PyArray_GETPTR1((PyArrayObject*)arr, i);
obj = *(GeometryObject**)ptr;
/* fail and cleanup incase obj was no geometry */
if (!get_geom(obj, &geom)) {
errstate = PGERR_NOT_A_GEOMETRY;
GEOSSTRtree_destroy_r(ctx, tree);
// free the geometries
for (i = 0; i < counter; i++) {
Py_XDECREF(_geoms[i]);
}
free(_geoms);
GEOS_FINISH;
return NULL;
}
/* If geometry is None or empty, do not add it to the tree or count.
* Set it as NULL for the internal geometries used for predicate tests.
*/
if (geom == NULL || GEOSisEmpty_r(ctx, geom)) {
_geoms[i] = NULL;
} else {
// NOTE: we must keep a reference to the GeometryObject added to the tree in order
// to avoid segfaults later. See: https://github.com/pygeos/pygeos/pull/100.
Py_INCREF(obj);
_geoms[i] = obj;
count_indexed++;
// Store the address of this geometry within _geoms array as the item data in the
// tree. This address is used to calculate the original index of the geometry in
// the input array.
// NOTE: the type of item data we store is GeometryObject**.
GEOSSTRtree_insert_r(ctx, tree, geom, &(_geoms[i]));
}
counter++;
}

@caspervdw caspervdw added the to-shapely This issue needs to be transferred to shapely label Dec 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request to-shapely This issue needs to be transferred to shapely
Projects
None yet
Development

No branches or pull requests

3 participants