Multithreading

[christophlingg]

Thanks for providing RTree for Python!

I was wondering if rtree can be used in a multithreaded context?

[songololo]

+1

[marfago]

Actually seems it is not possible. At least with recent versions.

libspatialindex/libspatialindex#71

[hobu]

libspatialindex is not threadsafe and Rtree does not do thread safety. It is something that could be done, but it would take a lot of meticulous work to achieve.

[christophlingg]

Ok I can comprehend that concurrent writing and reading may cause problems, but are read-only accesses safe?

[marfago]

In my experience it seems it is not.
In a project I use https://github.com/collective/collective.geo.index, wich is based on RTree. For each search it instantiates a new instance of RTree and now and then it crashes with segmentation fault. I solved this issue moving the rtree to a global variable (just one instance). Not sure it can help.

[christophlingg]

That's strange because I am accessing one rtree from multiple threads (only read), and so far i did not encounter any problems.

[snorfalorpagus]

@christophlingg Race conditions can be like that. It may be that certain issues are only triggered under high(er) loads.

[hobu]

Ok I can comprehend that concurrent writing and reading may cause problems, but are read-only accesses safe?

No. libspatialindex, especially as used through the C API like Rtree does, is not thread safe. If it is working for you, it is most likely due to luck.

[christophlingg]

Now I know I need to fix something ;-) Thanks to your feedback I can do this before it crashes!

[hobu]

The most pressing issue is probably the one described in libspatialindex/libspatialindex#71 especially in regards to the C API.

If you put some aggressive locking around things in python land, you are likely to be reasonably protected. The issues most likely manifest themselves in regard to error messages, and the error stack of the C API of libspatialindex not being protected in any way.

[marfago]

I'm working with a single thread instance of the index, but it daily crashes!
Can anyone suggest what exactly should be "protected" in python?

[hobu]

Can anyone suggest what exactly should be "protected" in python?

Protect any access you do to libspatialindex with a mutex and lock accordingly.

[marfago]

@hobu :Adding mutex or lock did not work for me. Moreover, also using different processes working on the same index file ends up in segmentation fault.

[hobu]

libspatialindex has a thread safety issue. Someone needs to develop a clear case that demonstrates the issue so it can be resolved.