Pickling and unpickling messes up Index

[bobtrekdrop]

Pickling and unpickling renders the Index unusable. The test_pickle.py does not test the correct properties, should check Index.bounds as well. Code to reproduce:

import cPickle as pickle
from rtree import index
import numpy as np

lonR = [-2.,9.]
latR = [50.,54.]
lonD = 0.125
latD = 0.125
lon = np.arange(lonR[0],lonR[1],lonD)
lat = np.arange(latR[0],latR[1],latD)
	
rtree = index.Index()
i = 0
for y in lat:
	for x in lon:
		rtree.insert(i, (x,y,x+lonD,y+latD))
		i += 1

unpickled = pickle.loads(pickle.dumps(rtree))
print(rtree.get_bounds())
print(unpickled.get_bounds())

[ghost]

Would really love to see this fix/implemented. I also notice that specifying an index name doesn't lead to the index to be stored at a particular file path? It would be useful to be able to find the serialised data somehow.

I've also tried pickling with dill and it gives the same issue.

Wish I could help but I couldn't understand the serialisation code at all.

[cgmike]

I am having exactly the same problem here on Ubuntu 18.04 with Python 3.6.
It would be extremely helpful if this gets fixed.

[SergeBouchut]

Is this issue is planned to be solved? It could be super useful to keep the hand on the "right time" and the conditions under which the index has to serialized.

[adamjstewart]

We just ran into this as well. TorchGeo uses rtree to store a database of raster file bounding boxes. However, if you try to use a parallel data loader, the multiprocessing library will pickle and unpickle the index, and all entries in the index will be removed.

I tried digging into this but couldn't get any further than anyone else. I guess it would be useful to understand why we need to delete the state handle instead of just pickling it. It seems that if I remove the del state["handle"] line in Index.__getstate__ it causes a segfault, but I don't know why that is.

[adamjstewart]

Update: I found a hack to make this work: #197. I don't like it, but it's better than nothing.