Doc: flatgeobuf: document amount of RAM needed for packed Hilbert RTree building

[rouault]

Cf thread at https://lists.osgeo.org/pipermail/gdal-dev/2023-September/057727.html

[rouault]

CC @bjornharrtell Can you confirm or infirm my theory about the RAM requirement ? sizeof(NodeItem) == 40 clearly , and we need 2 instances per feature: one in the list provided to the PackedRTree constructor, and another one for the PackedRTree._nodeItems[] ?

[bjornharrtell]

Without diving deep I would say that seems correct yes. But I don't immediately understand/recall why I had to have this double representation.

[bjornharrtell]

Looked into it a bit and think it should be possible to avoid the duplication, but I'm not sure when I will find the time and brain power to do it so I think this docs change is correct for now.

[bjornharrtell]

@rouault actually "at least the number of features times 80 bytes" can be misleading, it'a bit more. The private _nodeItems also include nodes for all levels above the lowest.

[rouault]

The private _nodeItems also include nodes for all levels above the lowest.

any estimation of that amount? Perhaps some log2(number of features) * something involved ?

[bjornharrtell]

It depends on node size but in GDAL it's not configurable so it's set to 16. The full tree size can be calculated like this:

gdal/ogr/ogrsf_frmts/flatgeobuf/packedrtree.cpp

Lines 436 to 443 in b33e3b3

	uint64_t n = numItems;
	uint64_t numNodes = n;
	do
	{
	n = (n + nodeSizeMin - 1) / nodeSizeMin;
	numNodes += n;
	} while (n != 1);
	return numNodes * sizeof(NodeItem);

Where in the case of GDAL nodeSizeMin is 16. Subtract by numItems * 40 and that should be the size of the part of the tree that is above the feature level. There is probably some nice mathemetic way to formulate this but I'm not clever enough right now. :)

[rouault]

There is probably some nice mathemetic way to formulate this

So, assuming numNodes would be a power of 16 to simplify things (and if it is not, that should be really close), that's numNodes / 16 + numNodes / 16**2 + numNodes / 16**3 + ... until infinity (which comes much sooner than infinity). So a geometric series of coefficient 1./16 and first term numNodes/16, which converges to numNodes/16 / (1. - 1./16) = numNodes / 15 ~= 0.06667 * numNodes
So the total size required would be (2 + 0.06667) * 40 * numItems ~= 83 * numItems