Include Python object size in .nbytes attribute

[jorisvandenbossche]

To get a slightly better estimate (but still cheap) of the memory usage (the np.ndarray.nbytes for an object dtype array only counts the array of pointers, and not the size of objects it is pointing to).

The size of the Python geometry object is fixed (for pygeos at least, 32 bytes on Python 3.9), so also does not yet include an estimate of the size of the coordinate sequences.

In principle we could add a count_coordinates(self.data) * 8 to get a better estimate, but that of course incurs some computation (although a relatively cheap one).

[martinfleis]

In principle we could add a count_coordinates(self.data) * 8 to get a better estimate, but that of course incurs some computation (although a relatively cheap one).

Do you have an idea how often is this called internally? I assume that Dask uses it, in which case having more reasonable estimate might resolve some of our memory issues in dask-geopandas.

I am sort of not super happy about this solution as it is better but still way off (unless you have just point geometry), so if we think that the slowdown caused by calling count_coordinates is not terrible, I'd be up for it (though we should probably use it with pygeos only).

[jorisvandenbossche]

I think getting the number of coordinates is generally pretty cheap.

For example for the US zip codes:

In [1]: import pyogrio

In [2]: data = pyogrio.read_dataframe("benchmark-data/tl_2019_us_zcta510/tl_2019_us_zcta510.shp")

In [3]: import pygeos

In [4]: pygeos.count_coordinates(data.geometry.array.data)
Out[4]: 53011221

In [5]: %timeit pygeos.count_coordinates(data.geometry.array.data)
3.8 ms ± 64.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [6]: len(data)
Out[6]: 33144

Although that's a dataset with relatively small number of rows with larger polygons each.

With some random data:

In [8]: polys = pygeos.polygons(np.random.randn(10_000_000, 3, 2))

In [9]: %timeit pygeos.count_coordinates(polys)
264 ms ± 6.81 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The question is maybe how often dask calls this. If it doesn't do this repeatedly, the additional time is probably not a worry.

[jorisvandenbossche]

Given that this is a strict improvement, although certainly not yet ideal, we can maybe merge this already, and leave counting the coordinates for a future issue/PR?

[martinfleis]

Let's give it a go.

[jorisvandenbossche]

We could actually also add a memory_usage() method to GeometryArray, which accepts a deep parameter. For pandas users, that can then be a way to get a correct idea of the memory usage of a GeoDataFrame/GeoSeries.

However, that wouldn't help for dask. But, dask-geopandas could register a function to measure the size of a GeoDataFrame/GeoSeries (@sizeof.register). There, we could also be less precise (eg calculate it for a sample, and extrapolate, which is what dask does for pandas object dtype, eg for strings)

[m-richards]

Not very well across the dask/ dask-geopandas context, but from a pure geopandas/ python perspective, PR looks fine to me

Edit: unless there is also a readme message to add.