Things to know on profiling and benchmarking

GeoUtils separates profiling, large data tests and benchmarks. Below is what each piece means and where it lives.

Profiler

The profiler is in geoutils/profiler.py, and aims to be reusable through the public API.

Its main helper, geoutils.profiler.profile_call(), runs one function and returns the result plus runtime and peak RAM. It uses psutil for the main Python process. If a distributed Dask client is active, it also uses Dask's MemorySampler to report worker RAM and spilled memory. This is the measurement layer reused by tests and benchmarks.

Dask benchmark workflows

A benchmark workflow is a reusable code path that we want to measure (CPU/RAM) depending on raster/pointcloud size.

The Dask workflows are defined in benchmarks/_dask_workflows.py. They create lazy rasters larger than worker memory, run package operations we want to test such as reproject() followed by mean(), and finish with a small result: a scalar, a sample, or a chunked write. They are shared by the heavy pytest tests and ASV.

Large data tests

The large data tests are pytest tests in benchmarks/test_large_data.py.

They run the Dask workflows on a constrained Dask LocalCluster and check that the result is correct and execution stays out-of-core. They are marked as large_data, skipped by default, and only run when pytest is called with -large-data. They can also be triggered manually through the benchmarks workflow in GitHub Actions.

ASV benchmarks

Airspeed Velocity (ASV) is a benchmarking tool used to record timings and memory metrics across commits.

The ASV configuration is in asv.conf.json, and the benchmark classes are in benchmarks/. They reuse the Dask workflows defined above and expose time_* methods for runtime and track_* methods for memory metrics. ASV can also compare implemented strategies, for example the GeoUtils polygonize strategies label_union, label_stitch and geometry_stitch.

CI

Normal PR CI runs the ordinary tests only. The separate GitHub Actions workflow .github/workflows/benchmarks.yml runs scheduled (currently every two weeks) ASV checks and manual ASV or large-data runs.

Publishing long-term ASV history is possible and practical. (not done yet when writing this)

Commands

For running ASV locally:

asv check -E existing
asv run --quick --show-stderr -E existing

For running the large dataset locally, possibly with tuned parameters through environment variables (first two lines optional):

GEOUTILS_DASK_LARGE_DATA_SHAPE=8192 \
GEOUTILS_DASK_LARGE_DATA_MEMORY_LIMIT=512MB \
pytest -large-data -m large_data -ra