Skip to content

Feature/spatial splits#61

Merged
vdplasthijs merged 5 commits into
developfrom
feature/spatial-splits
Mar 12, 2026
Merged

Feature/spatial splits#61
vdplasthijs merged 5 commits into
developfrom
feature/spatial-splits

Conversation

@robknapen

Copy link
Copy Markdown
Collaborator

Spatial-cluster split strategy for crop yield experiments

  • Adds a spatial-cluster split script and Hydra configs so that yield_africa experiments can be evaluated on geographically unseen locations, alongside the existing random and LOCO split strategies
  • Speeds up DBSCAN clustering in BaseDataModule by replacing a slow Python geodesic lambda with sklearn's built-in haversine metric and BallTree index

A random 70/15/15 split for yield_africa allows nearby fields, and the same field across different years, to appear in both train and test. Because agricultural yield is strongly spatially autocorrelated (shared soils, rainfall patterns, agro-ecological zones), this inflates apparent generalisation performance. The spatial-cluster split closes this loophole by ensuring that no two geographically close locations straddle a split boundary.

Comment thread src/data/base_datamodule.py
Comment thread tests/test_yield_africa.py

@vdplasthijs vdplasthijs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good, great thank you! Cool implementation of DBScan!!

@vdplasthijs vdplasthijs merged commit 7a82a18 into develop Mar 12, 2026
4 checks passed
@robknapen robknapen deleted the feature/spatial-splits branch March 12, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants