edaphos 1.3.0 — honest benchmark on 1212 real Cerrado profiles
Summary
Closes the single biggest honesty gap in the README: the
"beyond regression-tree state-of-the-art" claim had never been
rigorously tested against a real dataset with a held-out test set.
This release provides that test on 1212 real Brazilian Cerrado
topsoil profiles, and adjusts the README to reflect what the
numbers actually say.
The benchmark
Setting. On an 80 / 20 spatial train/test split (961 train, 240
test, stratified by 2×2 longitude × latitude quadrants so the test
set always contains points from every sub-region of the biome),
three competing stacks are evaluated with the same covariates,
seeds, split and metrics:
- B1
rangerquantile regression forest on the raw 32-layer
SoilGrids + WorldClim + SRTM covariate stack. - B2 B1 +
gstatresidual kriging — the canonical
Hengl-style classical DSM recipe. - E B1 + the 64-dim MoCo v2 embedding from the publicly
releasededaphos-cerrado-moco-v1encoder (Zenodo DOI
10.5281/zenodo.19701276).
Results on the held-out test set:
| method | n | RMSE (g/kg) | MAE | R² | PICP @ 95 | Interval score |
|---|---|---|---|---|---|---|
B1 ranger — raw covariates |
240 | 12.28 | 7.51 | 0.24 | 0.946 | 57.5 |
B2 ranger + gstat kriging |
141* | 11.51 | 7.18 | 0.09 | 0.738 | 104.1 |
E ranger + MoCo v2 embedding |
240 | 12.53 | 8.28 | 0.21 | 0.858 | 85.5 |
* Kriging returned NA outside the fitted variogram's effective range;
the reduced n is itself a lesson.
Honest reading. Plain QRF is the calibration champion (PICP
0.946 at a 0.95 nominal level; best interval score). Residual
kriging lowers point RMSE but blows up calibration and drops data
far from the variogram range. The foundation-model embedding does
not beat B1 on this AoI — the encoder was trained for only 20 k
InfoNCE steps on a smaller core-Cerrado AoI, and the raw covariate
stack is already rich enough that the embedding adds little marginal
signal over it. The Pillar 4 payoff is expected when the raw stack
is thinner (SAR-only or MODIS-only regions), which is the v1.4.0
agenda.
Data sources (all real, all open-licensed, all attributed)
- Cerrado biome polygon — IBGE 1:250 000 Biomes via
geobr
(Pereira & Gonçalves 2019). - SOC observations — WoSIS snapshot 2019 (Batjes, Ribeiro and
van Oostrum 2020, DOI
10.5194/essd-12-299-2020,
CC-BY-4.0), fetched live from
ISRIC WFS. - Covariates — SoilGrids 250 m
(Hengl et al. 2017),
WorldClim 2.1
(Fick & Hijmans 2017),
SRTM 30-arcsec (Jarvis et al. 2008), all viageodata.
Deliverables
- New vignette
case-cerrado-end-to-end— narrative walk-through
with per-dataset profile counts, a stratified-split map,
observed-vs-predicted scatter (per method), PICP + interval-width
bars, and residual geography. Every plot uses real data; every
claim is sourced to a DOI / URL. R/edaphos_metrics.R— standardisededaphos_rmse()/
edaphos_mae()/edaphos_r2()/edaphos_bias()/
edaphos_picp()/edaphos_interval_score()/
edaphos_ece()/edaphos_metrics_summary()helpers. 21 new
unit tests (all green on hand-computed fixtures).data-raw/case_cerrado_prepare.R+_run.R— fully
reproducible benchmark pipeline (~1 h end-to-end, ~2 GB
downloads).inst/extdata/case_cerrado_results.rds— the pre-computed
benchmark results shipped with the package so the vignette
builds on any installation without running the heavy prep.- README Benchmarks section at the top of the file, with the
honest table and the "what we learned" paragraph.
New Suggests
geobr (IBGE biome polygon), dplyr and patchwork (vignette
plotting stack).
Quality
- 21 new tests for
edaphos_metrics(hand-computed fixtures). - Full test suite passes on all pillars.
R CMD check --as-cran: 0 errors / 0 warnings / 1 harmless
NOTE (torch tempdir).