Skip to content

edaphos 1.3.0 — honest benchmark on 1212 real Cerrado profiles

Choose a tag to compare

@HugoMachadoRodrigues HugoMachadoRodrigues released this 22 Apr 23:13
· 171 commits to main since this release

Summary

Closes the single biggest honesty gap in the README: the
"beyond regression-tree state-of-the-art" claim had never been
rigorously tested against a real dataset with a held-out test set.
This release provides that test on 1212 real Brazilian Cerrado
topsoil profiles
, and adjusts the README to reflect what the
numbers actually say.

The benchmark

Setting. On an 80 / 20 spatial train/test split (961 train, 240
test, stratified by 2×2 longitude × latitude quadrants so the test
set always contains points from every sub-region of the biome),
three competing stacks are evaluated with the same covariates,
seeds, split and metrics:

  • B1 ranger quantile regression forest on the raw 32-layer
    SoilGrids + WorldClim + SRTM covariate stack.
  • B2 B1 + gstat residual kriging — the canonical
    Hengl-style classical DSM recipe.
  • E B1 + the 64-dim MoCo v2 embedding from the publicly
    released edaphos-cerrado-moco-v1 encoder (Zenodo DOI
    10.5281/zenodo.19701276).

Results on the held-out test set:

method n RMSE (g/kg) MAE PICP @ 95 Interval score
B1 ranger — raw covariates 240 12.28 7.51 0.24 0.946 57.5
B2 ranger + gstat kriging 141* 11.51 7.18 0.09 0.738 104.1
E ranger + MoCo v2 embedding 240 12.53 8.28 0.21 0.858 85.5

* Kriging returned NA outside the fitted variogram's effective range;
the reduced n is itself a lesson.

Honest reading. Plain QRF is the calibration champion (PICP
0.946 at a 0.95 nominal level; best interval score). Residual
kriging lowers point RMSE but blows up calibration and drops data
far from the variogram range. The foundation-model embedding does
not beat B1 on this AoI
— the encoder was trained for only 20 k
InfoNCE steps on a smaller core-Cerrado AoI, and the raw covariate
stack is already rich enough that the embedding adds little marginal
signal over it. The Pillar 4 payoff is expected when the raw stack
is thinner (SAR-only or MODIS-only regions), which is the v1.4.0
agenda.

Data sources (all real, all open-licensed, all attributed)

  • Cerrado biome polygon — IBGE 1:250 000 Biomes via geobr
    (Pereira & Gonçalves 2019).
  • SOC observations — WoSIS snapshot 2019 (Batjes, Ribeiro and
    van Oostrum 2020, DOI
    10.5194/essd-12-299-2020,
    CC-BY-4.0), fetched live from
    ISRIC WFS.
  • Covariates — SoilGrids 250 m
    (Hengl et al. 2017),
    WorldClim 2.1
    (Fick & Hijmans 2017),
    SRTM 30-arcsec (Jarvis et al. 2008), all via geodata.

Deliverables

  • New vignette case-cerrado-end-to-end — narrative walk-through
    with per-dataset profile counts, a stratified-split map,
    observed-vs-predicted scatter (per method), PICP + interval-width
    bars, and residual geography. Every plot uses real data; every
    claim is sourced to a DOI / URL.
  • R/edaphos_metrics.R — standardised edaphos_rmse() /
    edaphos_mae() / edaphos_r2() / edaphos_bias() /
    edaphos_picp() / edaphos_interval_score() /
    edaphos_ece() / edaphos_metrics_summary() helpers. 21 new
    unit tests
    (all green on hand-computed fixtures).
  • data-raw/case_cerrado_prepare.R + _run.R — fully
    reproducible benchmark pipeline (~1 h end-to-end, ~2 GB
    downloads).
  • inst/extdata/case_cerrado_results.rds — the pre-computed
    benchmark results shipped with the package so the vignette
    builds on any installation without running the heavy prep.
  • README Benchmarks section at the top of the file, with the
    honest table and the "what we learned" paragraph.

New Suggests

geobr (IBGE biome polygon), dplyr and patchwork (vignette
plotting stack).

Quality

  • 21 new tests for edaphos_metrics (hand-computed fixtures).
  • Full test suite passes on all pillars.
  • R CMD check --as-cran: 0 errors / 0 warnings / 1 harmless
    NOTE
    (torch tempdir).