Annual IO LULC series for bulk, necr, lnth, kotl (#79) - #82
Merged
NewGraphEnvironment merged 5 commits intoSep 5, 2026
Conversation
Annual IO LULC series for bulk, necr, lnth, kotl. Scope corrected during planning: PINE dropped (pre-flooded-0.5.0 AOI, tracked by #76), the proposed A/B gate replaced with a provenance_ab-compare expected-failure set, and the run split across m1 and m4. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZBWEFxANHj7gRrmQYcENt
`03_lulc_classify.R` built `years` as the change-interval endpoints plus the midpoint, so every area published 2017/2020/2023 only. drift's `dft_rast_break_class()` needs the full annual series to separate a sustained land-cover switch from flicker. `lulc_annual: true` in `area.yml` makes `years` the whole interval. The transition is untouched either way -- it reads `yrs` (change_interval), never `years` -- so turning the flag on adds classified years and moves no transition. The `available_years` guard already covered a seven-element set unedited. Turned on for bulk, necr, lnth, kotl. PINE is deliberately NOT in that list: its `data/` predates flooded 0.5.0, so step 3 there would classify land cover over a floodplain the repo has already declared dead rather than superseded (#76). Two guards, because the failure is silent in both places: - the config value must be a single unquoted logical. yaml gives TRUE for true/yes/on, but a QUOTED "true" is a character vector that `isTRUE()` reads as off, so the area would run three years under a config that reads as annual. - `FP_LULC_ANNUAL` takes a CLOSED vocabulary rather than a truthiness test. `%in% c("1","TRUE",...)` would read a typo as FALSE, which is the same silent-off failure one layer out. `lulc_annual` is area-owned, not region-owned: the five annual areas span three regions, so a region file must never set or clear it. `region_config-check.R` now asserts it survives a region run, and the assertion was proven to fire -- adding `lulc_annual` to `FP_REGION_OWNED` turns it PASS -> FAIL. Nothing is recorded in provenance for the flag itself. `inputs$years` already carries the resolved set and the guard already asserts it matches the digest year set; a boolean beside the vector it produced is a second source of truth. README prose drops the literal year list rather than restating a new one -- a count in prose is what #77 exists to stop, and this is now per-area. Rendered; determinism check green on all three properties. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZBWEFxANHj7gRrmQYcENt
Step 3 re-run with `lulc_annual: true` for bulk, necr, lnth and kotl. Steps 1-2 untouched, so every area's floodplain geometry and sub-basins are the bytes step 2 last wrote. All four pass the full acceptance set: `years` = 2017..2023, seven per-year digests, the 2017/2020/2023 digests unchanged element-wise, `transition_content_sha256` and `transition_patches` unchanged, `outputs_hash` unchanged, `inputs_hash` moved, exactly seven `classified_*` layers and tifs with no eighth, `provenance-check.R` green, `bridge-check.R` green where `attribute_by` is set. bulk's 7,161 change patches match the figure CLAUDE.md records for its 2026-09-02 run. The acceptance is greppable rather than eyeballed, because `provenance_ab-compare.R` reports a differing `inputs_hash` as a FAILURE and under this change it must differ. The expected failure set is written down in the evidence log: landcover inputs DIFFER / outputs same, and network and floodplain datetimes SAME because steps 1-2 did not re-run. Splitting the run needed a control, since anything differing between the two machines is a confound. m4 was levelled to m1 on drift (0.8.0 -> 0.13.0; >= 0.10.0 is not optional, 0.8.0 fetches a single STAC page and would have truncated KOTL's item set into a wrong raster with no error), sf and gdalcubes. terra could not be matched, which left it the only variable, and neexdzii on m4 reproduced m1's `outputs_hash`, transition digest, patch count and all three per-year digests across terra 1.9.11 vs 1.9.34 AND drift 0.8.0 vs 0.13.0. necr and kotl then did the same on real published areas. Every year was genuinely re-fetched: all four baselines were built under drift 0.8.0, whose cache keys predate the 0.10.0 change, so nothing was cache-served and the shared-year assertion is not vacuous. Peak RSS does not track grid size -- KOTL at 203 Mcells peaked at 54.3 GB and BULK at 168 Mcells at 20.6 GB, on different hosts. Recorded with that caveat rather than as a per-area requirement, because it was not isolated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZBWEFxANHj7gRrmQYcENt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZBWEFxANHj7gRrmQYcENt
…ector (#79) Seven findings from an adversarial review of the branch diff. Four are fixed here, three were verified clean or left alone deliberately. An EMPTY `lulc_annual` silently ran three years. `lulc_annual:`, `~` and `null` all parse to NULL, so the `!is.null()` short-circuit skipped the type guard entirely and `isTRUE(NULL)` is FALSE -- the exact silent-off class the guard exists to close, reached by blanking a value or commenting out the `true`. Measured on all three spellings. Now keyed on `%in% names(cfg)`, which is TRUE for a present-empty key while `lulc_annual: false` stays legal. The drift floor was two minor versions low, and this branch is what documented it. `fp_lulc` asserted `>= 0.6.0` (where `tile_size` arrived) while the branch's own evidence log states that before 0.10.0 the fetch issues a single `get_request()` with no paging, so an AOI spanning more than one page is built from a partial item set -- a wrong raster, silently, with `item_ids_complete` structurally unable to report it (#81). That is a correctness floor, not a feature floor, and KOTL is one of the four areas this branch turned annual. The one-way door had no detector. `provenance-check.R` only ever compared `classified_content_sha256`'s year set against `inputs$years` -- two values the same run wrote, which cannot disagree. Nothing read `rasters/<scen>/` or the gpkg layer list, so an area reverted to three years recorded three, passed green, and kept four orphan `classified_*` layers and four orphan tifs (#55's class, which `gpkg_prune-legacy.R`'s transition-only pattern does not sweep). 7c now reconciles both artefact year sets against `inputs$years`. Proven to fire: mutating necr's provenance to three years turns both arms red naming the disk and recorded sets, and the restore leaves it green. The env vocabulary was narrower than the config's -- `lulc_annual: y` was legal in YAML but `FP_LULC_ANNUAL=y` was refused, and `T`/`F` too. They now agree, and the value is trimmed so a trailing newline from a shell capture is not a typo. Also noted that `FP_LULC_ANNUAL` is inherited by `run_region.R`'s children, so setting it for a region run flips every not-yet-cached group with no trace in any `area.yml` -- which interacts badly with the one-way door. CLAUDE.md gained the `lulc_annual` bullet its sibling keys all have, the one-way-door and RSS caveats, the corrected drift floor, and a note on bulk's entry that it is annual since 2026-09-05 with its recorded numbers unmoved. Corrected a number I got wrong: the evidence log generalized neexdzii's 14-item STAC response to all four areas. Measured, necr and kotl each record 7 item ids, one per year, on one-tile AOIs. Fixed here and in stac#59. Also flagged that `lulc_summary.rds` now carries seven years of rows, which nothing in this repo reads but a report repo might. Verified clean and left alone: the four `area.yml` appends touch nothing else and cannot land inside another key's block; no live code assumes three years; the README carries no surviving three-year claim; `change_interval` length validation is pre-existing and unchanged in kind. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZBWEFxANHj7gRrmQYcENt
NewGraphEnvironment
deleted the
79-annual-io-lulc-series-2017-2023-for-bulk
branch
September 5, 2026 20:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #79.
03_lulc_classify.Rbuiltyearsas the change-interval endpoints plus the midpoint, so everyarea published 2017/2020/2023 only. drift's
dft_rast_break_class()needs the full annual seriesto tell a sustained land-cover switch from flicker.
lulc_annual: trueinarea.ymlmakesyearsthe whole interval; the transition is untouched either way, because it reads
yrs(
change_interval), neveryears.Turned on for bulk, necr, lnth, kotl and step 3 re-run for each.
Scope changed during planning, and the issue body was rewritten to match
PINE is out.
data/pine/has noprovenance.jsonand itsrasters/bt_ff04/*.tifare dated2026-07-12 — before
flooded0.5.0, the bankfull-units vintageCLAUDE.mdcalls dead, notmerely superseded. Step 3 there would classify land cover over a floodplain the repo has already
declared wrong and stamp fresh landcover provenance beside two absent sections. PINE and MCGR are
the only 2 of 23 area dirs in that state; both are tracked by #76, which now carries a note asking
for PINE's annual run to happen there so it is not run twice.
The proposed A/B gate could not have failed, so it was replaced. drift's
stac_cache_key()excludes
years, anddft_stac_fetchbuilds the STAC query asmin(years)..max(years)— identicalfor three years and seven. The committed log
scripts/floodplain_lcc/logs/runs/20260902_032619_run-area_neexdzii_prov-m4.log:461alreadyrecorded
14 items returned(2 tiles x 7 years) for a three-year request. Comparing the sharedyears across a 3-year and a 7-year run would have asserted that a file equals itself.
The 23.6 min cost figure described a path these runs do not take. No area sets
tile_sizein acommitted config, and
CLAUDE.mdrecords tiling as benchmarked and rejected under #8. Untilednumbers are below.
The change
lulc_annualinfp_read_configbesidetile_size, with anFP_LULC_ANNUALenv twin — thecomment at
run_area.R:73-77exists specifically to forbid committing a flag to the parityfixture and reverting it.
unquoted logical: yaml gives
TRUEfortrue/yes/on, but a quoted"true"is a charactervector that
isTRUE()reads as off, so the area would run three years under a config thatreads as annual. And
FP_LULC_ANNUALtakes a closed vocabulary rather than a truthinesstest —
%in% c("1","TRUE",...)reads a typo asFALSE, the same silent-off failure one layerout. That second one was found reviewing my own diff, after writing the first.
available_yearsguard. The issue asked for one;03:90-95already doessetdiff(years, lc_available), which covers a seven-element set unedited.inputs$yearsalready carries theresolved set and
provenance-check.Ralready asserts it matches the digest year set. A booleanbeside the vector it produced is a second source of truth, and recording it would force a
KEYS_LANDCOVERedit for no information.lulc_annualis area-owned: the annual areas span three regions, so a region file must neverset or clear it.
region_config-check.Rasserts it survives a region run, and the assertion wasproven to fire — adding
lulc_annualtoFP_REGION_OWNEDturns it PASS -> FAIL.what Modernize the README, publish it via GitHub Pages, and set the repo description + homepage #77 exists to stop, and this is per-area now. Re-rendered, determinism check green on all
three properties.
The runs
Step 3 only, so every area's floodplain geometry and sub-basins are the bytes step 2 last wrote.
Smallest area first on each machine, so a scaling data point landed before the large one.
49.3 min wall against 91.9 sequential.
bulk's 7 161 change patches match the figureCLAUDE.mdrecords for its 2026-09-02 run exactly — independent corroboration that the transitiondid not move.
Peak RSS does not track grid size, and I am not claiming to know why. KOTL at 203 Mcells peaked
2.6x higher than BULK at 168 Mcells; NECR at 55 Mcells peaked above LNTH at 62. The two large runs
were on different hosts (m4 128 GB, m1 64 GB), and the plausible cause is terra sizing its working
set against available RAM — so the number may describe the host rather than the job. It was not
isolated and should not be quoted as a per-area requirement. What the data does support: 64 GB
sufficed for the largest area run on it.
Acceptance — a named expected-failure set, greppable rather than eyeballed
provenance_ab-compare.Rreports a differinginputs_hashas a failure, and under this changelandcover[<scen>].inputs_hashmust differ. It also fails whenrun.datetime_utcdid not move, andsteps 1-2 deliberately do not re-run. So reading its exit code would have inverted the result:
landcover[<scen>]item_hash, drift 0.8.0 -> 0.13.0)network[*],floodplain[*]All four areas returned rc=0 on the full set:
years= 2017..2023, seven per-year digests, the2017/2020/2023 digests unchanged element-wise,
transition_content_sha256andtransition_patchesunchanged,outputs_hashunchanged,inputs_hashmoved, exactly sevenclassified_*gpkg layers and seven.tifs with no eighth,provenance-check.Rgreen, andbridge-check.Rgreen on the three areas carryingattribute_by.Baselines were backed up and asserted real (schema v2, three years, digests present) before
anything overwrote them — a v1 record would have degraded the acceptance to nothing, which is #73's
trap.
Disturbance attribution survived on all four (
in_fire/in_harvestpopulated), m4 reaching m1'sfresh-dbover tailscale. Not incidental:readme_functions.Rstops when those columns aremissing, and
bulkis the README'sFIG_AREA.Splitting across two machines needed a control first
Anything differing between the machines is a confound. m4 was levelled to m1 on
drift0.8.0 ->0.13.0,
sf-> 1.1.2,gdalcubes-> 0.7.4. The drift bump is not cosmetic: 0.8.0 fetches asingle STAC page, and paging to exhaustion landed at 0.10.0 — it would have truncated KOTL's item
set into a wrong raster with no error. An 8-operation capability probe ran before and after with no
regression (versions are not the assertion).
terracould not be matched — m1 has 1.9.34, CRAN current is 1.9-46 — so it was left at 1.9.11,which made terra the single remaining variable and the control a test of exactly it:
outputs_hashsha256:504624f6…transition_content_sha256sha256:1e379aee…transition_patchesSo the content survived a drift minor jump and the terra gap. necr and kotl then reproduced the
same result on real published areas — regenerated on m4 under terra 1.9.11 / drift 0.13.0, shared-year
digests matching baselines built on m1 under terra 1.9.34 / drift 0.8.0. That is
fp_raster_content_sha256()(#64) demonstrated rather than asserted.Every year was genuinely re-fetched. All four baselines were built under drift 0.8.0, whose
untiled cache keys predate the 0.10.0 change, so nothing was cache-served. That matters: drift's
stac_cache_key()excludesyears, so a warm cache would have made the shared-year assertionvacuous. A 1.1 GB copy of m1's cache was rsynced to m4 beforehand expecting a hit; it did not hit,
for that reason. Harmless, but the rationale was wrong and is recorded rather than repeated.
What a consumer will see move — neither is a defect
Both flagged on
stac_floodplains_bc#59:floodplain_landcover.gpkgbytes move for every area even where content does not — rewritingone layer into an existing GeoPackage is not byte-stable (GeoPackage writes are not byte-deterministic: gpkg_contents.last_change churns every rebuild #45). Byte equality answers "same
build?", not "same content?".
nge:landcover_keymoves for all four with no land-cover change, because the publish layermaps it to
inputs$item_hash, built from the requested years — seven year-lines instead ofthree, over an item set the widened request did not change. Item counts are per-AOI, not a
constant: necr and kotl each record 7 ids, one per year.
lulc_summary.rdsgains four years of rows. Nothing in this repo reads its contents(
readme_functions.Rbuilds both figures from the GeoPackages,run_region.Rtests existenceonly), but
scripts/floodplain_lcc/README.mdcalls it the store the report reads — a report repoplotting land cover by year off it will silently gain four categories.
Follow-ups filed rather than folded in
gdalcubeswrites every landcover cell and is in no provenance field. Levelled to 0.7.4on both machines so this PR carries no unrecorded difference, but recording it needs a
per-section key set: making it a required
KEYS_TOOLCHAINmember would failprovenance-check.Ron the
floodplain[*]andnetwork[*]entries this PR deliberately does not re-run.item_ids_completecannot be FALSE on drift >= 0.10; thenextlink it reads isstripped before it reaches callers, so
03:106-110's truncation warning is unreachable.Pre-existing, and it matters more now that seven
item_idsgroups ride behind it.lulc_annualalready on, so neither is run twice.stac_floodplains_bc#61— the year abstraction, split out of stac#59, which bundled a codegeneralization, a collection-wide policy decision (the Support multiple species per area coexisting in shared data/<area>/ outputs #23 uniform-key rule now sees two
populations) and a data operation behind three different blockers. stac#59 rescoped to the publish
and to four areas.
Review findings closed in
fc3b5ceAn adversarial review of the branch diff returned seven findings. Four were real and are fixed:
lulc_annualsilently ran three years.lulc_annual:,~andnullall parse toNULL, so the!is.null()short-circuit skipped the type guard andisTRUE(NULL)is FALSE —the exact silent-off class the guard exists to close, reached by blanking a value or commenting
out the
true. Measured on all three spellings; now keyed on%in% names(cfg).fp_lulcasserted
>= 0.6.0(wheretile_sizearrived) while the evidence log states that before0.10.0 the fetch pages nothing, so an AOI spanning more than one page is built from a partial
item set — a wrong raster, silently. Correctness floor, not a feature floor, and KOTL is one of
the four areas turned annual. Raised, with
CLAUDE.mdPrerequisites to match.provenance-check.Ronly compared the digest year setagainst
inputs$years— two values the same run wrote, which cannot disagree. Nothing readrasters/<scen>/or the gpkg layer list, so a reverted area recorded three years, passed green,and kept four orphan layers and four orphan tifs (Stale transition layers published alongside the current one (three layers, two obsolete) #55's class, which
gpkg_prune-legacy.Rdoesnot sweep). 7c now reconciles both artefact year sets. Proven to fire: mutating necr's
provenance to three years turns both arms red naming the disk and recorded sets; the restore
leaves it green.
lulc_annual: yis legal YAML butFP_LULC_ANNUAL=ywas refused, as wasT/F. They now agree, and the value is trimmed.Also:
CLAUDE.mdgained thelulc_annualbullet its sibling keys all have (plus the one-way-doorand RSS caveats, and a note that
FP_LULC_ANNUALis inherited byrun_region.R's children), and Icorrected a number I had generalized wrongly — the 14-item STAC response is neexdzii's two-tile
figure, not the four areas'. Fixed here, in the evidence log and in stac#59.
Verified clean and deliberately left alone: the four
area.ymlappends touch nothing else andcannot land inside another key's block; no live code assumes three years; no three-year claim
survives in the README;
change_intervallength validation is pre-existing and unchanged in kind.37 guard paths re-exercised after the fixes, all passing.
provenance-check.Rgreen on all fourareas plus the fixture on both machines, with the new arms confirmed by grep rather than exit
code.
Verification
Evidence:
scripts/floodplain_lcc/logs/20260905_lulc-annual_split-run.md, over the gitignoredlogs/runs/20260905_lulc-annual_*set.Closes #79.
🤖 Generated with Claude Code
https://claude.ai/code/session_014ZBWEFxANHj7gRrmQYcENt