Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
## Outcome

Folded `prep_subsurfaceflow` into `prep_natural` so subsurfaceflow positions enter the per-species observation/habitat lift via `<schema>.natural_barriers` (gated on `cfg$pipeline$break_order`). Restores bcfp parity for blkey 356286055 BT rearing on HARR (0 → 6.509 km). HARR full-WSG diffs collapsed: rearing_stream −10.4% → −4.19%, rearing −1.84%, spawning −1.6%. 15-WSG `tar_make` 33/33 across both bundles, 53m. Reproducibility verified — second `tar_make` byte-identical (`link_value` digest `5a641892b82604259b0ba168ea093661`, 0/1057 rows differ). Default-bundle bit-identical vs pre-fix (0/581 rows changed). HORS −7.68% BT residual confirmed as a separate mechanism (parent-stream-order/child-order rearing bypass — fresh#158 / link#96).

Closed by: PR #89 (merged into link v0.20.0 release commit `d8566d2`).
75 changes: 75 additions & 0 deletions planning/archive/2026-05-issue-88-subsurfaceflow-fold/findings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Findings — link#88

## Diagnosis (2026-04-30)

### Single-stream trace: HARR blkey 356286055

- link `fresh.streams_habitat`: 21 segments, all `rearing=FALSE` for BT, all `accessible=FALSE` for BT
- bcfp `streams_access`: `access_bt = 1` on every segment, `barriers_bt_dnstr = {}` (zero natural BT barriers)
- bcfp `barriers_anthropogenic_dnstr`: 2 entries (DAM at 356282804 DRM 739, ROAD/DEMOGRAPHIC at 356282804 DRM 658) — both flagged in `barriers_remediations` (REMEDIATED)
- Two **subsurfaceflow** points downstream on 356282804 (DRMs 265, 279) in `barriers_subsurfaceflow`
- 55 anadromous obs (CH/CM/CO) upstream of (356282804, 265) — clears bcfp's threshold (1 for BT, 5 for anadromous)
- bcfp lifts both subsurfaceflow points → `barriers_bt` and `barriers_ch_cm_co_pk_sk` empty in this drainage → BT/CH/CO credit upstream

### Why link doesn't lift

`lnk_pipeline_prepare()` build order (current):

1. `prep_load_aux` → falls, definite, control, habitat
2. `prep_gradient` → gradient_barriers_raw (pruned, ltree-enriched)
3. **`prep_natural` → `<schema>.natural_barriers` = gradient + falls** ← subsurfaceflow NOT here
4. `prep_overrides` → calls `lnk_barrier_overrides(barriers = natural_barriers)` → per-species skip list
5. **`prep_subsurfaceflow`** (opt-in) → `<schema>.barriers_subsurfaceflow` ← runs AFTER overrides

`lnk_pipeline_classify_build_breaks()` then UNIONs `barriers_subsurfaceflow` directly into `fresh.streams_breaks` with label `blocked`. Since the override skip list never saw it, `frs_habitat_classify(barrier_overrides = ...)` cannot lift it. All species get blocked at the subsurfaceflow position.

### bcfishpass natural barrier construction (per-species)

`model/01_access/sql/model_access_bt.sql` — `barriers_bt`:

- gradient_25 + gradient_30 + falls + **subsurfaceflow** + user_definite
- LIFT: any obs upstream (BT/CH/CM/CO/PK/SK/ST), or any habitat upstream
- user_definite always retained

`model/01_access/sql/model_access_ch_cm_co_pk_sk.sql` — `barriers_ch_cm_co_pk_sk`:

- gradient_15/20/25/30 + falls + **subsurfaceflow** + user_definite
- LIFT: ≥5 anadromous obs upstream (post-1990), or any habitat upstream
- `user_barriers_definite_control.barrier_ind = TRUE` blocks the obs lift; habitat lift unaffected
- user_definite always retained

bcfp's `barriers_anthropogenic` (PSCIS, dams, road crossings) is **NOT** in the per-species barrier set. It's tracked separately for downstream-of-crossing accountability. Anthropogenic barriers don't gate species access in bcfp's habitat sums.

### Design diagnosis

`.lnk_pipeline_prep_subsurfaceflow` was added in PR #82 as its own helper. Treated subsurfaceflow as a parallel concept needing a parallel pipeline phase. But subsurfaceflow is just **a third row in the same union** that `prep_natural` already builds for gradient + falls. The wiring miss: subsurfaceflow's positions never reached `natural_barriers`, so the per-species lift skipped it entirely.

`prep_natural` is the right home — bcfp's source of truth confirms gradient + falls + subsurfaceflow is *the* natural-barrier union per species.

### Where it surfaces (15-WSG rollup, parity)

| WSG | sp | metric | link | bcfp | diff_pct |
|------|----|-----------------|------|------|---------:|
| HARR | CH | rearing_stream | 118 | 139 | -14.8 |
| HARR | CO | rearing_stream | 134 | 155 | -13.3 |
| HARR | ST | rearing_stream | 157 | 177 | -11.6 |
| HARR | BT | rearing_stream | 292 | 326 | -10.4 |
| HORS | BT | rearing_stream | 366 | 396 | -7.7 |
| LFRA | BT | rearing_stream | 1020 | 1103 | -7.5 |
| LFRA | BT | rearing | 1670 | 1800 | -7.2 |
| HORS | CH | rearing_stream | 167 | 179 | -6.8 |

LILL/VICT unaffected — sparse subsurfaceflow + sparse fish observations above what does exist.

## What is NOT involved

- #83 (anthropogenic dam design): the dam at DRM 739 and road at DRM 658 carry `barrier`/`potential` labels in `fresh.streams_breaks`. Default `label_block = "blocked"` means they don't gate. Not the cause.
- `barriers_definite`: intentionally separate per bcfp — never lifted via obs/hab. Link mirrors that.
- `barriers_remediations`: bcfp tracks for downstream reporting only; doesn't gate access.

## Versions at diagnosis

- link 0.19.0 (commit e4e7a6e)
- fresh 0.25.0
- bcfishpass 440bc1e (2026-04-28)
- fwapg local Docker (port 5432)
17 changes: 17 additions & 0 deletions planning/archive/2026-05-issue-88-subsurfaceflow-fold/progress.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Progress — link#88

## Session 2026-04-30

- Diagnosis: traced HARR blkey 356286055 BT under-credit to subsurfaceflow positions on downstream tributary 356282804 not reaching `natural_barriers` for the per-species observation/habitat lift.
- Read bcfp SQL (`model_access_bt.sql`, `model_access_ch_cm_co_pk_sk.sql`) — confirmed bcfp's natural-barrier union includes subsurfaceflow with same lift rules.
- Confirmed default-bundle off-switch is preserved verbatim (omit `subsurfaceflow` from `cfg$pipeline$break_order`).
- Filed link#88 with diagnosis + proposed fix.
- Branch `88-fold-subsurfaceflow-natural` from main.
- PWF baseline (commit 4bd9ca0).
- Code change: extended `.lnk_pipeline_prep_natural` signature `(conn, aoi, cfg, loaded, schema)`; absorbed subsurfaceflow body, gated on `cfg$pipeline$break_order`; deleted standalone `.lnk_pipeline_prep_subsurfaceflow`; pruned conditional call from `lnk_pipeline_prepare()`. `devtools::document()` clean.
- Tests: 3 new test cases in `tests/testthat/test-lnk_pipeline_prepare.R` — opted-out, opted-in (per-statement assertion that link#88 fix INSERT fires), control-table honoured. 44/44 pass.
- Code-check: 3 rounds. Rounds 1–2 clean. Round 3 caught a fragile cross-statement regex in the test; replaced with per-statement `any(grepl & grepl)`. Sanity-verified the assertion catches the regression.
- Pre-flight: HARR single-WSG `compare_bcfishpass_wsg(wsg = "HARR", config = lnk_config("bcfishpass"))` 89.5 s. blkey 356286055 BT credits 6.509 km (was 0). HARR BT diffs collapsed: rearing_stream -10.4% → -4.19%, rearing -1.84%, spawning -1.6%.
- 15-WSG `tar_make` (53m 2s, 33/33). Parity dramatic on HARR (CH/CO/ST <0.32%; BT residual -4.19%) and LFRA (CH/CO/ST <0.6%; BT residual -3.75%). HORS unchanged (-7.68% rearing_stream BT) — different mechanism, follow-up needed. Default-bundle bit-identical (0 of 581 rows changed).
- Reproducibility re-run (52m 55s, 33/33). 0 of 1057 link_value rows differ; digest match (`5a641892b82604259b0ba168ea093661`). ✓
- Code commit `a21a8f8`. PWF + verification logs commit. Ready for PR.
37 changes: 37 additions & 0 deletions planning/archive/2026-05-issue-88-subsurfaceflow-fold/task_plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Task Plan — link#88: Fold subsurfaceflow into natural barriers

## Phase 1: Setup
- [x] File link#88 with diagnosis + proposed fix
- [x] Branch `88-fold-subsurfaceflow-natural` from main
- [x] PWF baseline (task_plan, findings, progress)

## Phase 2: Code change
- [x] Extend `.lnk_pipeline_prep_natural(conn, aoi, cfg, loaded, schema)` to absorb subsurfaceflow body, gated on `"subsurfaceflow" %in% cfg$pipeline$break_order`
- [x] Append subsurfaceflow rows to `<schema>.natural_barriers` (label `blocked`)
- [x] Delete `.lnk_pipeline_prep_subsurfaceflow` helper
- [x] Remove conditional call from `lnk_pipeline_prepare()`
- [x] `devtools::document()` — refresh roxygen for prep_natural

## Phase 3: Tests
- [x] `tests/testthat/test-lnk_pipeline_prepare.R`: subsurfaceflow opted in → INSERT into natural_barriers fires (per-statement assertion)
- [x] Same file: subsurfaceflow not opted in → no subsurfaceflow code path runs
- [x] Same file: subsurfaceflow honours barriers_definite_control
- [x] `devtools::test(filter = "lnk_pipeline_prepare")` clean (44/44 PASS)

## Phase 4: Code-check
- [x] `/code-check` on staged diff — 3 rounds. Round 3 caught fragile cross-statement regex; tightened to per-statement `any(grepl & grepl)`. Final clean.

## Phase 5: Verification
- [x] HARR single-WSG pre-flight `tar_make` — `data-raw/logs/20260430_11_preflight_harr_link88.txt`
- [x] blkey 356286055 BT rearing credits **6.509 km** (was 0)
- [x] Full 15-WSG `tar_make` — `data-raw/logs/20260430_12_tar_make_15wsg_link88.txt` (53m 2.2s, 33/33 targets)
- [x] HARR CH/CO/ST rearing_stream closed to ±0.32% (BT residual -4.2%, separate mechanism noted)
- [x] LFRA CH/CO/ST closed to ±0.6% (BT residual -3.75%)
- [x] HORS unchanged (-7.68% BT) — different mechanism, follow-up issue
- [x] Default-bundle rollup bit-identical (0 of 581 link_value rows changed)
- [x] Reproducibility: second `tar_make` byte-identical (`link_value` digest `5a641892b82604259b0ba168ea093661` matches across runs; 0 of 1057 rows differ)

## Phase 6: Ship
- [x] Atomic commits with PWF checkbox flips
- [x] PR with `Fixes #88` and `Relates to NewGraphEnvironment/sred-2025-2026#24` — PR #89 merged 2026-05-01 as part of link 0.20.0
- [x] Archive PWF after merge