Honour barriers_definite_control at the override step, per-species (#44)

DESCRIPTION

@@ -1,6 +1,6 @@
 Package: link
 Title: Crossing Connectivity Interpretation
-Version: 0.5.0
+Version: 0.6.0
 Authors@R:
     person("Allan", "Irvine", , "airvine@newgraphenvironment.com",
            role = c("aut", "cre"),

NEWS.md

@@ -1,3 +1,14 @@
+# link 0.6.0
+
+Honour `user_barriers_definite_control.csv` at the observation-override step.
+
+- `lnk_barrier_overrides()` now excludes observations upstream of control-flagged positions from counting toward the override threshold, matching bcfishpass's access SQL. Previously controlled positions (concrete dams, long impassable falls, diversions) could be re-opened by upstream historical observations ([#44](https://github.com/NewGraphEnvironment/link/issues/44)).
+- Gated per-species by a new `observation_control_apply` column in `parameters_fresh.csv` — TRUE for CH/CM/CO/PK/SK/ST; FALSE for BT/WCT; NA for CT/DV/RB. Residents routinely inhabit reaches upstream of anadromous-blocking falls (post-glacial headwater connectivity, no ocean-return requirement), so their observations still override. Matches bcfishpass's per-model application.
+- Habitat-confirmation override path intentionally bypasses the control table — expert-confirmed habitat is higher-trust than observations, and bcfishpass's `hab_upstr` CTE has no control join either.
+- `.lnk_pipeline_prep_overrides` now passes the control table to `lnk_barrier_overrides()` when the config manifest declares `barriers_definite_control`. Manifest key is the contract; no DB probe.
+- `.lnk_pipeline_prep_load_aux` now always creates a schema-valid (possibly empty) `barriers_definite_control` table when the manifest declares the key — fixes an asymmetric gating bug that would have raised "relation does not exist" on AOIs with zero control rows.
+- End-to-end validation WSG: DEAD (Deadman River) added to `data-raw/_targets.R`. It has a single `barrier_ind = TRUE` control row at FALLS (356361749, 45743) with six anadromous observations upstream and zero habitat coverage — the unique combination that actively exercises the filter. All four prior WSGs (ADMS/BULK/BABL/ELKR) were rescued by either the observation threshold or habitat path, making them parity checks rather than filter tests.
+
 # link 0.5.0
 
 Documentation and narrative for the targets pipeline.

R/lnk_barrier_overrides.R

@@ -27,11 +27,20 @@
 #' @param control Character or `NULL`. Schema-qualified table of barrier
 #'   controls with columns: `blue_line_key`, `downstream_route_measure`,
 #'   `barrier_ind`. Barriers in this table with `barrier_ind = TRUE` cannot
-#'   be overridden.
+#'   be overridden **by observations** — but only for species where
+#'   `params$observation_control_apply` is TRUE. Resident species routinely
+#'   inhabit reaches upstream of anadromous-blocking falls (post-glacial
+#'   connectivity, no ocean-return requirement), so their observations still
+#'   count unless this flag says otherwise. Habitat confirmations
+#'   (`habitat` argument) are higher-trust than observations — they bypass
+#'   the control table entirely, for all species.
 #' @param params Data frame with per-species parameters. Must have columns:
 #'   `species_code`, `observation_threshold`, `observation_date_min`,
-#'   `observation_buffer_m`, `observation_species`. See
-#'   `configs/bcfishpass/parameters_fresh.csv` for format.
+#'   `observation_buffer_m`, `observation_species`. Optional column
+#'   `observation_control_apply` (logical) — when TRUE, the `control` table
+#'   blocks overrides for this species; when FALSE/NA/missing, the species
+#'   ignores control. Bcfishpass defaults: TRUE for CH/CM/CO/PK/SK/ST,
+#'   FALSE for BT/WCT. See `configs/bcfishpass/parameters_fresh.csv`.
 #' @param cols_index Character vector. Column names to index on the
 #'   barriers table for `fwa_upstream()` performance. Indexes are created
 #'   `IF NOT EXISTS`. Default `c("blue_line_key", "wscode_ltree",
@@ -135,22 +144,36 @@ lnk_barrier_overrides <- function(conn,
     obs_sp_list <- if (is.na(obs_sp_str)) sp else trimws(strsplit(obs_sp_str, ";")[[1]])
     obs_sp_sql <- paste0("'", obs_sp_list, "'", collapse = ", ")
 
+    # Species-level opt-in for the control filter. bcfishpass applies control
+    # only in the anadromous access models (CH/CM/CO/PK/SK, ST) — residents
+    # (BT, WCT) and sub-CT species routinely live upstream of anadromous
+    # barriers (post-glacial headwater connectivity, no ocean-return
+    # requirement), so their observations should still override.
+    ctrl_apply_col <- species_to_process$observation_control_apply[i]
+    ctrl_apply <- isTRUE(as.logical(ctrl_apply_col))
+
     overrides_found <- 0L
 
-    # Control table: any matching control row prevents the override.
-    # barrier_ind is used separately in barrier loading (true = keep, false = remove).
-    # Here we only care about presence — if a control row exists for this barrier
-    # position, observations/habitat don't override it.
-    ctrl_where <- if (!is.null(control)) {
+    # Control table: a matching control row with barrier_ind = TRUE
+    # blocks the override. `NOT EXISTS` (rather than a LEFT JOIN + filter)
+    # keeps two things right in one shot — the barrier is blocked only
+    # when at least one TRUE control row matches (mixed TRUE/FALSE within
+    # the 1 m tolerance resolves to "blocked"), and the outer GROUP BY /
+    # HAVING count(...) aggregation does not get row-multiplied by a join
+    # to control. Gated per-species by `observation_control_apply`.
+    ctrl_where <- ""
+    ctrl_filter <- if (!is.null(control) && ctrl_apply) {
       sprintf(
-        "LEFT JOIN %s c
-           ON b.blue_line_key = c.blue_line_key
-           AND abs(b.downstream_route_measure - c.downstream_route_measure) < 1",
+        "AND NOT EXISTS (
+           SELECT 1 FROM %s c
+           WHERE c.blue_line_key = b.blue_line_key
+             AND abs(b.downstream_route_measure - c.downstream_route_measure) < 1
+             AND c.barrier_ind::boolean = true
+         )",
         control)
     } else {
       ""
     }
-    ctrl_filter <- if (!is.null(control)) "AND c.blue_line_key IS NULL" else ""
 
     # --- Observation-based overrides (JOIN pattern, not correlated subquery) ---
     if (!is.null(observations) && threshold > 0) {
@@ -192,6 +215,11 @@ lnk_barrier_overrides <- function(conn,
     }
 
     # --- Habitat confirmation overrides (any confirmed habitat upstream) ---
+    # Control filter intentionally NOT applied here. Expert-confirmed
+    # habitat is a higher-trust signal than the control table — by the
+    # time a reviewer has marked habitat as confirmed upstream of a
+    # position, they have already considered the barrier's passability.
+    # bcfishpass does the same: `hab_upstr` CTE has no control join.
     if (!is.null(habitat)) {
       sql <- sprintf(
         "INSERT INTO %s (blue_line_key, downstream_route_measure, species_code)
@@ -204,18 +232,15 @@ lnk_barrier_overrides <- function(conn,
            ON s.blue_line_key = h.blue_line_key
            AND round(h.upstream_route_measure::numeric) >= round(s.downstream_route_measure::numeric)
            AND round(h.upstream_route_measure::numeric) <= round(s.upstream_route_measure::numeric)
-         %s
          WHERE whse_basemapping.fwa_upstream(
            b.blue_line_key, b.downstream_route_measure,
            b.wscode_ltree, b.localcode_ltree,
            h.blue_line_key, h.upstream_route_measure,
            s.wscode_ltree, s.localcode_ltree,
            false, 200)
-         %s
          ON CONFLICT DO NOTHING",
         to, sp,
-        barriers, habitat, obs_sp_sql,
-        ctrl_where, ctrl_filter)
+        barriers, habitat, obs_sp_sql)
 
       n <- DBI::dbExecute(conn, sql)
       overrides_found <- overrides_found + n

R/lnk_pipeline_prepare.R

@@ -125,14 +125,27 @@ lnk_pipeline_prepare <- function(conn, aoi, cfg, schema,
          downstream_route_measure double precision)", schema))
   }
 
-  # --- Barriers-definite control (per-WSG, used to prune gradient barriers) ---
+  # --- Barriers-definite control (per-WSG, used to prune gradient barriers
+  # AND to lock positions against observation-based overrides). Mirror the
+  # barriers_definite pattern above — whenever the manifest declares the
+  # key, ensure a schema-valid table exists even if this AOI has zero rows,
+  # so downstream steps can gate on the manifest field rather than probing
+  # the DB.
   ctrl_all <- cfg$overrides$barriers_definite_control
   if (!is.null(ctrl_all)) {
     ctrl <- ctrl_all[ctrl_all$watershed_group_code == aoi, ]
     if (nrow(ctrl) > 0) {
       DBI::dbWriteTable(conn,
         DBI::Id(schema = schema, table = "barriers_definite_control"),
         ctrl, overwrite = TRUE)
+    } else {
+      .lnk_db_execute(conn, sprintf(
+        "DROP TABLE IF EXISTS %s.barriers_definite_control", schema))
+      .lnk_db_execute(conn, sprintf(
+        "CREATE TABLE %s.barriers_definite_control (
+           blue_line_key integer,
+           downstream_route_measure double precision,
+           barrier_ind text)", schema))
     }
   }
 
@@ -252,10 +265,21 @@ lnk_pipeline_prepare <- function(conn, aoi, cfg, schema,
     .lnk_quote_literal(schema)))
   habitat_arg <- if (nrow(habitat_exists) > 0) habitat_tbl else NULL
 
+  # Manifest-driven gating. `.lnk_pipeline_prep_load_aux` writes
+  # `<schema>.barriers_definite_control` exactly when this manifest key
+  # is declared on the config bundle, so the config field itself is the
+  # direct contract for whether control is in play — no DB probe needed.
+  control_arg <- if (!is.null(cfg$overrides$barriers_definite_control)) {
+    paste0(schema, ".barriers_definite_control")
+  } else {
+    NULL
+  }
+
   lnk_barrier_overrides(conn,
     barriers = paste0(schema, ".natural_barriers"),
     observations = observations,
     habitat = habitat_arg,
+    control = control_arg,
     params = cfg$parameters_fresh,
     to = paste0(schema, ".barrier_overrides"),
     verbose = FALSE)

comms/rtj/20260423_m1_r_worker_verified.md

@@ -0,0 +1,35 @@
+---
+from: rtj
+to: link
+topic: M1 verified as a ready R-worker host; crew.cluster 0.4.0 API gap
+status: open
+---
+
+## 2026-04-23 — rtj
+
+### M1 verified as a ready R-worker host (2026-04-22)
+
+Ran `rtj/scripts/hosts/crew-worker_verify.R` to validate the infra primitive under whatever launcher you pick for PR 3-of-3. 7/7 checks pass, 1.1s M4→M1→M4 round-trip via raw `ssh m1 'Rscript -'` stdin pipe.
+
+Confirmed on M1:
+
+- R 4.5.2 with `link`, `fresh`, `targets`, `crew` all loading cleanly
+- `.libPaths()` has user library first (`~/Library/R/arm64/4.5/library`)
+- `PG_DB_SHARE` propagates to non-interactive SSH R via `~/.Renviron`
+- tailnet ACL permits peer → M4 TCP callbacks
+
+### One pitfall worth flagging on launcher choice
+
+`crew.cluster` 0.4.0 does NOT export `crew_controller_cluster` — only HPC-scheduler variants (`crew_controller_sge/lsf/pbs/slurm`). If you were planning to use a generic "ssh" controller from crew.cluster, it doesn't exist. Options I see:
+
+- `crew::crew_controller_local()` on M4 + custom `crew_class_launcher` subclass for SSH
+- `clustermq` (mature, ssh-native)
+- Raw `mirai::daemon` + bespoke dispatcher
+
+Not pushing an opinion; just saving you the 5 min I spent discovering this.
+
+### Landing note (per soul 2026-04-23 branch ruling)
+
+This thread is landing on your `44-barriers-definite-control` branch because that's where link's local clone is currently checked out. It won't reach `main` until PR #44 merges. If you need it visible on `main` sooner, cherry-pick or merge when convenient.
+
+Close when acknowledged.

data-raw/_targets.R

@@ -1,8 +1,10 @@
 # data-raw/_targets.R
 #
 # Pipeline definition for the bcfishpass comparison. Orchestrates the
-# six lnk_pipeline_* phase helpers across the four validated watershed
-# groups and rolls up the per-WSG diff tibbles into one aggregate.
+# six lnk_pipeline_* phase helpers across five watershed groups and
+# rolls up the per-WSG diff tibbles into one aggregate. ADMS/BULK/BABL/
+# ELKR are the numerical-parity WSGs; DEAD (added 2026-04-23 with #44)
+# is the end-to-end test for the `barriers_definite_control` filter.
 #
 # Run from the link repo root:
 #   Rscript -e 'targets::tar_config_set(script = "data-raw/_targets.R",
@@ -36,7 +38,13 @@ tar_option_set(
   packages = c("link", "fresh", "DBI", "RPostgres", "tibble", "dplyr")
 )
 
-wsgs <- c("ADMS", "BULK", "BABL", "ELKR")
+# DEAD (Deadman River) is the end-to-end test for the control filter.
+# It has one `barrier_ind = TRUE` control row with 6 observations upstream
+# in the CH/CM/CO/PK/SK pool and zero habitat-classification coverage —
+# the unique combination that actively exercises the filter. The other
+# four WSGs are numerical-parity checks; their TRUE control rows are
+# all rescued by the habitat path or sit below the observation threshold.
+wsgs <- c("ADMS", "BULK", "BABL", "ELKR", "DEAD")
 
 list(
   tar_target(cfg, link::lnk_config("bcfishpass")),
@@ -53,7 +61,8 @@ list(
     rollup,
     dplyr::bind_rows(
       comparison_ADMS, comparison_BULK,
-      comparison_BABL, comparison_ELKR
+      comparison_BABL, comparison_ELKR,
+      comparison_DEAD
     )
   )
 )