Provincial run autonomy + 8 script renames (v0.38.0)#174
Merged
Conversation
Provincial run autonomy + script renames. Built on top of v0.37.0 (#168) which decoupled pipeline/compare and switched the resume gate to PG state — that foundation lets the autonomy CLI surface stay thin and the 8 renames stay mechanical. 7 phases: CLI on trifecta_provincial.sh, CLI on province_run.sh, M4+M1 integration test, 8-script rename, post-rename smoke, rtj cross-repo update, release v0.38.0. Cypher integration deferred to a follow-up issue. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…sthrough Three new CLI flags + two bug fixes layered onto the multi-host dispatcher: - --wsgs=A,B,C — restrict to a subset of the bundle's WSG list. Intersected with the presence-filtered set inside SPLIT_R; unknown WSGs error loud via stop(call.=FALSE). - --no-cyphers — skip cypher hosts entirely. Wipes CY_WORKSPACES so N_CY=0; downstream cypher loops (wrap, launch, RDS pullback, R-log pullback) become no-ops naturally. - --force — forward to every per-host Rscript invocation. Bypasses both the PG-state and RDS resume gates from #168. Phantom-cy fix: R's paste0("cy", integer(0)) returns "cy" length-1 (constant recycling), which would put a non-existent cypher in host_keys when n_cy=0. Three-branched cy_host_keys construction: character(0) / "cy" / paste0(...). Empty CY_WORKSPACES init now explicit CY_WS_ARR=() rather than read -r -a (which yields a single-element-empty-string array). Error-surface fix: SPLIT_OUT=$(Rscript ...) under set -e propagates the inner exit code but discards the output. A typo like --wsgs=BOGUS aborted bash with no operator-visible message. Wrapped with explicit || block that dumps SPLIT_OUT to stderr and exits with the R rc, so the R-side stop() reaches the operator. Validated: - bash -n clean - isolated SPLIT_R run with --wsgs=DEAD,ADMS --no-cyphers produces all_wsgs=ADMS,DEAD, n_cy=0, host_keys=m4,m1, cy_host_keys length 0 - --wsgs=BOGUS,ADMS surfaces the R stop() message verbatim Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…e passthrough Five new CLI flags layered onto the 10-step provincial wrapper, plus config-aware paths so the script works with any bundle / WSG subset / host count. - --wsgs=A,B,C restrict to a subset of the bundle WSG list - --config=<name> bundle (default: bcfishpass) - --schema=<name> override cfg$pipeline$schema for the output - --no-cyphers M4+M1 only — skip cypher spin/prep/archive/burn - --force forward to per-host Rscript (bypass resume gates) Build DISPATCH_FLAGS for trifecta_provincial.sh passthrough. Step gating under --no-cyphers: - Step 3 (cypher_up.sh) skipped - Step 4 (cypher_prep.sh) skipped - Step 5 cypher archives skipped (M4+M1 always archive) - Step 7 invocation omits --cy-workspaces=... - Step 9 consolidate sources M1-only variant - Step 10 trap-EXIT burn no-op (CYPHERS_UP stays 0) Step 8 ANN_CSV path is now config-aware: provincial_parity/ for bcfishpass (back-compat), provincial_<config>/ otherwise. Step 9 target-schema resolution: explicit --schema wins, else lnk_config(CONFIG_NAME)$pipeline$schema lookup with stop() inside the Rscript heredoc plus post-lookup empty/NULL guards. Replaces a silent "fresh" fallback that masked misconfigured --config= flags (round-1 code-check fix). Auto-skip-smoke when --no-cyphers OR --wsgs is set — the smoke harness assumes 3 cyphers + fixed per-host WSGs, both break under the new subset modes. Validated: - bash -n clean - TARGET_SCHEMA lookup: bcfishpass→"fresh", default→"fresh", BOGUS errors loud with bundle-path hint. - /code-check round 2 clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-clean
When the umbrella receives --schema=<name>, fire a Step 0 pre-clean
that drops <name> on every host before dispatch starts. Eliminates
the leftover-WSG class of consolidate failures: pg_dump --schema=<X>
on a source host that has accumulated WSGs from prior runs would
pull rows that collide with destination's current bucket data
(observed empirically in today's 16-WSG Phase 3 dispatch — 4 of M1's
8 WSGs failed pg_restore on duplicate keys).
state_clean.sh (currently named province_clean.sh until Phase 4
rename) gains a --schemas=A,B,C scoped mode:
- Drops ONLY the listed schemas (parameterized IN-list via awk)
- Skips the heuristic working* / fresh_* / fresh wipe
- Skips the snapshot_bcfp.sh reload (canonical fresh isn't touched)
- Wall: ~10-20s vs ~2-3 min for full mode
Empty --schemas= is explicitly rejected with FATAL to prevent
silent destructive fall-through under dynamic-arg construction
(round-1 code-check fix).
province_run.sh Step 0:
- Fires when --schema= is set on the umbrella
- Routes through state_clean --schemas=$SCHEMA [--skip-cy]
- Captures non-zero exit and aborts the umbrella loud
- Skipped when --schema is empty (writes go to canonical fresh; Step
1+2 snapshot handles that path)
Validated:
- bash -n clean both files
- Empty --schemas= guard fires as expected ("FATAL: --schemas=
requires at least one schema")
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two-attempt journey: Attempt 1 (no pre-clean): 16-WSG dispatch landed cleanly on M4+M1's fwapg per-host, RDS files all came back, annotated CSV written, but consolidate hit 6 duplicate-key errors during pg_restore. Root cause: M1's fresh_default had leftover WSGs from yesterday's province-wide run; pg_dump --schema=fresh_default pulled rows outside the current bucket which collided with M4's data. Only 12 of 16 landed. Root-cause fix landed in commit 89da284 (state_clean.sh scoped mode + province_run.sh Step 0 pre-clean). Operator-friendly: --schema=<name> auto-pre-cleans that schema on all hosts before dispatch. Attempt 2 (with pre-clean): 16 of 16 WSGs in M4 fresh_default.streams (468,631 rows), 20m total wall (Step 0 + cold-cache pipeline + Step 9 consolidate), exit 0, no operator prompts. Annotated CSV: 343 rows (263 NOT_APPLICABLE + 66 UNEXPLAINED at >=2% + 14 WITHIN_TOLERANCE). The 66 UNEXPLAINED are methodology divergence for the default bundle vs bcfishpass; expected for these northern WSGs and surfaced to the operator as a non-blocking WARNING. Autonomy CLI validated end-to-end: - --wsgs=CARP,CRKD,FINA,FINL,FIRE,FOXR,INGR,LOMI,MESI,NATR,OSPK,PARA, PARS,PCEA,TOOD,UOMI - --config=default --schema=fresh_default - --no-cyphers --with-mapping-code --force Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mechanical rename pass on a known-working state (Phases 1-3 of #172 shipped + validated end-to-end before this commit lands): province_run.sh -> wsgs_run_pipeline.sh province_clean.sh -> state_clean.sh province_progress.sh -> progress_check.sh trifecta_provincial.sh -> wsgs_dispatch.sh run_provincial_parity.R -> wsgs_run_host.R consolidate_schema.R -> schema_consolidate.R archive_provincial_runs.sh -> runs_archive.sh balance_provincial_buckets.R -> buckets_balance.R All renames done via git mv so `git log --follow` traces history across the rename boundary. Internal reference updates and all live tree cross-references applied in this same commit so blame stays clean. Updated callers / docs: - The 8 renamed scripts themselves (self-references in usage blocks, log filename literals, cross-script invocations). - data-raw/{trifecta_smoke.sh,query_schema_delta.R,wsg_compare.R, wsg_pipeline_run.R,README.md} - research/{provincial_run_runbook,post_compact_provincial_handoff, bcfp_compare_mapping_code,distributed_2hosts_2026_05_01, provincial_parity_2026_05_{01,11,12}}.md - CLAUDE.md, R/utils.R Sealed (intentionally NOT updated): - NEWS.md historical entries (v0.36.1 / v0.37.0 etc. reference the old names because that's what those releases shipped) - planning/archive/** (sealed history of prior PWFs) The wsg_*-family naming makes the singular/plural distinction meaningful: wsg_pipeline_run.R / wsg_compare.R are per-WSG halves (from #168); wsgs_run_host.R loops over a host's bucket; wsgs_dispatch .sh fans the work across hosts; wsgs_run_pipeline.sh is the operator umbrella that wraps the whole thing. Other wrappers got "mixed nouns" (more descriptive) per yesterday's locked-in decision: state_clean / progress_check / runs_archive / buckets_balance / schema_consolidate. bash -n clean on all renamed shells. /code-check round 1 clean: 7 concerns verified including sed idempotency (no new name contains any old name as substring). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
3-WSG smoke via the renamed wsgs_run_pipeline.sh ran end-to-end: - Exit 0, 3m 13s wall - Step 0 pre-clean dropped fresh_default on M4+M1 - Step 7 dispatch via wsgs_dispatch.sh -> wsgs_run_host.R - Step 9 consolidate succeeded (target schema fresh_default) - 18/3 OK (pulled 18 RDS including legacy artifacts), 0 errors Pre-existing LPT-fallback edge case surfaced on the first 1-WSG attempt: with no timing CSVs for the requested config, the weighted split assigns all WSGs to M4 (highest weight host) -> M1 empty bucket -> empty-bucket guard fires. Not a rename bug; same behavior pre-rename. Worked around by using --config=default which has timing CSVs from Phase 3 runs. Cosmetic sweep: 10 occurrences of `[trifecta-provincial]` log prefix inside wsgs_dispatch.sh -> `[wsgs-dispatch]`. The Phase 4 sed pass only matched filename substrings, not hyphenated prose. devtools::test() still passes 1172/0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Provincial-run autonomy CLI + 8 operational-script renames to noun_verb. Closes #172.
bash data-raw/wsgs_run_pipeline.sh --wsgs=A,B,C --config=<name> --schema=<name> --no-cyphers --forceruns end-to-end (pre-clean → snapshot → archive → dispatch → acceptance → consolidate → burn) with zero mid-run prompts.wsgs_*for collection orchestrators,wsg_*for per-WSG (from Decouple bcfp compare from link pipeline run — separate compare-only step #168), mixed nouns for the rest.Phase 3 acceptance (live)
fresh_default.streamson M4Renames
data-raw/province_run.shdata-raw/wsgs_run_pipeline.shdata-raw/province_clean.shdata-raw/state_clean.shdata-raw/province_progress.shdata-raw/progress_check.shdata-raw/trifecta_provincial.shdata-raw/wsgs_dispatch.shdata-raw/run_provincial_parity.Rdata-raw/wsgs_run_host.Rdata-raw/consolidate_schema.Rdata-raw/schema_consolidate.Rdata-raw/archive_provincial_runs.shdata-raw/runs_archive.shdata-raw/balance_provincial_buckets.Rdata-raw/buckets_balance.RAll via
git mvsogit log --followpreserves history across the rename boundary./code-check (3 real bugs caught and fixed)
SPLIT_OUT=$(Rscript ...)+set -e)--config=--schemas=silently fell through to destructive defaultFiled-but-not-closed follow-ups (in NEWS + archived task_plan)
Test plan
devtools::test()— 1172 PASS / 0 FAILdevtools::check()— 0 errors; 3 warnings + 2 notes all pre-existing (no regression vs v0.37.0)bash -nclean on all renamed shell scripts/code-checkclean on every phase commitRelated Issues
Generated with Claude Code