Harmonization gains a second execution mode and starts producing evidence about its own output. Eleven PRs, three contributors.
Parallel Multi-Consent Execution Mode
A cohort now runs as one Seven Bridges task instead of one task per consent group. bdc-cohort-workflow.sh fans out bdc-workflow.sh under GNU parallel, then runs hv_dataqc source-vs-harmonized comparison across every consent group as a fan-in step — a cohort-wide comparison that had nowhere to run when each consent group was its own task.
Single Consent Execution Mode is unchanged and still supported; bdc-workflow.sh was not modified. --consent-parallelism 1 reduces the new mode to exactly serial for debugging.
Trans-specs are resolved to an immutable SHA and cloned once at startup, so every worker reads an identical tree and the run receipt records exactly which YAMLs produced the output.
Note that no cohort-mode app exists on the prod tier yet — putting it there is an app-creation step, not a code change. (#333, csiege)
Provenance and output validation
dm-bip extract-mapping-provenance emits PROV-shaped records of which studies, datasets, and variables feed each harmonized concept, plus a run Activity recording agent and timing. Specs are identified by commit-pinned GitHub permalinks, falling back to local path ids when a file is untracked or modified. Study → dataset → variable alignment is carried structurally, which is the property downstream consumers need and cannot reconstruct from flattened annotations. (#353, ptgolden)
A validate-output stage checks mapped output against DM_MAP_TARGET_SCHEMA and writes an advisory report. It never fails the pipeline — enforcement is a separate decision once the real failure profile on BDCHM is known.
Expect it to report findings immediately. On toy data every record fails, because linkml-map emits source-native int/float values into slots whose target range resolves to string and does not coerce. That non-conformance is pre-existing; this release makes it visible rather than introducing it. Zero records is reported as a problem, never a pass. (#358)
Diagnostics
dm-bip seven-bridges submit --profile turns on map-step diagnostics — py-spy CPU profiles and cgroup memory — through the app's Profile input. Single-consent mode only.
The cgroup capture now reads v1 paths as well as v2. BDC containers are cgroup v1, so peak memory previously came back empty on exactly the infrastructure it was meant to measure. Unreadable files print an explicit - rather than a bare header, so a silent blank is distinguishable from a real zero. (#343, #344)
Deployment
Releases now deploy by moving a tag rather than by hand-editing an app. A non-rc bdc-v* build publishes a :prod promotion pointer that the prod app pulls permanently. Pre-release tags are excluded by an explicit guard, so a release candidate cannot become production.
Every build also publishes an immutable :sha-<12-char-commit> tag alongside :latest, and each deployment tier writes to its own registry repository so no two triggers overwrite the same :latest.
DEFAULT_APP no longer pins an app revision. Seven Bridges resolves a bare app id to its latest revision, so the default stops going stale whenever an app is re-saved; --app <id>/<N> still pins explicitly for testing.
RELEASING.md now documents version-numbering policy, the registry tier mapping, and — since app definitions live only on the platform — what each Seven Bridges app runs and which inputs it declares. (#343, #333, #364)
Orchestration groundwork
A pluggable source seam and completion-notification hook land as Stage 0 of automated harmonization (#267), alongside a design document for the DST/JIRA → Seven Bridges → dm-bip flow. bdc-workflow.sh's trans-spec slug parsing and directory resolution moved into tested Python helpers. (#331, #332, #324)
Dependencies
linkml-map0.5.3 → 0.5.4schema-automator0.5.6 → 0.5.7
Both release candidates were validated on real BDC data before the finals shipped — ARIC c2 and CHS c4 harmonized cleanly with no OOM pressure — and CHS c4 was re-run on the final versions. Toy output is byte-identical to the previous pins. (#363)
Upgrading
No action required for single-consent runs; the operating interface is unchanged and additive.
Two things need deliberate steps rather than happening on upgrade:
- The prod app must be repointed at the registry namespace the release publishes to. Tagging creates the image; it does not move the app.
- Cohort mode needs an app on whichever tier you intend to run it.
bdc-cohort-workflow.shships in every image, but only an app that declares the cohort inputs can invoke it.
Known issues
- The map step reports success when an entity fails and produces no output, in non-strict mode only — dev-tier images. Prod runs strict. (#361)
- An output directory reused for a different study co-mingles both studies' products silently. Guard is written and held for the next release. (#357, #366)
--profileis accepted alongside--cohort-modeand silently ignored. (#365)