Skip to content

Releases: eunmann/mro2nf

v0.3.0

Choose a tag to compare

@github-actions github-actions released this 01 Jul 17:54
38fb44f
v0.3.0 — idiomatic, portable Nextflow data plane

Epic #18: the generated Nextflow now stages only what each process needs and
carries files as first-class per-file path items — the same pipeline runs
unchanged on AWS Batch + S3, AWS HealthOmics, and local, with a materially
leaner DAG.

Highlights
- De-bundled data plane (#13): payloads cross a process boundary as
  tuple(sidecar, individual leaf paths), not one staged bundle dir — files are
  first-class, per-file-staged, and de-duplicated by the backend.
- Emit-once routing (#14): a call/return that forwards a producer's outputs
  verbatim routes that channel straight through — no intervening re-materialize.
- Fold BIND (#16): non-split and split stage calls resolve their bindings inline
  in the stage task — no standalone BIND_* plumbing processes.
- Publish without the single-node funnel (#12): a sidecar-only LAYOUT computes
  the outs/ layout and a compressed manifest, and a PUBLISH_LEAF fan-out
  publishes each output leaf in parallel. The physical outs/ tree is unchanged.
- Compressed output manifest (#11): manifest.json.gz — a flat, versioned index
  of published outputs a control plane ingests in one GetObject (no S3 LIST).
- Data-movement benchmark + gate (#17): make bench reports tasks / plumbing /
  DAG edges / per-file transfer multiplier and guards against regressions.

DAG reduction (benchmark, before -> after)
- chain: tasks 13 -> 8, DAG edges 40 -> 22, per-file transfer multiplier 11 -> 6
- split: tasks 22 -> 20, DAG edges 28 -> 25, per-file transfer multiplier 21 -> 19

Fidelity
- Byte-identical to real Martian (mrp) across the local differential suite and
  golden e2e, and validated LIVE on AWS Batch + S3 (13/13) and AWS HealthOmics
  (4/4), with manifest.json.gz set-equal to the published outs/ tree on both.
- New reusable parallel harnesses: test/e2e/aws_run.sh, test/e2e/aws_healthomics.sh.

No transpiler changes were required for either cloud backend.

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 28 Jun 02:18
eec11f0
Since v0.1.0:
- GPU support: stage using(special="gpu[:N]") -> accelerator N (AWS Batch/HealthOmics).
- Runtime fidelity: OOM memory escalation on retry (--auto-adjust-memory),
  preflight pipeline gating, array<map<S>>.field projection, split-returned
  __special routing, and declared vmem_gb enforced under --monitor.
- Tooling: 'mro2nf overrides' converts mrp --overrides to a Nextflow -c config;
  RUNTIME_TUNING.md maps the remaining mrp knobs to Nextflow.
- Release: multi-arch runtime base image published to ghcr.io/eunmann/mro2nf-runtime.
- CI: lint/e2e green under branch protection; squash-merge workflow.

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 27 Jun 19:46
cb7ec0d
v0.1.0 — first release

Martian (.mro) -> Nextflow transpiler. Ships the mro2nf transpiler and the mre
runtime shim; runs original Martian stage code unchanged on local/HPC/AWS Batch/
HealthOmics. Validated byte-identical to mrp across the fixture suite and live AWS.