Skip to content

perf(delta): request the whole node's cores; --mem already reserves it - #504

Merged
joshfactorial merged 1 commit into
developfrom
delta/sv-pipeline-full-node
Aug 4, 2026
Merged

perf(delta): request the whole node's cores; --mem already reserves it#504
joshfactorial merged 1 commit into
developfrom
delta/sv-pipeline-full-node

Conversation

@joshfactorial

Copy link
Copy Markdown
Collaborator

--mem=240G on a 256 GB Delta cpu node reserves the entire node, so --cpus-per-task=64 was leaving 64 cores idle for zero scheduling benefit. Raised to 128.

Sized from measurement

seff 20719077 — soy whole assembly, full pipeline (simulate + align + Manta + Delly + GATK + truvari):

wall-clock 2:54:15
CPU efficiency 14.13% at 64 cores
memory peak 91.78 GB of 240 GB

14% efficiency means serial phases dominate, so this speeds up bwa-mem2 and the callers rather than the whole run. It costs nothing either way.

Do NOT lower --mem

That was my first instinct and it was wrong. 91.78 GB was on a 1 Gb genome; GRCh38 is ~3×, and the parts that scale — bwa-mem2 index (~9 GB soy vs ~28 GB human), sort buffers, eidolon's in-memory sequence — all grow with it. 240 G is closer to necessary than padded, and may even be tight.

The trap this nearly walked into

Job 20745149 reports 39.86 GB and 2.49% CPU. It is not a counter-example: that run re-scored existing artifacts and skipped simulation and alignment entirely (8m39s wall). Sizing from it would have suggested ~48 GB and OOM-killed every replicate mid-alignment.

That is the wrong-measurement failure this repo keeps finding, in its most literal form — a precise, confident, useless number from the wrong job. The comment in the file says so explicitly so the next person doesn't repeat it.

Not fixed by this

The queue wait for an 8-wide array is unchanged — the memory ask means one node per task regardless. sbatch --array=1-8%2 starts on two free nodes instead of waiting for eight, and quarters peak disk as a side benefit.

--mem=240G on a 256 GB Delta cpu node reserves the entire node, so
--cpus-per-task=64 left 64 cores idle for no scheduling benefit. Raised to 128.

Sized from measurement, not guesswork. seff on job 20719077 (soy whole assembly,
FULL pipeline: simulate + align + Manta + Delly + GATK + truvari):

  wall-clock        2:54:15
  CPU efficiency    14.13% at 64 cores
  memory peak       91.78 GB of 240 GB

14% efficiency means serial phases dominate, so this is a speedup on bwa-mem2
and the callers rather than on the whole run. It is free either way.

DO NOT lower --mem to schedule more easily, which was my first instinct and was
wrong: 91.78 GB was on a 1 Gb genome, and GRCh38 is ~3x, with the bwa-mem2 index
(~9 GB soy vs ~28 GB human), sort buffers and eidolon's in-memory sequence all
scaling with it. 240G is closer to necessary than padded.

Job 20745149's 39.86 GB is NOT a counter-example and the comment says so: that
run re-scored existing artifacts and skipped simulation and alignment entirely
(8m39s, 2.49% CPU). Sizing from it would have suggested 48 GB and OOM-killed
every replicate mid-alignment — the wrong-measurement failure this repo keeps
finding, in its most literal form.

The queue wait for an 8-wide array is unchanged by this; the memory ask means
one node per task regardless. Use 'sbatch --array=1-8%2' to start on two free
nodes instead of waiting for eight, which also quarters peak disk.

Co-Authored-By: Claude <noreply@anthropic.com>
@joshfactorial
joshfactorial merged commit 0db136b into develop Aug 4, 2026
4 checks passed
@joshfactorial
joshfactorial deleted the delta/sv-pipeline-full-node branch August 4, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant