perf(delta): request the whole node's cores; --mem already reserves it - #504
Merged
Conversation
--mem=240G on a 256 GB Delta cpu node reserves the entire node, so --cpus-per-task=64 left 64 cores idle for no scheduling benefit. Raised to 128. Sized from measurement, not guesswork. seff on job 20719077 (soy whole assembly, FULL pipeline: simulate + align + Manta + Delly + GATK + truvari): wall-clock 2:54:15 CPU efficiency 14.13% at 64 cores memory peak 91.78 GB of 240 GB 14% efficiency means serial phases dominate, so this is a speedup on bwa-mem2 and the callers rather than on the whole run. It is free either way. DO NOT lower --mem to schedule more easily, which was my first instinct and was wrong: 91.78 GB was on a 1 Gb genome, and GRCh38 is ~3x, with the bwa-mem2 index (~9 GB soy vs ~28 GB human), sort buffers and eidolon's in-memory sequence all scaling with it. 240G is closer to necessary than padded. Job 20745149's 39.86 GB is NOT a counter-example and the comment says so: that run re-scored existing artifacts and skipped simulation and alignment entirely (8m39s, 2.49% CPU). Sizing from it would have suggested 48 GB and OOM-killed every replicate mid-alignment — the wrong-measurement failure this repo keeps finding, in its most literal form. The queue wait for an 8-wide array is unchanged by this; the memory ask means one node per task regardless. Use 'sbatch --array=1-8%2' to start on two free nodes instead of waiting for eight, which also quarters peak disk. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--mem=240Gon a 256 GB Deltacpunode reserves the entire node, so--cpus-per-task=64was leaving 64 cores idle for zero scheduling benefit. Raised to 128.Sized from measurement
seff 20719077— soy whole assembly, full pipeline (simulate + align + Manta + Delly + GATK + truvari):14% efficiency means serial phases dominate, so this speeds up
bwa-mem2and the callers rather than the whole run. It costs nothing either way.Do NOT lower
--memThat was my first instinct and it was wrong. 91.78 GB was on a 1 Gb genome; GRCh38 is ~3×, and the parts that scale — bwa-mem2 index (~9 GB soy vs ~28 GB human), sort buffers, eidolon's in-memory sequence — all grow with it. 240 G is closer to necessary than padded, and may even be tight.
The trap this nearly walked into
Job 20745149 reports 39.86 GB and 2.49% CPU. It is not a counter-example: that run re-scored existing artifacts and skipped simulation and alignment entirely (8m39s wall). Sizing from it would have suggested ~48 GB and OOM-killed every replicate mid-alignment.
That is the wrong-measurement failure this repo keeps finding, in its most literal form — a precise, confident, useless number from the wrong job. The comment in the file says so explicitly so the next person doesn't repeat it.
Not fixed by this
The queue wait for an 8-wide array is unchanged — the memory ask means one node per task regardless.
sbatch --array=1-8%2starts on two free nodes instead of waiting for eight, and quarters peak disk as a side benefit.