perf: pooled mmseqs search bundling (opt-in via --pool_searches)#10
Merged
Conversation
… dbcan3 config newborn2 no longer exists; newborn holds all newborn samples. Also add the missing -c ~/dram_dbs.config (else dbcan-v3 DB is never used), --call (required by the run_dbcan path), and pin to eluring-prod-dev for cohort consistency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Annotation DB searches run per-genome (N genomes x M dbs = NxM process_huge
tasks, each re-loading its DB) — the root of the per-task-overhead blowup and
much of the operational pain (array clone-race, huge fragile resume cache, disk
churn). Result attribution is already genome-safe via {genome}_{scaffold}_{gene}
query ids + the ___ filename delimiter, so searches can be pooled and split back.
This commit adds only the design note and inert scaffolding params
(pool_searches=false, search_chunk_size=0) wired into config + schema; defaults
preserve current per-genome behaviour. No execution path changed yet.
See docs/dev/search-bundling.md for the phased plan + extraves validation gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-2 search bundling (docs/dev/search-bundling.md), MEROPS only, opt-in. When pool_searches is set, the MEROPS path concatenates all genomes' called proteins (collectFile), indexes + searches the pool ONCE, then SPLIT_POOLED_HITS partitions the formatted hits back per genome (query_id -> genome via the per-genome gene-locs tables) and feeds the unchanged SQL/COMBINE path. Default (pool_searches=false) keeps the per-genome path untouched. Hits are bit-score filtered (DB-size independent) so per-genome results are identical; final row order is irrelevant because COMBINE_ANNOTATIONS sorts. Replaces N per-genome MEROPS tasks with 1 index + 1 search + 1 split. UNVALIDATED -- to be checked on extraves (pool vs per-genome diff) before trust. nextflow lint: no errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ing idiom) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validation on extraves caught the bug: Prodigal gene ids are {scaffold}_{gene}
and unique only WITHIN a genome (megahit k141_* scaffold names repeat across
bins), so pooling conflated same-id genes from different bins and the split-back
misassigned them (134467 vs 134507 rows).
New PREFIX_GENES_FOR_POOL prepends <genome>___ to gene ids in both the faa
headers and the gene-locs query_id column before pooling; SPLIT_POOLED_HITS now
recovers the genome from that prefix and strips it (no gene-locs map needed).
nextflow lint: no errors. Re-validating on extraves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ial index Hoist the pooled setup (PREFIX_GENES_FOR_POOL + MMSEQS_INDEX_POOLED + pooled gene-locs -> ch_pooled_search_in) into the MMSEQS_INDEX block so the pooled query DB is built ONCE and reused by every pooled mmseqs search; the per-genome MMSEQS_INDEX is now skipped entirely when --pool_searches (removes the 56 vestigial index tasks seen in the merops prototype run). MEROPS, VIRAL and METHYL now gate per-genome vs pooled on --pool_searches (these are the mmseqs searches the eluring runs use). A guard errors if --pool_searches is combined with the not-yet-pooled mmseqs DBs (kegg/pfam/camper/canthyd/uniref), which still reference the per-genome index. HMM searches are unaffected. nextflow lint: no errors. Validating merops+viral+methyl on extraves next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously the pooled mmseqs path split per genome BEFORE SQL_MEROPS/SQL_VIRAL, so SQL still ran N per-genome tasks. query_ids keep their <genome>___ prefix through sql_add_descriptions (it only adds columns), so run SQL once on the pooled hits and split AFTER. SQL_MEROPS/SQL_VIRAL drop from N tasks to 1 each under --pool_searches. methyl has no SQL (unchanged). nextflow lint: no errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hoist the pooled prefixed-FASTA + gene-locs build out of the mmseqs-index block so
it is available to HMM searches even with no mmseqs DB (ch_pooled_hmm_in). KOFAM and
SULFUR now gate per-genome vs pooled: one hmmsearch over the pooled proteins, split
back per genome by the <genome>___ query-id prefix.
CAVEAT (HMM only): hmm_parser keeps hits on per-profile bit-score thresholds
(DB-size independent), but hmmsearch's -E ${e_value} prefilter scales with the
searched database size, so pooling (larger Z) could in principle drop bit-score-
significant hits near the 1e-5 boundary. Validating on extraves to quantify; the
mmseqs searches have no such dependence (bit-score filtered).
Other HMM searches (camper/fegenie/canthyd/metals/vog) stay per-genome (unaffected,
they read ch_called_proteins directly). nextflow lint: no errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pooling kofam into one hmmsearch over all genes is a single un-parallelised task (slow at scale). With search_chunk_size > 0 the pooled prefixed FASTA is split into that-many-gene chunks, searched in parallel, concatenated (collectFile), then split per genome — task count is O(#chunks) not O(#bins), and the fan-out is back. search_chunk_size = 0 keeps the single-task pooled path. Unified: HMM_SEARCH_KOFAM_POOLED runs once over a channel that is either the single pooled FASTA or N chunks; outputs are collectFile'd into one pooled CSV before the per-genome split. nextflow lint: no errors. Chunked validation on extraves to follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g idiom) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dependence) Reverts the kofam/sulfur HMM pooling + chunking (commits 25a13f7, 653bffd, 4e6b6c6, 8a8e50f). Keeps the mmseqs pooling (merops/viral/methyl), which IS byte-identical and is the shipped win. Why: validated on extraves (v6). hmm_parser keeps hits on per-profile bit-score thresholds (DB-size independent), but hmmsearch's -E prefilter and domain reporting depend on the search-space size (Z). Pooling changes Z, so ~5% of kofam calls churn at the significance boundary (3823 dropped, 3799 gained on extraves). A gene's KO call genuinely depends on what else is in the hmmsearch input — there is no byte-identical HMM pooling without pinning hmmsearch -Z/--domZ, which would ALSO change the per-genome output (vs the cohorts already finished with default Z). So: --pool_searches stays mmseqs-only. HMM searches remain per-genome. The fixed-Z HMM-pooling option is left for a future, deliberate science decision. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…le task) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Opt-in search bundling for the mmseqs annotation searches. Today every DB search runs per genome (
tag { input_fasta }), so a cohort of N bins costs N × #DBsprocess_hugetasks — each re-loading its DB. With--pool_searchesthe called proteins are pooled and each mmseqs DB is searched once, then hits are split back per genome. Output is byte-identical; the win is collapsing the search-task count from #bins × #DBs down to #DBs (e.g. on a 4621-bin cohort, ~tens of thousands of search tasks become ~9).This directly relieves the scheduling/queue pressure (huge SLURM array bursts, giant resume caches) that dominates large-cohort runs.
Default behavior is unchanged — everything is gated behind
--pool_searches(defaultfalse).What's pooled
MMSEQS_INDEX--pool_searches)How it works
PREFIX_GENES_FOR_POOLprepends<genome>___to gene ids (Prodigal ids are unique only within a genome — megahitk141_*scaffolds collide across bins).MMSEQS_INDEXover the pooled proteins; each DB searched once against it.SPLIT_POOLED_HITSpartitions hits back per genome by the prefix and strips it.Validation
Every step gated by an extraves (56-bin) old-vs-new
diff:Task counts for merops+viral+methyl at 56 bins: 336 → 9 (1 index + 3 search + 3 split + 2 SQL). At ~4621 bins that's ~tens-of-thousands of tasks → ~9.
Why HMM is not pooled
kofam/sulfur pooling was implemented + chunked (parallelism) and tested, but reverted:
hmmsearch's-Eprefilter and domain reporting depend on the searched database size (Z), so pooling churns ~5% of kofam calls at the significance boundary (3823 dropped / 3799 gained on extraves). Byte-identical HMM pooling isn't possible without pinninghmmsearch -Z/--domZ, which would also change per-genome output. Left as a future, deliberate decision.Notes
pool_searches(bool),search_chunk_size(int, scaffolding) — both hidden, defaults preserve current behavior.docs/dev/search-bundling.md.nextflow lint: no errors.🤖 Generated with Claude Code